Raymond Raymond

Modern Web Application - Azure Blob Storage for Uploaded Files

event 2020-04-11 visibility 11,628 comment 8 insights toc
more_vert
insights Stats
Modern Web Application - Azure Blob Storage for Uploaded Files

File upload is a very common and critical feature for many web applications. It allows website users to uploaded images and documents to the web server, which will then be displayed in web pages or apps.  

warning The following information is based on my own knowledge and doesn't represent Azure or any other person/organization's opinions.

Classic monolith web application

In a classic monolith web application, web server usually stores all the uploaded files. The high-level process looks like the following:

2020041140722-image.png

The File Store/Retrieve Handler in the above diagram usually just stores the files locally and will call native I/O APIs to store or retrieve file from Web Server disk.

This architecture works well with small applications but will struggle to meet big online applications requirement as it has the following limitations:

  • Hard to scale. Usually the disk size in the web application is limited 
  • I/O heavy on Web Server. I/O read and write operations are required to handle web requests frequently. 
  • Potential security issues. Files permissions are coupled with web application executable permissions and sometimes can be leveraged by hackers.
  • Coupled and complex application. The file storage is bundled with web application program storage. It will become difficult to migrate and maintain.
  • Higher cost. We don't need I/O optimized storage for user uploaded files compared with the web application storage. To save those files on high performance web application disk is a waste of money.

Fortunately, with Cloud platforms like Azure, we can easily resolve these problems. I'm using Azure for this article, the concept can apply to other cloud platforms as GCP, AWS, Alibaba Cloud, etc. 

Modern scalable web application on Cloud

To address these issues in the monolith web application, we can migrate it to Cloud. For image or document file storage, we can use Azure Blob Storage; for web application hosting, we can use Azure App Service (elastic and containerization option is also available).

The following diagram shows the revised file upload and retrieve sequences:

2020041143548-image.png

In this architecture, Azure App Service application calls Azure Blob Storage APIs to store file into blob storage container.  Azure Blob Storage API also returns the URLs for the stored object file. The URL file will be then passed back to the user client. Users can then use the absolute Azure Blob Storage file object URL to view or download the application. 

Enhanced security

In this architecture, Azure Blob Storage file URLs are directly exposed. This means all users that know the absolute URL or each blob  file can directly access it and container needs to be configured as publicly accessible. 

What if you want to control the permissions to these resources in Azure Blob Storage?

There are two main approaches:

  1. Integrate Azure Active Directory into your web applications and configure IAM at container using Azure Active Directory.
  2. Implement another logic layer for file upload and download in your web application and manage permissions in your applications. In this approach, connections to Azure Blob Storage endpoints are only exposed to the web application.

In Kontext (this website), I'm using the second approach. There is a web API exposed to retrieve uploaded files from Azure Blob Storage. 

For example, this URL https://kontext.tech/api/flex/medias/obj-1713 will returns the following image:

2020041102023-microsoft-azure-sql-database-microsoft-sql-server-database.png

The absolute URL for this image on Azure is: https://***.blob.core.windows.net/site-images/u-42/2020041102023-microsoft-azure-sql-database-microsoft-sql-server-database.png (storage account name is masked).

No one will be able to directly access this image as I am not making it public; however everybody can access it through this website as the API layer is handling authentication and authorization for user's requests and it also interacts with Azure Blob Storage via APIs. You can find more details in the Code snippets section.

With this approach, the application can decide who can access the image based on their roles or claims in the web applications; when an resource is deleted, we can softly delete it in the application only without physically deleting it on Azure Blob Storage. This also gives me flexibility to handle the following scenarios:

  • Requested resources are deleted by users. 
  • User has no access to the requested resource.

For example, https://kontext.tech/api/flex/medias/obj-100000713 shows the following customized SVG image as it doesn't exist on Kontext:


The diagram for sequence activities now looks like the following:

2020041152346-image.png

Code snippets

Azure SDK provides APIs to access Azure Blob Storage resources easily. At the moment, the following SDK or tools are available to interact with this service:

2020041143301-image.png

Fro this article, I'm going to use .NET (ASP.NET Core) to show you how to write files into blob storage or retrieve data from it. 

Prerequisites 

Create a ASP.NET Core 3 web application with reference to Azure.Storage.Blobs package in the project file:

<ItemGroup>
    <PackageReference Include="Azure.Storage.Blobs" Version="12.2.0" />
</ItemGroup>
We will use this .NET library to access Azure Blob Storage.

Create a Azure Blob Storage container using Azure Portal. Get a connection string for read data from and write data into the container.

The connection string looks like the following:

DefaultEndpointsProtocol=https;AccountName=youraccountname;AccountKey=gkd******==;EndpointSuffix=core.windows.net

You can retrieve this from your Azure storage account Access keys in Settings:

2020041145057-image.png

Write file into Azure Blob Storage

The following code snippet shows you how to write a file upload HTTP stream into the storage.

public async Task<(bool IsSuccessful, string Path, string PhysicalPath, string Error)> UploadFileAsync(string fileName, string fileType, Stream stream, object bucket)
        {
            var actualFileName = fileName.Replace('\\', '_').Replace('/', '_').Replace(" ", string.Empty);
            try
            {
                BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
                var blobFilePath = $"{bucket}/{actualFileName}";
                var blobContainer = blobServiceClient.GetBlobContainerClient(containerName);
                var blobClient = blobContainer.GetBlobClient(blobFilePath);
                await blobClient.UploadAsync(stream);
                await blobClient.SetMetadataAsync(new Dictionary<string, string> { { "userid", bucket.ToString() } });
                stream.Close();
                return (true, blobFilePath, blobClient.Uri.ToString(), null);
            }
            catch (Exception ex)
            {
                return (false, null, null, ex.ToString());
            }

        }

In the above code snippet, I used one parameter bucket for each of web users to ensure the blob file path is unique to each user. The stream parameter is the object constructed from the HTTP file upload stream. 

BlobClient instance will return the full URI of the uploaded file which can be then directly accessed. I also called SetMetadataAsync function to tag the file uploaded. Tags can help us to link the stored files with other entities in your website applications. 

Download file from Azure Blob Storage

The following code snippet shows how to download file from the storage using BlobClient object. The returned BlobDownloadInfo object has attribute Value.Content which is an instance of Stream class that can be directly used in your HTTP handler. 

public async Task<(bool IsSuccessful, Stream Stream, string Error)> DownloadFileAsync(string fileName, string fileType, string virtualFilePath)
        {
            try
            {
                BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
                var blobContainer = blobServiceClient.GetBlobContainerClient(containerName);
                var blobClient = blobContainer.GetBlobClient(virtualFilePath);
                var download = await blobClient.DownloadAsync();
                return (true, download.Value.Content, null);
            }
            catch (Exception ex)
            {
                return (false, null, ex.ToString());
            }
        }

Delete file from Azure Blob Storage

Similarly, you can also delete file from the storage using the following code snippet:

 public async Task<(bool IsSuccessful, string Error)> DeleteFileAsync(string fileName, string fileType, string filePath)
        {
            try
            {
                BlobServiceClient blobServiceClient = new BlobServiceClient(connectionString);
                var blobContainer = blobServiceClient.GetBlobContainerClient(containerName);
                var blobClient = blobContainer.GetBlobClient(filePath);
                await blobClient.DeleteIfExistsAsync();
                return (true, null);
            }
            catch (Exception ex)
            {
                return (false, ex.ToString());
            }
        }

BlobClient.DeleteIfExistsAsync function is used to delete the file from storage.

There are many other operations you can perform using this library. Feel free to explore the documentation on official documentation website. 

Summary

It's very easy to utilize Azure to implement modern web applications that supports massive scalable user upload files. Once storage (files) is segregated form computing (web application program host), we can also deploy the application as package in Azure DevOps piepline. Refer the following documentation for details: Run your app in Azure App Service directly from a ZIP package.

If you have other better approaches to deign user content storage for web applications on Cloud, feel free to post a comment here. 

References

The following article used similar approach as this article but implemented through customized IActionResult and executor:

Efficiently serving Blob Storage in ASP.NET Core | CodeX

More from Kontext
comment Comments
Raymond Raymond #1876 access_time 5 months ago more_vert

You can find examples on the official documentation https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-download 



format_quote

person azhar access_time 5 months ago

Can you please give some more detail and code snippets of blow implementation as well 

  1. Implement another logic layer for file upload and download in your web application and manage permissions in your applications. In this approach, connections to Azure Blob Storage endpoints are only exposed to the web application.   

A azhar arshad #1875 access_time 5 months ago more_vert

Can you please give some more detail and code snippets of blow implementation as well 

  1. Implement another logic layer for file upload and download in your web application and manage permissions in your applications. In this approach, connections to Azure Blob Storage endpoints are only exposed to the web application.   

Raymond Raymond #1674 access_time 2 years ago more_vert

That is correct though the upload file stream can be directly used to as input to save to Azure Blob Storage, thus there is no need to save the stream to file in web server first. 

format_quote

person Daniel access_time 2 years ago

Looks to me like the file is uploaded to the web app first, then uploaded to the blob storage. Is that correct?

D Daniel Mejia #1673 access_time 2 years ago more_vert

Looks to me like the file is uploaded to the web app first, then uploaded to the blob storage. Is that correct?

Raymond Raymond #1504 access_time 3 years ago more_vert

Thanks for sharing, I've read through and it is a good article. 

format_quote

person Maryna access_time 3 years ago

Great stuff! I also recommend reading this article by Cleveroad to learn more about web app architecture.

M Maryna Bobrovskaya #1503 access_time 3 years ago more_vert

Great stuff! I also recommend reading this article by Cleveroad to learn more about web app architecture.

Raymond Raymond #429 access_time 4 years ago more_vert

Hello,

When uploading files to Azure Blob Storage, a fully qualified URL will be returned. That URL can be directly used in your website. For example,

<img src="${URL of the blob object file}"/>

However, this also means that your uploaded file needs to be accessible publicly if you need all the users to access it.

For Kontext, all the uploaded image files are accessed via a proxy API as mentioned in the article.

Hope that makes sense.

format_quote

person Data access_time 4 years ago

This is a great article.

One question for you: in the article you talk about having the users go direct to azure to get the blob images/data so as to not overtax your site. IE your site isn't downloading the images/data and then inserting into pages and serving them, the user is instead taxing the azure server farm to get azure resources. You mention that the URLs are not exposed somehow, can you tell us how you solved that, was it some sort of routing solution? 

thanks!

D Data Vids #427 access_time 4 years ago more_vert

This is a great article.

One question for you: in the article you talk about having the users go direct to azure to get the blob images/data so as to not overtax your site. IE your site isn't downloading the images/data and then inserting into pages and serving them, the user is instead taxing the azure server farm to get azure resources. You mention that the URLs are not exposed somehow, can you tell us how you solved that, was it some sort of routing solution? 

thanks!

Please log in or register to comment.

account_circle Log in person_add Register

Log in with external accounts