时间戳

首页 > 解决方案 > 将大文件上传到 Azure Blob 存储引发异常

file-upload - 将大文件上传到 Azure Blob 存储引发异常

问题描述

我有一个通过 Window Service 将数据库备份上传到 Azure Blob 存储的方案。它适用于 300-500 MB 之间的 bak 文件大小范围，但如果大小超过 700 MB 到 1 GB 或更多。花了一个多小时，然后抛出异常。

请检查下面的代码，让我知道我做错了什么以及将大文件上传到 blob 存储的有效方法是什么。这两种方法我都试过了。

public static void UploadFile(AzureOperationHelper azureOperationHelper)
{
    CloudBlobContainer blobContainer = CreateCloudBlobContainer(tenantId, applicationId,
                                       clientSecret, azureOperationHelper.storageAccountName, azureOperationHelper.containerName,
                                       azureOperationHelper.storageEndPoint);
    blobContainer.CreateIfNotExists();


    var writeOptions = new BlobRequestOptions()
    {
        SingleBlobUploadThresholdInBytes = 50 * 1024 * 1024,//maximum for 64MB,32MB by default  
        ParallelOperationThreadCount = 12,
    };

    CloudBlockBlob blob = blobContainer.GetBlockBlobReference(azureOperationHelper.blobName);
    //blob.UploadFromFile(azureOperationHelper.srcPath);            
    blob.UploadFromFile(azureOperationHelper.srcPath, options: writeOptions);
}

public static void UploadFileStream(AzureOperationHelper azureOperationHelper)
{
    CloudBlobContainer blobContainer = CreateCloudBlobContainer(tenantId, applicationId,
                                       clientSecret, azureOperationHelper.storageAccountName, azureOperationHelper.containerName,
                                       azureOperationHelper.storageEndPoint);
    blobContainer.CreateIfNotExists();
    CloudBlockBlob blob = blobContainer.GetBlockBlobReference(azureOperationHelper.blobName);

    //byte[] contents = File.ReadAllBytes(azureOperationHelper.srcPath);
    //var writeOptions = new BlobRequestOptions()
    //{
    //    SingleBlobUploadThresholdInBytes = 50 * 1024 * 1024,//maximum for 64MB,32MB by default  
    //    ParallelOperationThreadCount = 12,        
    //};
    //blob.UploadFromByteArray(contents, 0, contents.Length, AccessCondition.GenerateIfNotExistsCondition(), options: writeOptions);
    blob.StreamWriteSizeInBytes = 100 * 1024 * 1024; //100 MB
    blob.UploadFromFile(string.Format(azureOperationHelper.srcPath));
    //using (var fs = new FileStream(azureOperationHelper.srcPath, FileMode.Open))
    //{
    //    blob.UploadFromStream(fs);
    //}            

}

以下是我得到的例外。

Microsoft.WindowsAzure.Storage.StorageException：远程服务器返回错误：（403）禁止。---> System.Net.WebException：远程服务器返回错误：（403）禁止。在 Microsoft.WindowsAzure.Storage.Shared.Protocol.HttpResponseParsers.ProcessExpectedStatusCodeNoException[T]（HttpStatusCode expectedStatusCode，HttpStatusCode actualStatusCode，T retVal，StorageCommandBase`1 cmd，异常 ex）

Microsoft.WindowsAzure.Storage.StorageException：客户端无法在指定的超时时间内完成操作。---> System.TimeoutException：客户端无法在指定的超时时间内完成操作。

标签： file-uploadazure-storageazure-blob-storageazure-storage-files

解决方案

请使用下面的代码，它在我身边运行良好（大约 2GB 文件，完成上传大约需要 10 分钟）：

public string UploadFile(string sourceFilePath)
{
    try
    {
        string storageAccountConnectionString = "AZURE_CONNECTION_STRING";
        CloudStorageAccount StorageAccount = CloudStorageAccount.Parse(storageAccountConnectionString);
        CloudBlobClient BlobClient = StorageAccount.CreateCloudBlobClient();
        CloudBlobContainer Container = BlobClient.GetContainerReference("container-name");
        Container.CreateIfNotExists();
        CloudBlockBlob blob = Container.GetBlockBlobReference( Path.GetFileName(sourceFilePath) );
        HashSet<string> blocklist = new HashSet<string>();

        byte[] fileContent = File.ReadAllBytes(sourceFilePath);
        const int pageSizeInBytes = 10485760;
        long prevLastByte = 0;
        long bytesRemain = fileContent.Length;

        do
        {
            long bytesToCopy = Math.Min(bytesRemain, pageSizeInBytes);
            byte[] bytesToSend = new byte[bytesToCopy];
            Array.Copy(fileContent, prevLastByte, bytesToSend, 0, bytesToCopy);
            prevLastByte += bytesToCopy;
            bytesRemain -= bytesToCopy;

            //create blockId
            string blockId = Guid.NewGuid().ToString();
            string base64BlockId = Convert.ToBase64String(Encoding.UTF8.GetBytes(blockId));

            blob.PutBlock(
                base64BlockId,
                new MemoryStream(bytesToSend, true),
                null
                );

            blocklist.Add(base64BlockId);

        } while (bytesRemain > 0);

        //post blocklist
        blob.PutBlockList(blocklist);

        return "Success";
    }
    catch (Exception ex)
    {
        return ex.Message;
    }
}

它适用于上传大文件（有人从这里给出解决方案）。

请让我知道您是否可以使用它。

推荐阅读