java - 谷歌云存储同时上传多个大文件时抛出错误

问题描述

我们正在尝试将多个文件（即 100 个或更多文件）上传到 Google Cloud Storage，但我们随机收到以下错误。
平均文件大小：30MB - 40 MB

 08:57:15.488 [ForkJoinPool-1-worker-26] INFO  
 demo.storage.GCPStorageComponent - Error writing request body to
 server com.google.cloud.storage.StorageException: Error writing
 request body to server
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:189)
     at com.google.cloud.storage.spi.v1.HttpStorageRpc.write(HttpStorageRpc.java:581)
     at com.google.cloud.storage.BlobWriteChannel$1.run(BlobWriteChannel.java:51)
     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
     at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.storage.BlobWriteChannel.flushBuffer(BlobWriteChannel.java:47)
     at com.google.cloud.BaseWriteChannel.close(BaseWriteChannel.java:160)
     at demo.storage.GCPStorageComponent.uploadFile(GCPStorageComponent.java:56)
     at demo.storage.StorageRepository.uploadFile(StorageRepository.java:56)
     at demo.restful.UploadRepository.uploadFileToBucket(UploadRepository.java:132)
     at demo.restful.UploadRepository.processClipResults(UploadRepository.java:56)
     at demo.restful.UploadRepository.lambda$getClipModelCallableList$0(UploadRepository.java:85)
     at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424)
     at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
     at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
     at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
     at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
 Caused by: java.io.IOException: Error writing request body to server
     at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3536)
     at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3519)
     at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:55)
     at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
     at com.google.api.client.http.AbstractInputStreamContent.writeTo(AbstractInputStreamContent.java:72)
     at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:80)
     at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
     at com.google.cloud.storage.spi.v1.HttpStorageRpc.write(HttpStorageRpc.java:563)
     ... 17 more

下面是我们用来上传文件的代码。我们正在使用多部分方法上传 1 MB 的文件。

public void uploadFile(String bucketName, String bucketFileName, Path localFilePath)
        throws IOException, StorageException {
    BlobInfo blobInfo = BlobInfo.newBuilder(bucketName, bucketFileName).build();
    ByteBuffer buffer = ByteBuffer.allocate(10_000_000); //i.e. 1 MB
    try (WriteChannel writer = storage.writer(blobInfo);
            ByteChannel inChannel = Files.newByteChannel(localFilePath);) { // write to bucket
        while (inChannel.read(buffer) > 0) {
            buffer.flip();
            writer.write(buffer);
            buffer.clear();
        }
    } catch (com.google.cloud.storage.StorageException ex) {
        LOGGER.info(ex.getMessage());
        ex.printStackTrace();
    }
}

编辑：正在从 Google Cloud 内部进行访问
An Github Issue已针对同一问题打开

标签： javagoogle-cloud-storage

java - 谷歌云存储同时上传多个大文件时抛出错误

问题描述

解决方案

推荐阅读