时间戳

首页 > 解决方案 > Java，您如何有效地分块大型输入流？

java - Java，您如何有效地分块大型输入流？

问题描述

我有一个可能为 20-30mb 的输入流。我正在尝试将块作为多部分文件上传到 S3。

我有可用的内容长度，我有可用的输入流。考虑到记忆，我怎样才能有效地做到这一点。

我看到有人做过这样的事情，但不确定我是否完全理解：

    int contentLength = inputStreamMetadata.getContentLength();
    int partSize = 512 * 1024; // Set part size to 2 MB
    int filePosition = 0;

    ByteArrayInputStream bais = inputStreamMetadata.getInputStream();
    List<PartETag> partETags = new ArrayList<>();
    byte[] chunkedFileBytes = new byte[partSize];
    for (int i = 1; filePosition < contentLength; i++) {
      // Because the last part could be less than 5 MB, adjust the part size as needed.
      partSize = Math.min(partSize, (contentLength - filePosition));

      filePosition += bais.read(chunkedFileBytes, filePosition, partSize);

      // Create the request to upload a part.
      UploadPartRequest uploadRequest = new UploadPartRequest()
          .withBucketName(bucketName)
          .withUploadId(uploadId)
          .withKey(fileName)
          .withPartNumber(i)
          .withInputStream(new ByteArrayInputStream(chunkedFileBytes, 0, partSize))
          .withPartSize(partSize);

      UploadPartResult uploadResult = client.uploadPart(uploadRequest);
      partETags.add(uploadResult.getPartETag());
    }
}

具体来说这件作品：.withInputStream(new ByteArrayInputStream(bytes, 0, bytesRead))

标签： javaamazon-web-services

解决方案

对不起，我不能（轻松）测试它，但我认为你真的很接近，......只需要“修复”和“安排”你的循环！

将https://stackoverflow.com/a/22128215/592355与您的最新代码相结合：

int partSize = 5 * 1024 * 1024; // Set part size to 5 MB
ByteArrayInputStream bais = inputStreamMetadata.getInputStream();
List<PartETag> partETags = new ArrayList<>();
byte[] buff = new byte[partSize];
int partNumber = 1;
while (true) {//!
    int readBytes = bais.read(buff);// readBytes in [-1 .. partSize]!
    if (readBytes == -1) { //EOF
        break;
    }
    // Create the request to upload a part.
    UploadPartRequest uploadRequest = new UploadPartRequest()
                .withBucketName(bucketName)
                .withUploadId(uploadId)
                .withKey(fileName)
                .withPartNumber(partNumber++)
                .withInputStream(new ByteArrayInputStream(buff, 0, readBytes))
                .withPartSize(readBytes);

    UploadPartResult uploadResult = client.uploadPart(uploadRequest);
    partETags.add(uploadResult.getPartETag());
}
// Complete the multipart upload.... 
// https://docs.aws.amazon.com/AmazonS3/latest/dev/llJavaUploadFile.html

推荐阅读