java - 并行流不适用于将多个文件上传到 aws s3

Question

以下方法被写入将文件并行上传到 AWS S3，但没有发生并行处理。上传30个文件需要30秒，上传50个文件需要50秒。我不确定为什么 parallel() 流在我的用例中不起作用。任何指针将不胜感激

public ConcurrentHashMap<String, String> uploadMyFiles(MultipartFile[] files, String prefix) {
    ConcurrentHashMap<String, String> map = new ConcurrentHashMap<String, String>();
    Stream.of(files).parallel().forEach(file -> {
        String fStatus =  uploadMyFile(file, prefix);                   
        map.put(file.getOriginalFilename(), fStatus);
    }); 

    return fileStatusMap;
}


public String uploadMyFile(final MultipartFile multipartFile, String prefix) {
    try {
        final File file = convertMPartFileToFile(multipartFile);
        uploadToMyBucket("mybucket", file, prefix);

    } catch (final AwsServiceException exception) {         
    
        return exception.getMessage()
    }
    return "OK";
}

private File convertMPartFileToFile(final MultipartFile multipartFile) {
    final File file = new File(multipartFile.getOriginalFilename());
    try (final FileOutputStream outputStream = new FileOutputStream(file)) {
        outputStream.write(multipartFile.getBytes());
    } catch (final IOException ex) {
        LOGGER.error("log error here");
    }
    return file;
}

private String uploadToMyBucket(final String bucket, final File file, String fileKey) {

    PutObjectResponse putObjectResult = s3Client
            .putObject(PutObjectRequest.builder().bucket(bucket).key(fileKey).build(), RequestBody.fromFile(file));
    final URL myfileUrl = s3Client.utilities().getUrl(GetUrlRequest.builder().bucket(bucket).key(fileKey).build());

    return myfileUrl.toString();
}

标签： javaamazon-s3java-8

Answer 1

我建议使用TransferManagerwith ExecutorService，如下面的示例所示；

并行化大型上传以提高速度和可靠性

Amazon S3 中的分段上传