java - 通过多线程同时上传多个文件到 S3 抛出 java.nio.channels.ClosedChannelException: null
问题描述
我需要同时将多个文件上传到 AWS S3 。我使用 ExecuterService 在不同的线程上上传文件。当我开始 arround 100 个线程时,它开始抛出异常:
com.amazonaws.SdkClientException: Unable to execute HTTP request: null
| at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1114) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1064) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325) ~[aws-java-sdk-s3-1.11.248.jar!/:na]
| at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272) ~[aws-java-sdk-s3-1.11.248.jar!/:na]
| at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1749) ~[aws-java-sdk-s3-1.11.248.jar!/:na]
| ... 9 common frames omitted
| Caused by: java.nio.channels.ClosedChannelException: null
| at sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:110) ~[na:1.8.0_171]
| at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:147) ~[na:1.8.0_171]
| at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) ~[na:1.8.0_171]
| at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[na:1.8.0_171]
| at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_171]
| at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.read(MD5DigestCalculatingInputStream.java:128) ~[aws-java-sdk-s3-1.11.248.jar!/:na]
| at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[na:1.8.0_171]
| at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[na:1.8.0_171]
| at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[na:1.8.0_171]
| at com.amazonaws.internal.SdkBufferedInputStream.read(SdkBufferedInputStream.java:76) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:82) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140) ~[httpcore-4.4.9.jar!/:4.4.9]
| at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:160) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) ~[httpcore-4.4.9.jar!/:4.4.9]
| at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:160) ~[httpclient-4.5.4.jar!/:4.5.4]
| at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) ~[httpcore-4.4.9.jar!/:4.4.9]
| at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:63) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) ~[httpcore-4.4.9.jar!/:4.4.9]
| at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.4.jar!/:4.5.4]
| at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.4.jar!/:4.5.4]
| at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.4.jar!/:4.5.4]
| at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.4.jar!/:4.5.4]
| at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.4.jar!/:4.5.4]
| at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) ~[aws-java-sdk-core-1.11.248.jar!/:na]
| ... 20 common frames omitted
我使用的 Java AWS Sdk 版本是1.11.248
.
ExecuterSERvice 用作:
ExecutorService ioThreadPool = Executors.newFixedThreadPool(100);
ConcurrentLinkedDeque<Future<?>> uploadTasks = new ConcurrentLinkedDeque<>();
Future<?> future = ioThreadPool.submit(() -> {
try {
putObjectToS3();
} catch (IOException e) {
throw new InternalErrorException("IOException during file upload.", e);
}
});
uploadTasks.push(future);
CompletableFuture.allOf(uploadTasks.toArray(new CompletableFuture<?>[0])).join();
我还将maxNumberOfConnection
AWS 客户端中的设置为150
谢谢
解决方案
看来您需要使用ClientConfiguration
如下方式实现重试机制:
AmazonS3Client s3Client = AmazonS3ClientBuilder.standard().withRegion(REGION).
.withClientConfiguration(getClientConfiguration()).build();
private ClientConfiguration getClientConfiguration() {
ClientConfiguration clientConfiguration = new ClientConfiguration();
clientConfiguration.setRetryPolicy(new RetryPolicy(null, null, MAX_TRIES, false);
return clientConfiguration;
}
您可以在特定于您的应用程序的 s3 客户端构建器中添加/更新上述配置。我已经使用基本配置向您展示了它的用途。
现在,尝试使用此配置运行您的程序。
推荐阅读
- mongodb - 如何从实际日期开始(我的意思是今天)一直在一年前的 mongo 查询数据集?
- javascript - 通过 TYPESCRIPT 中的值获取 Enum 元素
- python - Flutter - 连接到 Python/Pytorch 后端
- java - 如何在Java中获取用户出生日期的月份和日期
- swiftui - Swift UI 将绑定传递给另一个视图以通过 init 发出警报
- c++ - 在 vcpkg 中检测三元组 x64-windows 故障的编译器哈希
- google-cloud-platform - Slack 与 Cloud Composer 的集成 - 损坏的 DAG 没有名为“slackclient”的模块
- r - cdfCompare 中的错误,找不到对象“离散”
- prism - 使用.Net Core 3.0 中的 XAML 行为导航到模块中的视图(用户控件)未将 GetFocus 事件绑定触发到事件触发器中
- python - Python Flask 在触发 xlsx 生成和服务文件存储库的过程中自动关闭