amazon-web-services - How to speed up the copy in S3 bucket?

Question

I have a bucket bucket_a and trying to copy the content of this bucket into another bucket bucket_b but it is taking huge time to copy the objects from a folder folder_a of bucket bucket_a to folder_a of bucket bucket_b as there are nearly 9k objects in each folder with size 600MB each and I have 20 folders like that.

I tried accelerate option of bucket and used cp command of aws cli and it looks like it will take another 2-3 days to copy the contents.

标签： amazon-web-servicesamazon-s3aws-cliawss3transfermanagerawss3transferutility

Answer 1

加速这种传输的一种方法是确保您用于传输的角色被允许读取源存储桶并写入目标存储桶。正确设置角色和存储桶策略后，您可以防止数据在 AWS 服务器之外遍历。这样可以加快传输时间并节省您自己的数据使用量。

请参阅：https ://aws.amazon.com/premiumsupport/knowledge-center/copy-s3-objects-account/和https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles .html

加快传输速度的另一种方法是调整复制过程中使用的一些设置。您可以指定块大小、可以同时进行传输的线程数以及其他设置。请参阅：https ://docs.aws.amazon.com/cli/latest/topic/s3-config.html

您可能希望至少使用 4-10 max_concurrent_requests（线程数），并multipart_chunksize根据可用内存和允许的并发请求数从默认值增加到某个值。

我发现 aws cli 工具基本上能够消耗你所有的 CPU 和内存，所以你需要小心你如何设置它。