amazon-web-services - S3 GetObjectTagging error for only subset of similar files

Question

I have a spark application that creates 1000 files in S3 bucket. Later I need to move these files to a different directory in the same S3 bucket. But when I try this using AWSCLI, a very few files (~10) are not moved and show the error of AccessDenied as follows:

[padmanabhpande]$ aws s3 cp s3://my-bucket-name/datasets_tmp s3://my-bucket-name/datasets-2020-06-22 --recursive

copy failed: s3://my-bucket-name/datasets_tmp/train-2020-06-22/id=138751/part-00020-6f77e24e-4df8-4ef5-9743-1d7a84deba40.c000.snappy.parquet to s3://my-bucket-name/datasets-2020-06-22/train-2020-06-22/id=138751/part-00020-6f77e24e-4df8-4ef5-9743-1d7a84deba40.c000.snappy.parquet An error occurred (AccessDenied) when calling the GetObjectTagging operation: Access Denied

Why this is happening for only 10s of files and rest 990 are moved successfully, when the same spark application creates all these files at the same time.

标签： amazon-web-servicesamazon-s3pyspark