amazon-web-services - 如何使用 shell 脚本列出未使用的 AWS S3 存储桶和空存储桶？

Question

我正在寻找过去 90 天未使用的 s3 存储桶列表以及空存储桶列表。

为了得到它，我尝试编写如下代码：

#/bin/sh
for bucketlist in  $(aws s3api list-buckets --query "Buckets[].Name");
do
  listobjects=$(\
    aws s3api list-objects --bucket $bucketlist \
    --query 'Contents[?contains(LastModified, `2020-08-06`)]')
done

此代码打印以下输出：[我只添加了一个桶的结果以供参考]

{
    "Contents": [
        {
            "Key": "test2/image.png",
            "LastModified": "2020-08-06T17:19:10.000Z",
            "ETag": "\"xxxxxx\"",
            "Size": 179008,,
            "StorageClass": "STANDARD",
        }
    ]
}

期望：

1. 在上面的代码中，我只想打印过去 90 天内未修改/使用的对象的存储桶列表。
2. 我也在寻找空的遗愿清单

我不擅长编程，有人可以指导我吗？预先感谢您对我们的支持。

标签： amazon-web-servicesshellamazon-s3shaws-cli

Answer 1

这是我今天写的一个脚本。它不会改变任何东西，但它确实为您提供了进行更改的命令行。

#!/bin/bash
profile="default"
olddate="2020-01-01"
smallbucketsize=10

emptybucketlist=()
oldbucketlist=()
smallbucketlist=()

#for bucketlist in  $(aws s3api list-buckets  --profile $profile  | jq --raw-output '.Buckets[6,7,8,9].Name'); # test this script on just a few buckets
for bucketlist in  $(aws s3api list-buckets  --profile $profile  | jq --raw-output '.Buckets[].Name');
do
  echo "* $bucketlist"
  if [[ ! "$bucketlist" == *"shmr-logs" ]]; then
    listobjects=$(\
      aws s3api list-objects --bucket $bucketlist \
      --query 'Contents[*].Key' \
      --profile $profile)
#echo "==$listobjects=="
    if [[ "$listobjects" == "null" ]]; then
          echo "$bucketlist is empty"
          emptybucketlist+=("$bucketlist")
    else
      # get size
      aws s3 ls --summarize  --human-readable --recursive --profile $profile s3://$bucketlist | tail -n1

      # get number of files
      filecount=$(echo $listobjects | jq length )
      echo "contains $filecount files"
      if [[ $filecount -lt $smallbucketsize ]]; then
          smallbucketlist+=("$bucketlist")
      fi

      # get number of files older than $olddate
      listoldobjects=$(\
        aws s3api list-objects --bucket $bucketlist \
        --query "Contents[?LastModified<=\`$olddate\`]" \
        --profile $profile)
      oldfilecount=$(echo $listoldobjects | jq length )
      echo "contains $oldfilecount old files"

      # check if all files are old
      if [[ $filecount -eq $oldfilecount ]]; then
        echo "all the files are old"
        oldbucketlist+=("$bucketlist")

      fi
    fi
  fi
done
echo -e "\n\n"

echo "check the contents of these buckets which only contain old files"
for oldbuckets in ${oldbucketlist[@]};
do
  echo "$oldbuckets"
done
echo -e "\n\n"

echo "check the contents of these buckets which don't have many files"
for smallbuckets in ${smallbucketlist[@]};
do
  echo "aws s3api list-objects --bucket $smallbuckets --query 'Contents[*].Key' --profile $profile"
done
echo -e "\n\n"

echo "consider deleting these empty buckets"
for emptybuckets in "${emptybucketlist[@]}";
do
  echo "aws s3api delete-bucket --profile $profile --bucket $emptybuckets"
done