Andrew Savetchuk
Andrew Savetchuk's Blog

Follow

Andrew Savetchuk's Blog

Follow
How to Find the N Last Largest Files in an AWS S3 Bucket Using Amazon Command Line Interface

How to Find the N Last Largest Files in an AWS S3 Bucket Using Amazon Command Line Interface

Using one simple command

Andrew Savetchuk's photo
Andrew Savetchuk
·Jan 20, 2023·

2 min read

Table of contents

  • Finding Files using AWS CloudShell
  • Ignore LastModified Property
  • Adjusting the Number of Files
  • Bonus Command

I recently struggled to find the right command to search for large files in an AWS S3 bucket and spent many hours researching this topic. I am new to Amazon CloudShell, so I had to ask the community for help, and I even created an account on AWS re:Post (aka the StackOverflow alternative for Amazon Web Services). Although I recommend using it if you need help with AWS, due to the small number of users on re:Post you can usually get help faster on StackOverflow. So if you have a question, it is better to ask on both platforms.

Finding Files using AWS CloudShell

For this example, I have chosen Amazon CloudShell as the easiest tool to use with the AWS CLI that does not require any software to be installed on your computer. You can use any other tool you like.

To find the N most recent largest files in an AWS S3 bucket, you need to open console.aws.amazon.com and find CloudShell (I recommend adding it to your favourites in case you need to use it again).

Before running the command you need to know your S3 bucket name. When you have it ready, you can use the command below to find files:

aws s3api list-objects-v2 --bucket BUCKETNAME --query 'sort_by(Contents[?LastModified>=`2023-01-01`], &Size)[-5:]'

Replace BUCKETNAME with your S3 bucket name and change the date (2023-01-01) as needed. After running the command, your output should look like this:

Ignore LastModified Property

If you want to find the last N files regardless of the LastModified property, you can configure your command like this:

aws s3api list-objects-v2 --bucket BUCKETNAME --query 'sort_by(Contents, &Size)[-5:]'

Adjusting the Number of Files

You can adjust the number of files to be returned by changing the number in the square brackets ([-5:]), or you can remove the square brackets with the contents inside to display all files sorted by size. Be careful to run such a command on a large bucket though, as it can take minutes to perform the operation (I did it once...).

Bonus Command

There is another command that I rarely use but thought it would be useful to share.

aws s3 ls --summarize --human-readable --recursive s3://BUCKETNAME

This command will display all files in the bucket in a compact form with LastModified, Size and Name properties. If you have ever run the ls command on a Linux/Unix operating system before, you know what to expect.


The end. I hope you found this article helpful. Stay tuned for more content! If you want to support me, you can buy me a coffee or send a donation using the instruction below :)

Did you find this article valuable?

Support Andrew Savetchuk by becoming a sponsor. Any amount is appreciated!

Learn more about Hashnode Sponsors
 
Share this