I Have a bucket with 3 million objects. I Even don't know how many folders are there in my S3 bucket and even don't know the names of folders in my bucket.I want to show only list of folders of AWS s3. Is there any way to get list of all folders ?

CodePudding user response：

I would use AWS CLI for this. To get started - have a look here.

Then it is a matter of almost standard linux commands (ls):

aws s3 ls s3://<bucket_name>/path/to/search/folder/ --recursive | grep '/$' > folders.txt

where:

grep command just reads what aws s3 ls command has returned and searches for entries with ending /.

ending > folders.txt saves output to a file.

Note: grep (if I'm not wrong) is unix only utility command. But I believe, you can achieve this on windows as well.

Note 2: depending on the number of files there this operation might (will) take a while.

Note 3: usually in systems like AWS S3, term folder is there only for user to maintain visual similarity with standard file systems however inside it does treat it as a part of a key. You can see in your (web) console when you filter by "prefix".

CodePudding user response：

Amazon S3 buckets with large quantities of objects are very difficult to use. The API calls that list bucket contents are limited to returning 1000 objects per API call. While it is possible to request 'folders' (by using Delimiter='/' and looking at CommonPrefixes), this would take repeated calls to obtain the hierarchy.

Instead, I would recommend using Amazon S3 Inventory, which can provide a daily or weekly CSV file listing all objects. You can then play with that CSV file from code (or possibly Excel? Might be too big?) to obtain your desired listings.

Just be aware that doing anything on that bucket will not be fast.