Read content of a file located under subfolders of S3 in Python-CodePudding

I'm trying to read a file content's (not to download it) from an S3 bucket. The problem is that the file is located under a multi-level folder. For instance, the full path could be s3://s3-bucket/folder-1/folder-2/my_file.json. How can I get that specific file instead of using my iterative approach that lists all objects?

Here is the code that I want to change:

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('s3-bucket')
for obj in my_bucket.objects.all():
   key = obj.key
   if key == 'folder-1/folder-2/my_file.json':
      return obj.get()['Body'].read()

Can it be done in a simpler, more direct way?

CodePudding user response：

Yes - there is no need to enumerate the bucket.

Read the file directly using s3.Object, providing the bucket name as the 1st parameter & the object key as the 2nd parameter.

"Folders" don't really exist in S3 - Amazon S3 doesn't use hierarchy to organize its objects and files. For the sake of organizational simplicity, the Amazon S3 console shows "folders" as a means of grouping objects but they are ultimately baked into your object key.

This should work:

import boto3
s3 = boto3.resource('s3')
obj = s3.Object("s3-bucket", "folder-1/folder-2/my_file.json")
body = obj.get()['Body'].read()