Home > Mobile >  How to load JSON data (call from API) without key directly to S3 bucket using Python?
How to load JSON data (call from API) without key directly to S3 bucket using Python?

Time:05-22

I am relatively new to AWS s3 I am calling an API to load the JSON data directly to s3 bucket. From s3 bucket data will be read by Snowflake. After researching I found that using Boto3 we can load data into s3 directly. Code will look something like below, however one thing I am not sure about is What should I put for the key as there is no object created in my S3 bucket. Also, what is the good practice to load the JSON data to s3 ? Do I need to encode JSON data to 'UTF-8' as done here by SO user Uwe Bretschneider.

Thanks in advance!

Python code:

import json,urllib.request
import boto3
data = urllib.request.urlopen("https://api.github.com/users?since=100").read()
output = json.loads(data) 
print (output) #Checking the data


s3 = boto3.client('s3')
s3.put_object(
     Body=str(json.dumps(data))
     Bucket='I_HAVE_BUCKET_NAME'
     Key='your_key_here'
)

CodePudding user response:

By using put_object, which means you are creating a new object in the bucket, so there is no existing key. This key is just like a file name in the file system. You can specify whatever names you like, such as my-data.json or some-dir/my-data.json. You can find out more in https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html.

As for encoder, it's always good to specify the encoding IMO, just to make sure your source file has properly encoded too.

  • Related