I am relatively new to AWS s3
I am calling an API to load the JSON data directly to s3
bucket. From s3
bucket data will be read by Snowflake
. After researching I found that using Boto3
we can load data into s3
directly. Code will look something like below, however one thing I am not sure about is What should I put for the key as there is no object created in my S3
bucket. Also, what is the good practice to load the JSON data to s3
? Do I need to encode JSON data to 'UTF-8' as done here by SO user Uwe Bretschneider.
Thanks in advance!
Python code:
import json,urllib.request
import boto3
data = urllib.request.urlopen("https://api.github.com/users?since=100").read()
output = json.loads(data)
print (output) #Checking the data
s3 = boto3.client('s3')
s3.put_object(
Body=str(json.dumps(data))
Bucket='I_HAVE_BUCKET_NAME'
Key='your_key_here'
)
CodePudding user response:
By using put_object
, which means you are creating a new object in the bucket, so there is no existing key.
This key is just like a file name in the file system. You can specify whatever names you like, such as my-data.json
or some-dir/my-data.json
. You can find out more in https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html.
As for encoder, it's always good to specify the encoding IMO, just to make sure your source file has properly encoded too.