I am copying files from a website to a S3 bucket. Everything else is copying fine, even odd extensions that I haven't heard of before. The extension that I am having problems with is ".2D".
Currently using this code, and it is working for all but the .2D files. Might be a VERSACAD file. Anyone work with this file or know how to figure out how to work with this? No, I can't include an example.
It is failing on the r.data.decode("utf'8") line. Using "utf-16" doesn't work either.
data=r.data.decode("utf-8")
key_path="downloaded_docs/{0}/{1}/{2}/{3}".format(year,str(month).zfill(2),str(day).zfill(2),docname)
To save to s3 bucket:
s3.Object('s3_bucket_name',key_path).put(Body=data)
WHAT WORKED I ended up having to do the following, but it worked. Thanks!!
if extracted_link.endswith('.2D'):
r=http.request("GET", url_to_file, preload_content=False)
# to get binary data
text_b=r.data
filepath=<filepath>
s3_client.upload_fileobj(io.BytesIO(text_b),Bucket=<bucketname>,Key=filepath)
CodePudding user response:
Upload it in binary
from file path
import boto3
client = boto3.client("s3")
client.upload_file(2D_filepath, bucket, key)
from file object
import boto3, io
client = boto3.client("s3")
client.upload_fileobj(io.BytesIO(b"binary data"), bucket, key)