I'm trying to upload an image to S3 from an external url.

Currently, I can successfully upload a file, but after I download and open it I see that it is corrupted.

Here is the code I'm using (got is just what I use to fetch the resource):

const got = require('got');
const Web3 = require('web3');

const s3 = new AWS.S3({
    accessKeyId: AWS_ACCESS_KEY_ID,
    secretAccessKey: AWS_SECRET_ACCESS_KEY,
});

const response = await got('https://example.com/image.jpg');

const uploadedFile = await s3
    .upload({
        Bucket: 'my_bucket',
        Key: 'images/',
        Body: response.body,
        ContentType: 'image/jpeg',
    })
    .promise();

I tried to create a buffer, and use putObject instead of upload, but I end up with with files that are only a few bytes on S3 instead.

CodePudding user response：

The request to get the object is converting it to a string. Pretty much whatever encoding you pick to do that will corrupt it, since a JPG is binary data not meant to be represented with an string's encoding.

The documentation for the got library states:

encoding

Type: string

Default: 'utf8'

Encoding to be used on setEncoding of the response data.

To get a Buffer, you need to set responseType to buffer instead. Don't set this option to null.

In other words, if you change your download to:

const response = await got('https://example.com/image.jpg', {'responseType': 'buffer'});

You'll get and upload a Buffer object without changing it by encoding it as a string.

CodePudding user response：

Your key is wrong when you are uploading the file to S3:

Key: 'images/'

It cannot be images/ because that would upload the image to an object that represents a folder. While that might work with the local file system on your Windows/Mac laptop, it doesn't work with object stores. It needs to be a key that represents a file, for example:

Key: 'images/image.jpg'