Upload Image URL To S3 With Buffering Process-CodePudding

I found many questions related to uploading image to S3, but I haven't found a post that can help me fully. So, I decided to ask my problem.

I have the case to upload image from the URL to S3, so what I did is just download the image from URL first, and then upload to S3.

// download image form url
resp, _ := http.Get("https://sample-videos.com/img/Sample-jpg-image-50kb.jpg")
byteImage, _ = ioutil.ReadAll(resp.Body)

// upload to s3 (using s3manager)
_, err = s3ManagerClient.Upload(&s3manager.UploadInput{
    Bucket: aws.String(awsS3Bucket),
    Key:    aws.String(awsS3Destination),
    Body:   bytes.NewReader(byteImage), 
})

But, there is a problem with my code, the upload process must wait until the download process is completed. My goal is to upload image to s3 without waiting for the download process to complete, meaning that the download and upload process occurs simultaneously.

Here is my new code (handling error is removed for more readable)

resp, _ := http.Get("https://sample-videos.com/img/Sample-jpg-image-50kb.jpg")
    
// init s3
sess := session.Must(session.NewSession(&aws.Config{
    Region:      aws.String(awsRegion),
    Credentials: credentials.NewStaticCredentials(awsS3Key, awsS3Secret, ""),
}))
s3ManagerClient := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
    u.PartSize = int64(DefaultS3OptionPartSizeInMB)
    u.Concurrency = DefaultS3OptionConcurrency
})

buf := make([]byte, 1024) // per kb
for {
    n, _ := resp.Body.Read(buf)
    if n == 0 {
        break
    }
    _, err = s3ManagerClient.Upload(&s3manager.UploadInput{
        Bucket: aws.String(awsS3Bucket),
        Key:    aws.String(awsS3Destination),
        Body:   bytes.NewReader(buf[:n]), // upload image per part
    })
}

This code successfully uploads the image to S3, but the file is broken, I suspect that the image is not fully uploaded (there is something wrong with the process).

CodePudding user response：

Your problem is that the for-loop will upload the image in chunks of bytes onto the awsS3Destination. What you really are doing is you are replacing the contents of the image with individual chunks of bytes, one at a time.

You can try instead to pass the response body into the s3 uploader:

resp, err := http.Get("https://sample-videos.com/img/Sample-jpg-image-50kb.jpg")
if err != nil {
    // handle error
}

_, err = s3ManagerClient.Upload(&s3manager.UploadInput{
    Bucket: aws.String(awsS3Bucket),
    Key:    aws.String(awsS3Destination),
    Body:   resp.Body, // upload image 
})