Home > Back-end >  Unable to PUT big file (2gb) to aws s3 bucket (nodejs) | RangeError: data is too long
Unable to PUT big file (2gb) to aws s3 bucket (nodejs) | RangeError: data is too long

Time:01-08

I scouted trough all of the internet and everybody gives out different advice but none of them helped me.

Im currently trying to simply send file.buffer that gets send to my endpoint directly to aws bucket.

im using PutObjectCommand have correctly entered all the details in but there's apparently problem with me using simple await s3.send(command) because my 2.2gbs video is way too big.

i get this error when attempting to upload said file to cloud.

RangeError: data is too long at Hash.update (node:internal/crypto/hash:113:22) at Hash.update (C:\Users\misop\Desktop\sebi\sebi-auth\node_modules\@aws-sdk\hash-node\dist-cjs\index.js:12:19) at getPayloadHash (C:\Users\misop\Desktop\sebi\sebi-auth\node_modules\@aws-sdk\signature-v4\dist-cjs\getPayloadHash.js:18:18) at SignatureV4.signRequest (C:\Users\misop\Desktop\sebi\sebi-auth\node_modules\@aws-sdk\signature-v4\dist-cjs\SignatureV4.js:96:71) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { code: 'ERR_OUT_OF_RANGE', '$metadata': { attempts: 1, totalRetryDelay: 0 } }

I browsed quite a lot,there's lots of people saying that i should be using presigned url,i did try however if i do await getSignedUrl(s3,putCommand,{expires:3600}); then i do get generated url but there's not PUT send to cloud. when i read little more into it getSignedUrl is just for generating signed url therefore there's no way for me to use Put command there so im not sure how to approach this situation.
Im currently working with :

"@aws-sdk/client-s3": "^3.238.0",
"@aws-sdk/s3-request-presigner": "^3.238.0",

enter image description here

Honestly i've been testing lots of different ways i saw online but i wasnt successful following even amazon's official documentation where they mention these thing and i trully dont want to implement multipart upload for smaller than 4 ~ 5gbs of videos.

I'll be honored to hear any advice on this topic, thank you.

Get advice on how to implement simple video upload to aws s3 because of my many failed attempts on doing so since there's lots of information and vast majority doesnt work.

CodePudding user response:

The solution to my problem was essentially using multer's s3 "addon" that had s3 property and had pre-done solution. "multer-s3": "^3.0.1" version worked even with file that have 5gbs and such. solutions such as using PutObject command inside presigned url method or presigned-post methods were unable to work with multer's file.buffer that node server receives after its being submitted.

If you experienced same problem and want quick and easy solution. use this Multer-s3 npm

  • Related