Description

I am building a music streaming SPA using NEXT.js.
Audio files are stored on AWS S3.

The goal is to stream audio file from S3 to client through REST
so that authentication is possible, and to "hide" the AWS endpoints.

Issue

When streaming data down to client through REST endpoint the audio glitches and loads only ~15seconds of the audio file being played.

I tested this behaviour on separate project with manually creating the read stream and providing options to it :

fs.createReadStream("path", {start: startByte, end: endByte}) 

and it works just fine.

Although the createReadStream from s3 (i believe im using v2) does not accept any options. So i am unable to fix this glitching this way.

I thought about many solutions one of which involved manually converting the incoming buffer from S3 to streamable data, but this will lead to data being processed in server's RAM i believe, and i dont want this behaviour even though audio files are usually quite "small".
I also thought about creating a presigned url to the file and then redirecting in the worst case scenario.

Question

I will provide source code below. I believe my audio loops on first ~15 seconds due to readstream lacking start and end positions.
How do i fix given behaviour and stream data corectly from s3 to server to client without saving whole files in servers RAM?

Code

Part of the utility function for data streaming:

const downloadParams = {
        Key,
        Bucket: bucketName,
    };
const fileStream = s3.getObject(downloadParams).createReadStream();

fileStream is returned from this function and accessed in API endpoint like so:

const CHUNK_SIZE = 10 ** 3 * 500; // ~500KB
const startByte = Number(range.replace(/\D/g, ""));
const endByte = Math.min(
    startByte   CHUNK_SIZE,
    attr.ObjectSize - 1
);
const chunk = endByte - startByte   1;

const headers = {
    "Content-Range": `bytes ${startByte}-${endByte}/${attr.ObjectSize}`,
    "Accept-Ranges": "bytes",
    "Content-Length": chunk,
    "Content-Type": "audio/*",
};

res.writeHead(206, headers);

fileStream.pipe(res);

Here is audio receiver on the client:

"use client";

const Audio = () => {
    return (
        <audio src="http://localhost:3000/api/stream/FILE_KEY_HERE" controls></audio>
    );
};

export default Audio;

here is how request headers look like:

Accept: */*
Accept-Encoding: identity;q=1, *;q=0
Accept-Language: en,ru;q=0.9,sv-SE;q=0.8,sv;q=0.7,en-US;q=0.6
Connection: keep-alive
Cookie:
Host: localhost:3000
Range: bytes=65536-
Referer: http://localhost:3000/
sec-ch-ua: "Not_A Brand";v="99", "Google Chrome";v="109", "Chromium";v="109"
sec-ch-ua-mobile: ?1
sec-ch-ua-platform: "Android"
Sec-Fetch-Dest: video
Sec-Fetch-Mode: no-cors
Sec-Fetch-Site: same-origin
sec-gpc: 1
User-Agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36

second sheader differs only by Range: bytes=65536-

first request:

Request URL: http://localhost:3000/api/stream/track/4
Request Method: GET
Status Code: 206 Partial Content
Remote Address: [::1]:3000
Referrer Policy: strict-origin-when-cross-origin

Response Headers:

Accept-Ranges: bytes
Connection: keep-alive
Content-Length: 500001
Content-Range: bytes 65536-565536/3523394
Content-Type: audio/*
Date: Wed, 25 Jan 2023 21:35:51 GMT
Keep-Alive: timeout=5

PS

I did check my network tab and headers contain thruthful information about objects that are being streamed. Requests seem to download full file size (3.2mb of 3.2mb for example), but the audio still loops on first 15 seconds. Even if i manipulate the duration bar manually.
Haven't found any information like this here so thought this would be helpful to someone in the future

Tried

On top of things mentioned I tried creating new streams and piping them, tried using stream events on createReadStream(), read poorly written aws docs. But due to lack of info it is less time consuming to ask someone than trying fixing same issue for 4 days straight.

CodePudding user response：

The issue is that the first X bytes were read from the source MP3, regardless of if the client requested a later 'range'.

The quick solution was to just tell the GetObject function to seek to the same bytes the request states in the Range header, since S3 itself also supports range requests.