I'm using a simple Node.js server to send a large JSON file to the client, where each line has a self-contained JSON object. I'd like to send this file to the client one line at a time. But my problem is that the server waits until response.end() has been called to send the whole thing at once.

The code for my server looks like this:

http.createServer(async function (request, response) {
   response.writeHead(200, {"Content-Type": "application/json; charset=UTF-8", "Transfer-Encoding": "chunked", "Cache-Control": "no-cache, no-store, must-revalidate", "Pragma": "no-cache", "Expires": 0});
   response.write(JSON.stringify('["The first bit of JSON content"]\n'));
   response.write(await thisFunctionTakesForever());
   response(end);
}

I really don't want to make the user wait until the entire JSON file has been loaded before my script can start parsing the results. How can I make my server send the data in chunks?

Additional info: How do I know my Node.js server isn't sending any part of the file until after response.end has been called?

I'm using XMLHttpRequest to handle the chunks as they arrive. I understand that http.responseText always grows with each chunk, so I filter through it to find the new lines that arrive each time:

let http = new XMLHttpRequest();
http.open('GET', url, true);
http.setRequestHeader('Content-type', 'application/x-www-form-urlencoded');
http.onreadystatechange = function() {
    if(http.readyState >= 3 && http.status == 200) {
        // Parse the data as it arrives, throwing out the ones we've already received
        // Only returning the new ones
        let json = http.responseText.trim().split(/[\n\r] /g)
        let dataChunks = json.map(e => JSON.parse(e));

        let newResults = [];
        for(let i=0; i<dataChunks.length; i  )
        {
            if(!previousResults.map(e => e[0]).includes(dataChunks[i][0]))
            {
                newResults.push(dataChunks[i]);
            }
        }
        previousResults = previousResults.concat(newResults);
    }
}
http.send();

The array previousResults should grow slowly over time. But instead, there's a huge delay, then everything suddenly appears all at once.

The following thread is related. But unfortunately, none of the proposed solutions solved my problem...

See the image:

1. Each PSH packet carries out a chunk.
2. There is a delay between each chunk transmission.

However, the HTTP client (like the browser) must accept all the chunks before handing them over to the application for the reasons: Once all the chunks are received, the server could also send some headers - trailer headers. Those headers include Content-MD5, Content-Length, etc. The client must verify like Content-MD5 once all chunks are received before handing them over to the application. I think that's why you can't receive chunks one by one on the browser side.