fetch big-json - How to return a json stream?-CodePudding

In my app I need to fetch a big json (>600MB) - I'd like to be able to fetch it from my api or from the file on my disk depending on the settings - unfortunately I cannot do JSON.parse because I get an exception that my json string is bigger than allowed string size. Because of that I'd like to use big-json library to handle this.

I have my method getMyDataStream which depending on the input would either read the json from the file or fetch it from my API but I have no idea how to return a json response body from the fetch as a stream to be able to be piped to big-json.createParseStream()

import fetch from 'node-fetch';

const json = require('big-json');

const myDataStream = getMyDataStream(mySettings);
const parseStream = json.createParseStream();

parseStream.on('data', function(pojo) {
    // => rest of the logic
});

myDataStream.pipe(parseStream);


const getMyDataStream = ({queryId, myFile, myApiEndpoint}) => {
  if (myFile)
    return fs.createReadStream('myFile')
  else
    return getDataFromAPI({queryId, myApiEndpoint});
}

const getDataFromApi = ({queryId, myApiEndpoint}) => {
  return fetch(
    `${myApiEndpoint}/queries/${queryId}`,
    {
      method: 'GET',
      compress: true,
    }
  )
. what to do next to return its response body as a stream ?
 ??????

Thanks for help

CodePudding user response：

node-fetch resolves with Response object that implements the Body interface. This has a property body that is a ReadableStream

const getDataFromApi = async ({ queryId, myApiEndpoint }) => {
  const { body, ok, status, statusText } = await fetch(
    `${myApiEndpoint}/queries/${queryId}`,
    {
      method: "GET",
      compress: true,
    }
  );

  if (!ok) {
    throw new Error(`Request failed: ${status} ${statusText}`);
  }

  return body;
};

Note that this is all asynchronous so you'd need to use it like this

const getMyDataStream = ({ queryId, myFile, myApiEndpoint }) => {
  if (myFile) {
    // always return a promise for a consistent API
    return Promise.resolve(fs.createReadStream("myFile"));
  } else {
    return getDataFromAPI({ queryId, myApiEndpoint });
  }
};

const parseStream = json.createParseStream();

parseStream.on("data", function (pojo) {
  // => rest of the logic
});

// Resolve the stream and pipe to the parser
getMyDataStream(mySettings).then((myDataStream) =>
  myDataStream.pipe(parseStream)
);