best way to load 100mb json file nodejs-CodePudding

I have a system that generates data every day And saves the data in the json file. The file about 120MB

I'm trying to send the data with nodejs to the client

router.get('/getData',(req, res) => {

    const newData = require(`./newDataJson.json`)
    res.json(newData)`});

and then from the client i use axios get request

    const fetchNewData = async () => {
const { data } = await axios.get(`/api/getData/`);}

The data reaches the client within about 3 minutes in production. and few second in localhost.

My question is if it possible to Shorten the load time in production.

thanks !!

CodePudding user response：

I would suggest you to use stream in Node.js that is suitable to send the large data from server to client. Stream are useful when you send big chunk of data. You should give it a try and check if there are any improvements after adding this.

const readStream = fs.createReadStream('./newDataJson.json');
return response.headers({
  'Content-Type': 'application/json',
  'Content-Disposition': 'attachment; filename="newDataJson.json"',
}).send(readStream);

Also, as Andy suggested this could be a way to divide your file data in smaller partition based on hours.

CodePudding user response：

120 MB is way over the limits of initial page loading times. The best thing to do would be to split the file into smaller chunks:

1. Right after that the user will see the loaded page you need (as I assume) only portion of that data. So initially send only small chunk to make the data visible. Keep it small, so the data would not block loading and first draw.

2. Keep sending rest of the data in smaller parts or load them on demand in chunks e.g. with pagination or scroll events.

You can start splitting the data upon receiving/saving them.

Loading data from api on scroll (infinite scroll) https://kennethscoggins.medium.com/using-the-infinite-scrolling-method-to-fetch-api-data-in-reactjs-c008b2b3a8b9

Info on loading time vs response time https://www.pingdom.com/blog/page-load-time-vs-response-time-what-is-the-difference/