I need to read huge CSV files and I need to stop conditionally (after I've processed 50 lines). I create a readable stream from the CSV file and pipe it to a csv parser

const fileStream = fs.createReadStream(FilePath);
const pipeFunction = parse({ delimiter: ",", from_line });
fileStream.pipe(pipeFunction).on("data", function (data) => {
  // Do operations here
  console.log(counter);
  if (counter===50) {
     fileStream.destroy();
     return resolve(processedData);
  }
  counter  ;
})

My problem is that the console.log prints 50 a lot of times which made me realize I probably don't stop the stream the way it should be done.

CodePudding user response：

I suggest using the readLine interface from Node.js where you can read the lines from the CSV without any need for a CSV parser if the file is already CSV.

A good example from Node.js documentation:

const fs = require('node:fs');
const readline = require('node:readline');
let lineCounted = 0;
async function processLineByLine() {
  const fileStream = fs.createReadStream('input.csv');

  const rl = readline.createInterface({
    input: fileStream,
    crlfDelay: Infinity
  });
  // Note: we use the crlfDelay option to recognize all instances of CR LF
  // ('\r\n') in input.txt as a single line break.

  for await (const line of rl) {
    // Each line in input.txt will be successively available here as `line`.
    lineCounted  = 1;
    console.log(`Line from file: ${line} with line number: ${lineCounted}`);
  }

  if (lineCounted === 50) rl.close();
}

processLineByLine();

Note: you can check here for more details.