NodeJS - Wait Until Streaming Multiple Files Is Complete Before Continuing Code-CodePudding

I'm new to Javascript and NodeJS. Im trying to read multiple CSV files before doing some processing on them. My current issue is when I run the code it tries to execute the processing before the reading of the file is complete. I would like to load both the CSVs before I start doing any processing on them.

Could some explain why this happens and how I can solve the problem in Javascript/NodeJS.

function readCSV(path){

    var events = []
    
    fs.createReadStream(path).pipe(csv()).on('data', (row) => {
        events.push(row);
    }).on('end', () => {
        console.log('CSV file successfully processed. Length: '   events.length);
    }); 

    return events
}

function app(){

    var file_list = listFiles(folder_path);

    for (let i = 0; i < file_list.length; i  ) {
        const file = file_list[i];
        var events = readCSV(file)
    }
    
    processCSV(events) // Some processing
}

app();

Any help would be great and any explanation on how I can control when code is executed would be much appreciated.

CodePudding user response：

Sorry, you code cannot be compiled, so a can answer only with untested code

My current issue is when I run the code it tries to execute the processing before the reading of the file is complete.

The main problem that fs.createReadStream don't read file, - it request file system to start reading and inform your callbacks when some chunks was read so event 'end' will be called match later, when readCSV will be completed and returned empty result

You code was written as if you expect synchronous answer, and you can make it work correctly with use of sync methods like fs.readFileSync

How to fix it in asynchronous way? write csv processing in "on(end)" callback or Use promises

Promises are match more simple and linear

First make readCSV to return Promise

function readCSV(path: string){ //return Promise<any[]> 
  return new Promise((resolve) => {
    var events = [];
    fs.createReadStream(path)
      .pipe(csv())
      .on('data', (row) => {
        // this code called in future
        events.push(row);
    }).on('end', () => {
        // this code called in future to, 
        console.log('CSV file successfully processed. Length: '   events.length);
        resolve(events); //return csv parsed result
    }); 
  })
}

then in main app, use Promise.all to wait all fileReading promises

function app(){
  // i don't know what is listFiles,
  // i hope it returns sync result
  var file_list = fs.listFiles(folder_path);

  const dataPromises: Promise[] = []
  for (let i = 0; i < file_list.length; i  ) {
      const file = file_list[i];
      //launch reading
      dataPromises.push(readCSV(file))
  }
  Promise.all(dataPromises).then(result => {
    //this code will be called in future after all readCSV Promises call resolve(..)
    for(const events of result){
      processCSV(events);    
    }
  })
}