JavaScript hebrew CSV parsing issue-CodePudding

so i got a csv file saves as CSV -UTF-8 file from excel on the right its in visual studio on the left its in excel.

when im trying to parse with PapaParse/csv-parser, i still see the output in <?> format which it means encoding : PapaParse try:

 papa.parse(file, {
    worker: true, 
    step: function (result) {
       count  ; 
       console.log(result.data[0])
    },
    complete: function (results, file) {//
       console.log('parsing complete read', count, 'records.');
    }
});

csv-reader try:

const csv = require('csv-parser');
const results = [];
fs.createReadStream('Cities.csv')
   .pipe(csv())
   .on('data', function (datarow) {
       results.push(datarow);
   })
   .on('end', function () {
       console.log(results.toString('utf-8");
   });

In both situations the output is in <?> cant read hebrew chars

CodePudding user response：

Looks to me like you're piping the raw chunk buffers to csv, not the result of converting to string using UTF-8 encoding. You aren't telling createReadStream to handle any encoding work for you, so it will read raw data and pass them to the csv function (since you're piping to it) as Buffer instances.

Rather than reading raw data and converting afterward, tell createReadStream you want it to handle conversion for you via its options parameter:

const csv = require('csv-parser');
const results = [];
fs.createReadStream('Cities.csv', 'utf-8')
// −−−−−−−−−−−−−−−−−−−−−−−−−−−−−^^^^^^^^^
   .pipe(csv())
   .on('data', function (datarow) {
       results.push(datarow);
   })
   .on('end', function () {
       // Use `results` here (it's an array of objects according to the
       // csv-parser documentation; calling `toString` on it probably isn't
       // what you want)
   });

CodePudding user response：

You can try to add this:

format:"UTF-8"

   papa.parse(file, {
        worker: true, 
        step: function (result) {
           count  ; 
           console.log(result.data[0])
        },
       format: "UTF-8"
        complete: function (results, file) {//
           console.log('parsing complete read', count, 'records.');
        }
    });

CodePudding user response：

The problem was not because of the code, but because of the console properties. I had to change the font so that the console can display the Hebrew letters correctly.