Saving data from a CSV in a global variable (fast-csv/papaparse)

Question

I am new to js and trying to parse a CSV in the backend using node.js.

I have an array of states in which I want to store the data of a column of CSV. This is a very simple code that i wrote using fast-csv to do so. But whenever I run the code, I get an empty array, [] . I tried doing the same using papaparse and got the same results.

const csv = require('fast-csv')
const file = fs.createReadStream('main.csv');
var states = []
file
    .pipe(csv.parse({ headers: false }))
    .on('data', row => states.push(row[2]))
console.log(states)

But whenever I console log it in then .on('end') block the values are logged.

const csv = require('fast-csv')
const file = fs.createReadStream('main.csv');
var states = []
file
    .pipe(csv.parse({ headers: false }))
    .on('data', row => states.push(row[2]))
    .on('end', () => console.log(states) // This works
console.log(states) // This doesn't

I think this is due to the parser working asynchronously. I have tried to resolve promises and used async/await methods but I cant use the parsed content in the global scope.

Would love some help on this one.

score 2 · Accepted Answer · answered Jun 24 '21 at 13:28

You are right, parser works asynchronously. So, it starts parsing and uses callback on events ('data', 'end' in your case). But code after your parser will be executed emediately after parser started to work. So all your actions with parsed data should be done in 'end' event callback.

// function to start parser
const startParsing = (res) => {
  const csv = require('fast-csv')
  const file = fs.createReadStream('main.csv');
  // you may use const cos it's type won't be changed
  const states = []
  file
    .pipe(csv.parse({ headers: false }))
    .on('data', row => states.push(row[2]))
    // execute function after parsing.
    .on('end', () => outputData(states, res));
};

const outputData = (states, res) => {
  // your next actions here
  console.log(states);
  // for example, res.send(states) or anything to complete server response if it's accessible
  res.send(states);  
};

// then, in any place you need use this startParsing function, for example
server.get('/parser', (req, res) => {
  startParsing(res);
});

Saving data from a CSV in a global variable (fast-csv/papaparse)

1 Answers1