0

ISSUE: I am trying to use Nodejs streams to read a small CSV file (1 row) using the fast-csv module.

The CSV 'rows' are pushed to an array(rows []) when the 'data' event is emitted. When 'end' is emitted, the data is update in a DB. However, the 'end' event is triggered before the rows[] array can be populated. This happens intermittently and sometimes the code works as intended.

My guess after reading the Nodejs docs is that this is due to the small size of the CSV file. The data is being read in the 'flowing' mode and as soon as the the first row is read, the 'end' even is triggered, which seems to happen before the record is pushed to the required array.

Tried using the 'paused' mode, but it didn't work.

I am new with Nodejs and not able to figure out how to make this function work. Any help or guidance would be appreciated.

CODE:

function updateToDb(filename, tempLocation) {
    const rows = [];

    const readStream = fs.createReadStream(tempLocation + '\\' + filename).pipe(csv.parse());
    return new Promise((resolve, reject) => {
        readStream.on('data', row => {
                console.log('Reading');
                rows.push(row);
            })
            .on('end', () => {
                console.log('Completed');

                let query = `UPDATE ${tables.earnings} SET result_date = CASE `;
                rows.forEach(element => {
                    query += `WHEN isin = '${element[0]}' AND announcement_date = '${element[1]}' THEN '${element[2]}' ELSE result_date`;
                });
                query += ' END';

                connection.query(query, (error, results) => {
                    if (error)
                        reject(error);
                    else
                        resolve(results.changedRows);
                });
            })
            .on('error', error => {
                reject(error);
            });
    });
}
  • 1
    i tried this code and this seems to work fine. I have tried multiple times and it seems to work ok. – Ashish Modi Feb 26 '20 at 13:52
  • you could try making your promise asynchronous instead and see if that does anything. – Michael Feb 26 '20 at 14:07
  • @AshishModi thanks for the tip! I also tried the code in isolation and it worked every time. I will have to dig deeper in the other parts of the code. Thanks for the help. – Rusty Banks Feb 27 '20 at 02:36

0 Answers0