0

I need to read a csv file from an external location. Then I need to parse each line and insert them one by one in the DB. Once all is done, I need to do some other DB operations so I need to wait for the parsing/processing is done. But in my code example below, in the .on('data', ...), the 'await' does not wait and the process finishes before all the records are processed. Can you help me to understand what I'm doing wrong? Thanks!

import { promisify } from 'util'; 
import * as stream from 'stream'; 
import csvParser from 'csv-parser'; 
import axios from 'axios';

async function newFunc(url: string) {
  console.log('START newFunc')
  const userService = new UserService()

  const axiosResponse = (async () => {
    const finishedDownload = promisify(stream.finished);
    const csv = csvParser({separator: '|'}) //fs.createWriteStream('/path/to/saved/file');
  
    const response = await axios({
      method: 'GET',
      url: url,
      responseType: 'stream',
    });

    await response.data
        .pipe(csv)
        .on('data', async (userBeneRequest: IUserBeneficiaryBulkStandardRequest) => {
            const a = await userService.getUserById(userBeneRequest.email!) // this is just an example of a DB operation
            console.log('got the user', a)
        })

    await finishedDownload(csv);
  })();

  console.log('FINISHED newFunc')
  return axiosResponse

}

Then in the logs I get:

START newFunc 
FINISHED newFunc 
got the user 1 
got the user 2 
got the user 3 
got the user 4 
got the user 5

As you can see, the 'got the user 1', etc is showing after the 'FINISHED newFunc' and not in the middle of it.

Toba
  • 61
  • 1
  • 3
  • This happened because you have little misunderstanding how EventEmitter works. Usage `.on('data')` is kinda similiar to event emitter. You registered async handler for data - its correct. But once you get data chunk - it will run in event loop (not in main context). That how events works. Probably you want to use something like `.on('finish')` event. Once finish event triggered - then you have to show `Finished newFunc`. E.g. how its done with stream: `stream.on('finish', function () { ... });` – Evgheni Calcutin Apr 05 '22 at 11:08

0 Answers0