1

Currently i am trying to create a CSV reader that can handle very large CSV files. I chose for a streaming implementation with the event-stream NPM package.

I have created a function getNextp() that should return a promise and give me the next piece of data every time i call it.

"use strict";
const fs = require('fs');
const es = require('event-stream');
const csv = require('csv-parser');

class CsvFileReader {
  constructor(file) {
    this.file = file;
    this.isStreamReading = false;
    this.stream = undefined;
  }

  getNextP() {
    return new Promise( (resolve) => {
      if (this.isStreamReading === true) {
          this.stream.resume();
      } else {
        this.isStreamReading = true;
  
        // Start reading the stream.
        this.stream = fs.createReadStream(this.file)
        .pipe(csv())
        .pipe(es.mapSync( (row) => {

          this.stream.pause();
          resolve(row);  
        }))
        .on('error', (err) => {
          console.error('Error while reading file.', err);
        })
        .on("end", () => {
          resolve(undefined);
        })
      }
    });
  }
}

I call this then with this code.

const csvFileReader = new CsvFileReader("small.csv");
setInterval( () => {
  csvFileReader.getNextP().then( (frame) => {
    console.log(frame);
  })
}, 1000);

However every time when i try this out i only get the first row and the subsequent rows i do not get. I can not figure out why this it not working. I have tried the same with a good old callback function and then it works without any problem.

Update: So what i basically want is a function (getNext()) that returns me every time when i call it the next row of the CSV. Some rows can be buffered, but yeah until now i could not figure out how to do this with streams. So if somebody could give me a pointer on how to create a correct getNext() function that would be great.

I would like to ask if somebody understands what is going wrong here, and ask kindly to share his/hers knowledge.

Thank you in advance.

wiegvlieg
  • 11
  • 2
  • 1
    How would you expect this to get the subsequent rows? `getNextP()` returns a promise that will be resolved exactly once, for the first row, and in further calls it never goes into the `else` and never calls `resolve()`. – Bergi Oct 29 '20 at 00:19
  • 1
    You really should [just use node streams as async iterables](https://stackoverflow.com/a/63753901/1048572). Much much simpler and better than anything hand-crafted. – Bergi Oct 29 '20 at 00:21

0 Answers0