How to sequentially read a csv file with node.js (using stream API)

Question

I am trying to figure out how to create a stream pipe which reads entries in a csv file on-demand. To do so, I thought of using the following approach using pipes (pseudocode)

const stream_pipe = input_file_stream.pipe(csv_parser)
// Then getting entries through:
let entry = stream_pipe.read()

Unfortunately, after lots of testing I figured that them moment I set up the pipe, it is automatically consumed until the end of the csv file. I tried to pause it on creation by appending .pause() at the end, but it seems to not have any effect.

Here's my current code. I am using the csv_parse library (part of the bigger csv package):

// Read file stream
const file_stream = fs.createReadStream("filename.csv")
const parser = csvParser({
    columns: ['a', 'b'],
    on_record: (record) => {
        // A simple filter as I am interested only in numeric entries
        let a = parseInt(record.a)
        let b = parseInt(record.b)
        return (isNaN(a) || isNaN(b)) ? undefined : record
    }
})
const reader = stream.pipe(parser) // Adding .pause() seems to have no effect
console.log(reader.read()) // Prints `null`

// I found out I can use this strategy to read a few entries immediately, but I cannot break out of it and then resume as the stream will automatically be consumed 
//for await (const record of reader) {
//    console.log(record)
//}

I have been banging my head on this for a while and I could not find easy solutions on both the csv package and node official documentation.

Thanks in advance to anyone able to put me on the right track :)

score 0 · Answer 1 · answered Dec 10 '21 at 15:27

You can do one thing while reading the stream you can create a readLineInterface and pass the input stream and normal output stream like this:

const inputStream = "reading the csv file",
      outputStream = new stream();

// now create a readLineInterface which will read 
// line by line you should use async/await 

const res = await processRecord(readline.createInterface(inputStream, outputStream));

async function processRecord(line) {
   return new Promise((res, rej) => {
       if (line) {
        // do the processing 
        res(line);
       }
       
       rej('Unable to process record');
   })
}

Now create processRecord function should get the things line by line and you can you promises to make it sequential.

Note: the above code is a pseudo code just to give you an idea if things work because I have been doing same in my project to read the csv file line and line and it works fine.

How to sequentially read a csv file with node.js (using stream API)

1 Answers1