PapaParse stream output to file

Question

const stream = require('stream');
const Transform = stream.Transform
const myTransform = new Transform({
  transform: (chunk, encoding, done)=>{
    console.log(`Received chunk for transform`)
    console.log(chunk)
    let result = chunk.join(",").toUpperCase()
    done(null, result)
  }
})
let papa = require('papaparse')
let file_stream = stream.Readable.from(Buffer.from("A,B,C\n1,2,3"));
let file_output_stream = fs.createWriteStream('final.csv');
file_stream.
  on("data", chunk => { console.log(`Read data`); console.log(chunk) }).
  pipe(papa.parse(papa.NODE_STREAM_INPUT, {})).
  on("data", row => {
    console.log(`Got row`)
    console.log(row)
  }).
  pipe(myTransform).
  pipe(file_output_stream)

The file seems to be read OK, but it doesn't seem my transform is being called at all. papaparse is throwing an exception:

TypeError [ERR_INVALID_ARG_TYPE]: The "chunk" argument must be of type string or an instance of Buffer or Uint8Array. Received an instance of Array

I want final.csv to contain the original CSV, but in all capitals. I need to do this streaming as the original file is quite large

Answer 1

As a workaround, I went with a different CSV module: how to use read and write stream of csv-parse

papaparse may be a perfectly viable solution, but I wasn't able to figure out how to use it to do what I needed

Answer 2

All you need to do is add objectMode: true to your Transform constructor.

This is because papaparse outputs an object, instead of a string or other data type normally expected by the transform stream.

like this:

const myTransform = new Transform({
  objectMode: true,
  transform: (chunk, encoding, done)=>{
    console.log(`Received chunk for transform`)
    console.log(chunk)
    let result = chunk.join(",").toUpperCase()
    done(null, result)
  }
})

I had a similar issue and found someone had answered this here which fixed it for me, so hopefully it does the same for you.