0

i am using json2csv package for downloading my data it is working fine for small data. but if the records are more then 300 values it gets crashed. here is my code

const csvString = json2csv.parse(responseData);
    res.setHeader('Content-disposition', 'attachment; filename=shifts-report.csv');
    res.set('Content-Type', 'text/csv');
    res.status(200).send(csvString);

this code is working perfectly fine on small data how can i stream data when there is large amount of data using the same approach that i followed.

i am trying something like this but it gives me an error that cannot set the headers.

const headers = {
      'Content-type': 'text/csv',
      'Transfer-Encoding': 'chunked',
      'Content-Disposition': 'attachment; filename="file.csv"'
    };
    res.writeHead(200, headers);
    res.flushHeaders();
    const stream = new Writable({
      write(chunk, encoding, callback) {
        res.write(chunk);
        callback();
      }
    });
try {
        stream.write(file, 'utf-8');
      } catch (error) {
        console.log('error', error);
      }
    }
    res.end();
Ryan M
  • 18,333
  • 31
  • 67
  • 74
Ihtisham Tanveer
  • 338
  • 4
  • 15
  • Please provide some sample data and its intended output. There may be optimisations that are possible based on the data being parsed. – samthecodingman Jun 08 '21 at 12:30
  • here is the dummy data formate that i am passing to download. the data is downloading perfectly i am trying to download this using chunk by chunk i posted below the approach. responseData = [ { "car": "Audi", "price": 40000, "color": "blue" }, { "car": "BMW", "price": 35000, "color": "black" }, { "car": "Porsche", "price": 60000, "color": "green" } ]; – Ihtisham Tanveer Jun 08 '21 at 12:32

1 Answers1

0

You should use Json2CSV stream instead.

npm install json2csv-stream

const fs = require('fs');
const MyStream = require('json2csv-stream');


// create the one-time-use transform stream
const parser = new MyStream();

// create the read and write streams
const reader = fs.createReadStream('data.json');
const writer = fs.createWriteStream('out.csv');

//You can use writer to write it to the FS
// or can stream the content to the response object
// however, you need to write this code in any route handler
// which is sending the CSV data back to the user
reader.pipe(parser).pipe(res);

//reader.pipe(parser).pipe(writer);

For more details check here.

Apoorva Chikara
  • 8,277
  • 3
  • 20
  • 35