0

I'm processing csv file data coming from request to create json object using multer,csv-parser. but can't process from long time. please help and thanks.following is example of csv file processing.

const multer = require('multer');
const fs = require('fs')
const csv =  require('csv-parser')
const fileStorageEngine = multer.memoryStorage({
  destination: (req, file, cb) => {
    cb(null, './csv');
  }
  ,
  filename: (req, file, cb) => {
    cb(null, file.originalname);
  },
});
const upload = multer({storage:fileStorageEngine});
app.post('/uploadcsv',upload.single("upfile"),async(req,res)=>{
const file = req.file;
fs.createReadStream(`./csv/${file.originalname}`)
.pipe(csv())
.on("data",(data)=>console.log(data));
res.send("file uploaded")
})

here I'm using fs module that works for me but I don't want to store data in file instead I want to process buffer data coming from req.file.buffer in chunk I'm stuck here please help. because storing file and reading same file getting process slow because csv file have thousands of data.

evolutionxbox
  • 3,932
  • 6
  • 34
  • 51
omkar p
  • 43
  • 1
  • 8
  • When you say "JSON object", please clarify if you mean a JS object or JSON (a string)? – evolutionxbox May 03 '22 at 08:38
  • If you don't want to store the uploaded file on the harddisk, why not remove all the code that does that? `memoryStorage()` has no options and returns a Buffer. –  May 03 '22 at 08:43

1 Answers1

1

The idea is to create a readable stream from the buffer instead of writing it to the file as follows:

const multer = require('multer');
const { Readable } = require('stream');
const fs = require('fs')
const csv =  require('csv-parser')
const fileStorageEngine = multer.memoryStorage({
  destination: (req, file, cb) => {
    cb(null, './csv');
  }
  ,
  filename: (req, file, cb) => {
    cb(null, file.originalname);
  },
});
const upload = multer({storage:fileStorageEngine});
app.post('/uploadcsv',upload.single("upfile"),async(req,res)=>{
const file = req.file;

const stream = Readable.from(file.buffer);

stream.pipe(csv()).on("data",(data)=>console.log(data));
res.send("file uploaded")
})
Sathya
  • 431
  • 3
  • 7