Memory leak using node, fs and readline

Question

I'm trying to pass a huge csv data to sql and at half transaction it fails by memory leak: the error is exactly 'JavaScript heap out of memory'. Can you see anything that smells bad? Thank you in advance.

const readline = require('readline');
const fs = require('fs');

const postFromSarenetCSV = async (rq, res = response) => {
 try {
 const rl = readline.createInterface({
 input: fs.createReadStream('data/unparsedSarenet.csv',"UTF8"),
 crlfDelay: Infinity
 });
 await rl.on('line', async (line) => {
 //create object with papaparse library:
 let object = "column1,column2,column3\r\n"
 object = object.concat(line)
 let json = Papa.parse(object, {newline: "\r\n", header: true})
 //create sql request:
 var data = json.data[0]
 let {query, values} = createReplaceQuery('services', Object.entries(data))
 await dbConnection.query(query, values);
 });
 await events.once(rl, 'close');
 res.status(200).json({
 ok: true,
 msg: 'Database updated'
 })
 } catch (err) {
 console.error(err);
 }
}

I've tried to use await at readline listener thinking that may the accumulation of readed lines create the memory leak, but do nothing to solve the problem. When i did logs to see the last one, the last one has been before papaparse.

How huge is 'huge'. Are you talking 1000s of lines or 1000000 of lines. — Yesthe Cia, Nov 24 '22 at 16:10
sorry about this information fault. 7 milion of lines. The error comes over the 20.000 line. I'ts solved, it was the listener of readline. — Alejandro Fabra Segarra, Nov 25 '22 at 12:32

score 0 · Answer 1 · answered Nov 25 '22 at 12:29

0

I've found the problem. That's the listener. The listener doesn't wait to finalize his function. I've changed this line:

rl.on('line', async (line) => {

to this:

for await (const line of rl) {

answered Nov 25 '22 at 12:29

Alejandro Fabra Segarra

1
3

Memory leak using node, fs and readline

1 Answers1