Node.js: How to process objects of a big JSON file one by one to avoid heap limit errors

Question

im trying to process a few hundreds of json.gz files using worker threads. at some point im getting js heap limit error due to 3 big files(about 3gb each unzipped).

i tried to find a way to process the objects of each file one by one, but all i've managed to get is all of the file's objects at once.

here is the worker code at the moment:

    for (let gzFile of zippedFiles) {
        const gunzip = zlib.createGunzip()
        const parser = JSONStream.parse('offers.*')
        const readStream = fs.createReadStream(gzFile)

        readStream.pipe(gunzip).pipe(parser)
            .pipe(es.map((offers, callback) => { //offers contains all of the current file objects array 
                offers.forEach(rawProduct => {
                    let processedProduct = getProcessedProduct(rawProduct)
                    parentPort.postMessage({ processedProduct })
                })
                callback()
            })
                .on('error', (e) => {
                    console.trace(`Error while reading file`, e)
                })
                .on('end', () => {
                    idxCount++
                    if (idxCount === lastIdx) {
                        parentPort.postMessage({ completed: true })
                    }
                })
            )
    }

jsons structure:

{
"offers":
    {
        "offer":
        [
            {}, // => the objects i wanna get one by one
            {},
            {}
        ]
    }
}

how can i avoid getting js heap limit error? thanks!

try to stream parse json - https://www.npmjs.com/package/stream-json — Nidhin David, Nov 09 '21 at 17:49

lightPT · Accepted Answer · 2022-10-15T18:03:01.230

Nidhim David suggestion is exactly what I was looking for.

here is the working code:

for (let gzFile of zippedFiles) {
        const pipeline = chain([
            fs.createReadStream(gzFile),
            zlib.createGunzip(),
            parser(),
            pick({ filter: 'offers.offer' }), //getting the array of objects
            streamArray(),
        ]);

        pipeline.on('data', ({key, value}) => {
            //getting objects one by one and processing them
            const rawProduct = value;
            const processedProduct = getProcessedProduct(rawProduct);
            parentPort.postMessage({ processedProduct });
        })

        pipeline.on('end', () => {
            idxCount++;
            if (idxCount === lastIdx) {
                debug(`last zipped file, sending complete message`);
                parentPort.postMessage({ completed: true });
            }
        });
}

Node.js: How to process objects of a big JSON file one by one to avoid heap limit errors

1 Answers1