How to read very large (> 1GB) tar.gz files in Node.js?

Question

I have never had to do this before so this is probably something really basic, but I thought I'd ask anyways.

What is the right way to read a very large file in Node.js? Say the file is just too large to read all at once. Also say the file could come in as a .zip or .tar.gz format.

First question, is it best to decompress the file first and save it to disk (I'm using Stuffit on the Mac to do this now), and then work with that file? Or can you read the IO stream straight from the compressed .zip or .tar.gz version? I guess you'd need to know the format of the content in the compressed file, so you probably have to decompress (just found out this .tar.gz file is actually a .dat file)...

Then the main issue is, how do I read this large file in Node.js? Say it's a 1GB XML file, where should I look to get started in parsing it? (Not, how to parse XML, but if you're reading the large file line-by-line, how do you parse something like XML which needs to know the context of previous lines).

I have seen fs.createReadStream, but I'm afraid to mess around with it... don't want to explode my computer. Just looking for some pointers in the right direction.

How about, assume it's a very large CSV and I just want to create a database record for each line. — Lance, Jun 18 '12 at 02:44
You have two issues, 1. Is there a streaming zip file reader for Node, and 2. Is there a streaming XML reader (that can use the first stream as input). Not sure what options are out there but that might help you search... — Joe, Jun 18 '12 at 02:47

score 9 · Accepted Answer · edited Jun 18 '12 at 04:42

9

there is built-in zlib module for stream decompression and sax for stream XML parsing

var fs = require('fs');
var zlib = require('zlib');
var sax = require('sax');

var saxStream = sax.createStream();
// add your xml handlers here

fs.createReadStream('large.xml.gz').pipe(zlib.createUnzip()).pipe(saxStream);

edited Jun 18 '12 at 04:42

Casey Chu

25,069
10
40
59

answered Jun 18 '12 at 04:35

Andrey Sidorov

24,905
4
62
75

How should I go about if I want to zip a directory which contains GBs of data ? – Vaibhav Pachauri Apr 05 '16 at 06:41
do you need to process that zip immediately or just want to zip & save ? I'd probably just spawn tar + gz child command – Andrey Sidorov Apr 05 '16 at 07:54
I just need to zip the directory and save it. I am little new to Node.js, I am not really familiar with spawn tar+gz :\ – Vaibhav Pachauri Apr 05 '16 at 09:05
Got it. Posted an answer as well. Thanks Mate :) – Vaibhav Pachauri Apr 05 '16 at 09:41

Vaibhav Pachauri · Answer 2 · 2016-04-06T19:10:04.927

2

We can also zip the directory something like the following :

var spawn = require('child_process').spawn;
var pathToArchive = './very_large_folder.tar.gz';
var pathToFolder = './very_large_folder';

var tar = spawn('tar', ['czf', pathToArchive, pathToFolder]);
tar.on('exit', function (code) {
        if (code === 0) {
                console.log('completed successfully');
        } else {
                console.log('error');
        }
});

This worked nicely :)

edited Apr 06 '16 at 19:10

answered Apr 05 '16 at 09:41

Vaibhav Pachauri

2,671
2
19
32

oldie but goldie! – loretoparisi Nov 19 '21 at 23:37

How to read very large (> 1GB) tar.gz files in Node.js?

2 Answers2