Node Read large zip file from URL and then Unzip it

Question

I am reading a large zip file(500MB) from a URL using request.get(url).File contains one CSV file which is pretty huge in size. I am reading the response and writing the filestream into a zip file using fs.createWriteStream(zipFile). On close event of the fs.createWriteStream I have tried using adm-zip file with which i got "error invalid or unsupported zip format. no end header found" and with Unzipper npm package I am getting "invalid signature unzip ". Below is the code

    const request = require('superagent');
const fs = require('fs');
const unzip = require('unzipper');
request.get(url).on('error', function(err) { console.log(err) }
                .pipe(fs.createWriteStream(zipFile))
                .on('close', function() {
                    const admZip = require('adm-zip');
                    console.log('start unzip');
                    var zip = new admZip(zipFile);
                    console.log('unzipping ' + uploadDir + "to");
                    zip.extractAllTo(uploadDir, true);
                    console.log('finished unzip');

with Unzipper

 request.get(url)
            .on('error', function(err) { console.log(err) }
                .pipe(fs.createWriteStream(zipFile))
                .on('close', function() {
                    fs.createReadStream(zipFile)
                        .pipe(unzipper.Extract({ path: UploadDir }));`enter code here`
                })

score 1 · Answer 1 · edited Aug 28 '22 at 18:17

1

Error is resolved. First step is to pipe the incoming readable response stream.

request.get(urlData)
  .pipe(writeStream);

Once the read is done pipe will trigger the write stream. Then I am triggering the unzipping process on close event of the writestream.

writeStream.on('close', function() {      
  fs.createReadStream(zipFile).pipepipe(unzip.Extract({
    path: uploadDir
  }));
  console.log('finished unzip');
});

edited Aug 28 '22 at 18:17

Tyler2P

2,324
26
22
31

answered Jul 11 '21 at 10:39

Saket Shahi

31
3

but by starting from on('close'), you still loaded all the content. – jiogai Nov 30 '21 at 09:27
Even though i wished to read zip file without unzipping it , i was facing many issues with it. So i used ur code to unzip it , do the processing and delete unzipped files later.. Thanks for helping. – Jay Teli Aug 28 '22 at 07:18

Node Read large zip file from URL and then Unzip it

1 Answers1