Download File from URL and store that file in zip using node

Question

**Solution 1:** 
const zipPath = path.resolve(tempDir, 'output.zip');
 const writeStream = assets.map((assets) => {
    fileNameArray.push(asset.fileName);
    return axios({
      method: 'GET',
      responseType: 'arraybuffer',
      url: asset.downloadUrl,
    });
  });
  const downloadResponses = await Promise.all(writeStream);

  downloadResponses.forEach((response, index) => {
    zip.addFile(fileNameArray[index], response.data);
  });

  zip.writeZip(zipPath, () => {
    logger.trace('Zip file created successfully');
  });

**Solution 2:**
import * as AdmZip from 'adm-zip';
import * as os from 'os';
import axios from 'axios';
import * as path from 'path';
import * as fs from 'graceful-fs';

const chunkSize = 104857600; //100MB
const maxMemoryUsage = 1024 * 1024 * 1024; //1024MB;

const zip = new AdmZip();
const tempDir = os.tmpdir();

export const addFilesToZip = async (assets) => {
  const ZipName = `file_${Date.now()}.zip`;
  const zipPath = path.resolve(tempDir, ZipName);

  const writeStream = await assets.map(async (asset) => {
      return await downloadAndZipFile(asset, zipPath);
  });

  const downloadResponses = await Promise.all(writeStream);
  console.log(downloadResponses);

  zip.writeZip(zipPath, () => {
      console.log('Zip file created successfully');
  });
  return { signedUrl: 'wewe', fileName: 'wewe' };
};

const downloadAndZipFile = async ({ fileName, downloadUrl, metadata }) => {
  let offset = 0, chunks = [], memoryUsage = 0, data;
  const assetSize = parseInt(metadata.fileSize, 10);
  console.log('offset,assetSize', fileName, offset, assetSize);

  while (offset < assetSize) {
    const end = Math.min(offset + chunkSize, assetSize);
    const rangeHeader = `bytes=${offset}-${end - 1}`;
    console.log('end, rangeHeader', fileName, end, rangeHeader);
    try {
      const response = await axios({
        url: downloadUrl,
        method: 'GET',
        headers: { Range: rangeHeader },
        responseType: 'arraybuffer',
      });
      const buffer = Buffer.from(response.data);
      memoryUsage += buffer.byteLength;
      chunks.push(buffer);
      console.log('chunks', fileName, chunks.length, buffer.byteLength);

      if (memoryUsage > maxMemoryUsage) {
        chunks.forEach((chunk) => {
          zip.addFile(fileName, chunk);
        });
        data = zip.toBuffer();
        fs.writeFileSync(zipPath, data);
        chunks = [];
        memoryUsage = 0;
      }
      offset = end;
      console.log('chunks offset memoryUsage', fileName, chunks.length,offset,memoryUsage
      );
    } catch (error) {
      throw new Error(`Error reading from ${fileName}: ${error.message} `);
    }
  }
  if (chunks.length > 0) {
    console.log('chunks.length', fileName, chunks.length);
    chunks.forEach((chunk) => {
      zip.addFile(fileName, chunk);
    });
    data = zip.toBuffer()
  }
    console.log(zip.toBuffer.length);
    return data;
};

I have tried Solution 1 where I used axios to download data as buffer from url(i.e.,downloadUrl) and used adm-zip library to store that downloaded file in zip file. It worked but for larger files, it loading and axios is taking time and getting execution Timedout error coming (i.e., responseType: 'arrayBuffer').

So, I came up with this above Solution 2 where I am downloading data based on range and splitting buffer response into 100MB and atlast store all into zip.addFile().

This one allowing me to download but its not downloading fully. For example, if size is 150MB , after downloading and extracting from zip , it will be somewhere around 20MB only.

So, please help me on this, where I can store larger files into zip file directly. It would be very greatful to get help from someone, I really need some solution on this. Thanks in Advance

Download File from URL and store that file in zip using node

0 Answers0