2

I want to build a cloud function that receives an object like this one:

{
    "files": [
        {
            "url": "https://myUrl/cat.jpg",
            "name": "cat.jpg"
        },
        {
            "url": "https://anyOtherUrl/mouse.jpg",
            "name": "mouse.jpg"
        },
        {
            "url": "https://myUrl2/dog.jpg",
            "name": "dog.jpg"
        }
    ],
    "referenceId": "cute-images"
}

I would like to get those files, compress them into a zip file (with name = referenceId), save that zip file to a bucket, and finally, send the zip URL back as a response.

My main problem lies with the use of memory and my desire to make correct use of pipes/streams. I would really appreciate good sources of where to find the documentation for this implementation.

This is what I got for now but I don't know if it's any good:

const ZipStream = require("zip-stream");
const fetch = require("node-fetch");
const { Storage } = require("@google-cloud/storage");

exports.zipBuilder = async (req, res) => {

  // Deleted lines of request validation

  let zip = new ZipStream();
  const queue = req.body.files;

  async function addFilesToZip() {
    let elem = queue.shift();
    const response = await fetch(elem.url);
    const stream = await response.buffer();
    zip.entry(stream, { name: elem.name }, (err) => {
      if (err) throw err;
      if (queue.length > 0) addNextFile();
      else zip.finalize();
    });
  }

  await addFilesToZip();

  const storage = new Storage();
  
  const ourBucket = storage.bucket(process.env.DESTINATION_BUCKET);

  zip.pipe(ourBucket); // this fails

  // Get ZIP URL from bucket

  res.send(zipUrl);
};

Edit: This question seems like a lot of questions in one. But since this must work as a single stream, I ask not for exact answers but of ideas on what to study to get a better understanding of the solution.

gmanmar
  • 23
  • 6
  • Why does `zip.pipe(ourBucket)` fail? Wha is the error given? – Amauri Jan 31 '21 at 04:10
  • It says `Unhandled rejection TypeError: dest.write is not a function` – gmanmar Jan 31 '21 at 04:13
  • The first mistake is not awaiting for `addFilesToZip()`. It should have `await addFilesToZip()` since it is an async function and we depend on the completed state to proceed with the next operation. – Amauri Jan 31 '21 at 04:15

1 Answers1

2

You are getting 'Unhandled rejection TypeError: dest.write is not a function' bc ourBucket is not a writable stream. It must be an instance of writable stream for you to pipe to it. You should create a bucket file writable stream and use that as ourBucket.

Serhii Rohoza
  • 4,287
  • 2
  • 16
  • 29
Amauri
  • 540
  • 3
  • 15
  • Overall you are headed in the right direction. It's the implementation details that need work. I would break it up into 3 steps and test each step individually. First, downloading the files. Second, zipping the files. Third, uploading. Fourth, generating a public url. – Amauri Jan 31 '21 at 04:33
  • For generating a public url, I suggest you use something like [https://stackoverflow.com/questions/20478369/how-do-you-get-or-generate-a-url-to-the-object-in-a-bucket](https://stackoverflow.com/questions/20478369/how-do-you-get-or-generate-a-url-to-the-object-in-a-bucket) – Amauri Jan 31 '21 at 04:33
  • 2
    Ok! I will do it! I implemented the file.createWriteStream() and now I'm not getting any error at all but no file is being written. LOL. Still thanks to your help I got my inertia back. – gmanmar Jan 31 '21 at 04:38