2

There was similar questions/answers but not recently and none with the exact requirements.

I have many pictures for a dating app on Firebase Storage, uploaded from the users, with a downloadUrl saved on Firestore. Just noticed it is saved as very big pictures, and slow down loading of the users. Result: I need to resize and reformat to jpeg all pictures on firebase storage.

My research and trials for now 2 months brought me to the following conclusions:

  1. It's not possible through Google Functions as the quota of 9 minutes is too slow to do the whole resizing.
  2. Sharp is the best library to do this, but better do it locally.
  3. I can use gsutil as in this Question Here to download all pictures and keep the path, resize it, and upload it later.

I was blocked at finding how I can resize/reformat with Sharp and whilst the name will be different and probably the metadata kept out, how can I uploaded it back and at the same time get a new downloadUrl so that I can in turn upload it to firestore in the users collection?

MY POTENTIAL SOLUTION (STEP 4): Not sure if it will work, but I'd have a listening function for changed (finalized) object and getting info from the image to upload it back on firestore, using a self-made downloadUrl.

MY NEW QUESTION: Is it going to work? I'm afraid to break the pictures of all my users...

For your better understanding, here is my process so far:

1. Download Images

gsutil cp -r gs://my-bucket/data [path where you want to download]

2. Script (typescript) to resize/reformat them.

import * as fs from "fs";
import sharp from "sharp";
import * as path from "path";

const walk = (dir: string, done) => {
  let results = [];
  fs.readdir(dir, (err, list) => {
    if (err) return done(err);
    let i = 0;
    (function next() {
      let file = list[i++];
      if (!file) return done(null, results);
      file = path.resolve(dir, file);
      fs.stat(file, (err, stat) => {
        if (stat && stat.isDirectory()) {
          walk(file, (err, res) => {
            results = results.concat(res);
            next();
          });
        } else {
          results.push(file);
          next();
        }
      });
    })();
  });
};

const reformatImage = async (filesPaths: string[]) => {
  let newFilesPaths: string[] = [];

  await Promise.all(
    filesPaths.map(async (filePath) => {
      let newFileName = changeExtensionName(filePath);
      let newFilePath = path.join(path.dirname(filePath), NewFileName);
      if (filePath === newFilePath) {
        newFileName = "rszd-" + newFileName;
        newFilePath = path.join(path.dirname(filePath), newFileName);
      }
      newFilesPaths.push(newFilePath);

      try {
        await sharp(filePath)
          .withMetadata()
          .resize(600, 800, {
            fit: sharp.fit.inside,
          })
          .toFormat("jpeg")
          .jpeg({
            mozjpeg: true,
            force: true,
          })
          .toFile(newFilePath)
          .then(async (info) => {
            console.log("converted file...", info);
          })
          .catch((error) => {
            console.log("sharp error: ", error);
          });
      } catch (error) {
        console.error("error converting...", error);
      }
    })
  );
  console.log("THIS IS THE RESIZED IMAGES");
  console.log(newFilesPaths);
};

const changeExtensionName = (filePath: string) => {
  const ext = path.extname(filePath || "");
  const virginName = path.basename(filePath, ext);
  const newName = virginName + ".jpg";
  return newName;
};

walk("./xxxxxx.appspot.com", (err, results) => {
  if (err) throw err;
  console.log("THIS IS THE DOWNLOADED IMAGES");
  console.log(results);
  reformatImage(results);
});

3. Re-upload the files

gsutil cp -r [path your images] gs://my-bucket/data

4. Listen for new file update through a Firebase Functions, and update the new downloadUrl

    export const onOldImageResizedUpdateDowloadUrl = functions.storage
  .object()
  .onFinalize(async (object: any) => {
    if (object) {
      functions.logger.log('OBJECT: ', object);
      const fileBucket = object.bucket;
      const filePath: string = object.name;
      const userId = path.basename(path.dirname(filePath));
      const fileName = path.basename(filePath);
      const isResized = fileName.startsWith('rszd-');
      if (!isResized) {return;}
      const token = object.metadata.firebaseStorageDownloadTokens;
      const downloadUrl = createDownloadUrl(
        fileBucket,
        token,
        userId,
        fileName
      );
      const pictureId = 'picture' + fileName.charAt(5); // pictures are named as eg "rszd-" + 1.jpeg
      admin
        .firestore()
        .collection('users')
        .doc(userId)
        .update({ [pictureId]: downloadUrl });
    }
  });

    function createDownloadUrl(
      bucketPath: string,
      downloadToken: string,
      uid: string,
      fileName: string) {
        return `https://firebasestorage.googleapis.com/v0/b/${bucketPath}/o/pictures-profil%2F${uid}%2F${fileName}?alt=media&token=${downloadToken}`;
}
Dharmaraj
  • 47,845
  • 8
  • 52
  • 84

0 Answers0