There was similar questions/answers but not recently and none with the exact requirements.
I have many pictures for a dating app on Firebase Storage, uploaded from the users, with a downloadUrl saved on Firestore. Just noticed it is saved as very big pictures, and slow down loading of the users. Result: I need to resize and reformat to jpeg all pictures on firebase storage.
My research and trials for now 2 months brought me to the following conclusions:
- It's not possible through Google Functions as the quota of 9 minutes is too slow to do the whole resizing.
- Sharp is the best library to do this, but better do it locally.
- I can use gsutil as in this Question Here to download all pictures and keep the path, resize it, and upload it later.
I was blocked at finding how I can resize/reformat with Sharp and whilst the name will be different and probably the metadata kept out, how can I uploaded it back and at the same time get a new downloadUrl so that I can in turn upload it to firestore in the users collection?
MY POTENTIAL SOLUTION (STEP 4): Not sure if it will work, but I'd have a listening function for changed (finalized) object and getting info from the image to upload it back on firestore, using a self-made downloadUrl.
MY NEW QUESTION: Is it going to work? I'm afraid to break the pictures of all my users...
For your better understanding, here is my process so far:
1. Download Images
gsutil cp -r gs://my-bucket/data [path where you want to download]
2. Script (typescript) to resize/reformat them.
import * as fs from "fs";
import sharp from "sharp";
import * as path from "path";
const walk = (dir: string, done) => {
let results = [];
fs.readdir(dir, (err, list) => {
if (err) return done(err);
let i = 0;
(function next() {
let file = list[i++];
if (!file) return done(null, results);
file = path.resolve(dir, file);
fs.stat(file, (err, stat) => {
if (stat && stat.isDirectory()) {
walk(file, (err, res) => {
results = results.concat(res);
next();
});
} else {
results.push(file);
next();
}
});
})();
});
};
const reformatImage = async (filesPaths: string[]) => {
let newFilesPaths: string[] = [];
await Promise.all(
filesPaths.map(async (filePath) => {
let newFileName = changeExtensionName(filePath);
let newFilePath = path.join(path.dirname(filePath), NewFileName);
if (filePath === newFilePath) {
newFileName = "rszd-" + newFileName;
newFilePath = path.join(path.dirname(filePath), newFileName);
}
newFilesPaths.push(newFilePath);
try {
await sharp(filePath)
.withMetadata()
.resize(600, 800, {
fit: sharp.fit.inside,
})
.toFormat("jpeg")
.jpeg({
mozjpeg: true,
force: true,
})
.toFile(newFilePath)
.then(async (info) => {
console.log("converted file...", info);
})
.catch((error) => {
console.log("sharp error: ", error);
});
} catch (error) {
console.error("error converting...", error);
}
})
);
console.log("THIS IS THE RESIZED IMAGES");
console.log(newFilesPaths);
};
const changeExtensionName = (filePath: string) => {
const ext = path.extname(filePath || "");
const virginName = path.basename(filePath, ext);
const newName = virginName + ".jpg";
return newName;
};
walk("./xxxxxx.appspot.com", (err, results) => {
if (err) throw err;
console.log("THIS IS THE DOWNLOADED IMAGES");
console.log(results);
reformatImage(results);
});
3. Re-upload the files
gsutil cp -r [path your images] gs://my-bucket/data
4. Listen for new file update through a Firebase Functions, and update the new downloadUrl
export const onOldImageResizedUpdateDowloadUrl = functions.storage
.object()
.onFinalize(async (object: any) => {
if (object) {
functions.logger.log('OBJECT: ', object);
const fileBucket = object.bucket;
const filePath: string = object.name;
const userId = path.basename(path.dirname(filePath));
const fileName = path.basename(filePath);
const isResized = fileName.startsWith('rszd-');
if (!isResized) {return;}
const token = object.metadata.firebaseStorageDownloadTokens;
const downloadUrl = createDownloadUrl(
fileBucket,
token,
userId,
fileName
);
const pictureId = 'picture' + fileName.charAt(5); // pictures are named as eg "rszd-" + 1.jpeg
admin
.firestore()
.collection('users')
.doc(userId)
.update({ [pictureId]: downloadUrl });
}
});
function createDownloadUrl(
bucketPath: string,
downloadToken: string,
uid: string,
fileName: string) {
return `https://firebasestorage.googleapis.com/v0/b/${bucketPath}/o/pictures-profil%2F${uid}%2F${fileName}?alt=media&token=${downloadToken}`;
}