0

Whenever a user has to publish an image in the storage I don't do it directly with the firebase storage functions but using an onCall cloud function by passing it a base64 image, modifying it (5 times), and posting it in the storage.

The function is as follows:

exports.uploadImage = functions.https.onCall(async (data, context) => {
    

   var bucket = admin.storage().bucket(); 

   // Convert the base64 string back to an image 
   var base64EncodedImageString = data.image,
   mimeType = 'image/jpeg',
   fileName = 'testName',
   imageBuffer = Buffer.from(base64EncodedImageString, 'base64');

   if (!fs.existsSync(os.tmpdir()+"/myfolder")){
      fs.mkdirSync(os.tmpdir()+"/myfolder");
  }

   const tempFilePath = path.join(os.tmpdir(),"myfolder", fileName+".jpg");


   fs.writeFileSync(tempFilePath,base64EncodedImageString,'base64',function(err){
      functions.logger.log("file scritto in"+tempFilePath);
   })
  
   await bucket.upload(tempFilePath, {
      destination: 'test/'+fileName,
      metadata: { contentType: mimeType,
         metadata: {
            firebaseStorageDownloadTokens: uuid()
         } 
       },
    });


   const tempFilePath_25 = path.join(os.tmpdir(),"myfolder", fileName+"_25.jpg");
   spawnSync('convert', [tempFilePath, '-scale', '10%','-scale','1000%>', tempFilePath_25]);
   await bucket.upload(tempFilePath_25, {
      destination: 'test/'+fileName+"_25.jpg",
      metadata: { contentType: mimeType,
         metadata: {
            firebaseStorageDownloadTokens: uuid()
         } 
       },
    });
    fs.unlinkSync(tempFilePath_25);
      
   const tempFilePath_50 = path.join(os.tmpdir(),"myfolder",  fileName+"_50.jpg");
   spawnSync('convert', [tempFilePath, '-scale', '5%','-scale','2000%>', tempFilePath_50]);
   await bucket.upload(tempFilePath_50, {
      destination: 'test/'+fileName+"_50.jpg",
      metadata: { contentType: mimeType,
         metadata: {
            firebaseStorageDownloadTokens: uuid()
         } 
       },
    });
    fs.unlinkSync(tempFilePath_50);

   const tempFilePath_75 = path.join(os.tmpdir(),"myfolder",  fileName+"_75.jpg");
   spawnSync('convert', [tempFilePath, '-scale', '3%','-scale','3333%>', tempFilePath_75]);
   await bucket.upload(tempFilePath_75, {
      destination: 'test/'+fileName+"_75.jpg",
      metadata: { contentType: mimeType,
         metadata: {
            firebaseStorageDownloadTokens: uuid()
         } 
       },
    });
    fs.unlinkSync(tempFilePath_75);

   const tempFilePath_100 = path.join(os.tmpdir(),"myfolder",  fileName+"_100.jpg");
   spawnSync('convert', [tempFilePath, '-scale', '1%','-scale','10000%>', tempFilePath_100]);
   await bucket.upload(tempFilePath_100, {
      destination: 'test/'+fileName+"_100.jpg",
      metadata: { contentType: mimeType,
         metadata: {
            firebaseStorageDownloadTokens: uuid()
         } 
       },
    });
    fs.unlinkSync(tempFilePath_100);


});

I tried to do a simulation with a for loop every 2 seconds and I get the deadline error for 60% of the requests. When I publish the app there will be many users (hopefully) who can potentially call the same function simultaneously to post a photo. How can I solve this problem? Thanks in advance.

Carlo Casadei
  • 5
  • 1
  • 3
  • What is the "deadline error"? Also, your code is writing to a hard-coded path every single time, so is it possible you are running into an issue where the same bucket path is being written to simultaneously? You should be adding some type of uniqueness to the filename - maybe a timestamp or use a library such as `uuid`. That way concurrent calls will not be overwriting each other. – Greg Fenton May 09 '21 at 16:22
  • The error is DEADLINE_EXCEEDED. The problem is that the function takes 5 seconds per user to ends. If I have a lot of users, due to the fact that the function is the same, I'm afraid of that becouse the function has limit of memory and limit of call.. – Carlo Casadei May 09 '21 at 16:49
  • Where are you seeing this error? Have you considered putting try/catches around your code to determine exactly which part of the code is throwing that error? Is the error in the Cloud Function, or the client app that is calling `onCall()`? – Greg Fenton May 09 '21 at 19:36
  • The error doesn't came from client side. The code is "perfect". Don't worry about uuid and the possibility that multiple clients can overwrite to each other. I know that, I will change the code. It's just a test. The problem is that when I simulate 25 users with a for loop calling the same functions every 2 seconds the server send to me this error. In practice I think that it cannot manage so much requests. But this is a problem because my app is a chat app, so this function will be called a lot of times from a lot of users. I cannot say to my users "the server is not available. Try later" – Carlo Casadei May 09 '21 at 19:47
  • If I run the function I don't have problem. Even if I call it every 5 seconds with the same for loop. No problem. The problem is when I call it every 2 seconds. – Carlo Casadei May 09 '21 at 19:49
  • Are you calling with 25 separate logins (user sessions) or all from the same one? That error message indicates that you are hitting one of the limitations within Firebase. There are several limits on Firebase services: per user API calls, overall write operations in a given time period, etc. I suspect your testing isn't simulating real-world usage ... Firebase is used by 100,000s of apps, many at large scale. That said, there are ways you could "cache" the work to be done and have another process "trickle" through the work rather than doing it all "on-demand". – Greg Fenton May 09 '21 at 20:12
  • Yes the for loop send 25 request every 2 seconds from the same user. But the error seems not to be a limit per user, but just a limit of the function itself to manage a lot of requests. I can't do without this feature, my users have to use it to publish their photos. What's your solution? Anyway thanks for the discussion – Carlo Casadei May 09 '21 at 20:33
  • You likely want to review the [quotas and limits page for Cloud Functions](https://firebase.google.com/docs/functions/quotas), including the section on [what to do when you hit a quota](https://firebase.google.com/docs/functions/quotas#when_you_reach_a_quota_limit) – Greg Fenton May 10 '21 at 03:45
  • this is not a limitation of the invocations - it appears there are shared and conflicting resources. and your structure doesn't account for that, using the temp directory is not localized per instance, it's rather shared in your entire cloud functions for that region the end result is it can't complete and hangs, returning a "deadline_exeeded" – DIGI Byte May 10 '21 at 04:52
  • I changed the code replacing "my Folder" with a var folder=uuid() at the start of the function and using that for the rest of the code. So I have different path for different call. With the same test of 25 calls only 4 fail with the same error of DEADLINE_EXCEEDED – Carlo Casadei May 10 '21 at 05:39

0 Answers0