0

We built a web application where we utilized firebase functions for lightweight works such as login, update profile etc. and we deployed 2 functions to App Engine with Nodejs.

Function 1: Downloading an audio/video file from firebase storage and converting it with ffmpeg and uploading converted version back to storage. But App Engine is terminating with a signal in the middle of download process (after ~40 seconds) if the file is larger (>500MB)

Function 2: Calling Google API (ASR) and waiting response (progress %) and writing this response to firestore until it's completed (100%). This process may take between 1 min - 20 min depending on the file length. But here we get two different problems.

  1. Either App Engine creates a new instance in the middle of API call process and kills current instance (since we set instances# to 1) even there is no concurrent requests. I don't understand this behavior since this should not require intensive CPU or memory usage. App Engine gives following Info in logs:

This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application

  1. Or App Engine terminates due to idle_timeout even it is waiting async API response and writing it to db. It looks like when there is no incoming requests, App Engine is considering itself as idle and terminating after a while (~10 minutes)

We are new to GCP and App Engine, so maybe we are using wrong product (e.g. Compute Engine?) or doing the wrong implementation. Also we saw PubSub, Cloud Tasks etc. which looks like a solution for our case. Thus I wonder what could be most elegant way to approach the problem and implement solution? Any comment, feedback is appreciated.

Regards A. Faruk Acar

App Engine app.yaml Configuration

runtime: nodejs10
manual_scaling:
    instances: 1
Soni Sol
  • 2,367
  • 3
  • 12
  • 23
afacar
  • 11
  • 2

1 Answers1

0

App Engine has a maximum timeout per request

  • 10 minutes for App Engine Standard
  • 60 minutes for App Engine Flex

in both cases the default values are less.

So for what you describe as your process it is not the optimal solution.

Cloud Tasks has similar limitations as App Engine so you might get to similar problems.

Compute Engine can work for you as it is virtual machines where you control the configuration. To keep it cost effective see what is the smallest computer type which can run your app.

Soni Sol
  • 2,367
  • 3
  • 12
  • 23