Multi-threading for heavy transcoding tasks in NodeJS on flex GAE

Question

I am transcoding lots of MP4 to FLAC/WAV using fluent-ffmpeg on flex GAE
Thinking about how GAE will handle 10-20 concurrent transcoding operations

Transcoding is computationally expensive and GAE might? spawn new instances for each request if not optimized

app.yaml

runtime: nodejs
env: flex

resources:
  cpu: 2
  memory_gb: 3.75

automatic_scaling:
  min_num_instances: 1
  max_num_instances: 30
  cpu_utilization:
    target_utilization: 0.8

Is there a way to allocate the load on a single GAE instance? Node web workers?
What's the conventional resources section in app.yaml for such tasks?

Answer 1

If you want to avoid the creation on new instances ( horizontal scaling ) for each request you should work on your vertical scaling - choose a powerful enough instance class that you think would be enough to process your requests and then set the max_num_instances parameter accordingly to your desire. Of course, being aware on how you structure and design your code in an "economical" way is the most important part.

If you want to be sure that only one GAE instance is running, set to manual scaling and chooseinstances: 1. Or leave it on automatic_scaling and set max_num_instances: 1. ( I did not try this, but I do not see why it wouldn't work).

Keep in mind to change the value of the parameter max_concurrent_requests to whatever value you want. By default it will be set to 10. So you would not be able to process more than 10 concurrent requests at a time.