1

I am going through my newly created Cloud Composer 2 environment configuration. Previously I only used the version 1 and I still don't feel familiar with some aspects of Cloud Composer 2. While looking at the configuration, I realised that there is a line:

parallelism = 0

The reason I'm surprised about this is that in normal Airflow environment this would mean that no tasks can run in parallel (the default Airflow value for this parameter is 32). So the question is:

Does the Cloud Composer 2 change the default configuration automatically on purpose, because it's doing some autoscaling under the hood that requires this, or is it just a mistake?

BloodthirstyPlatypus
  • 79
  • 6

1 Answers1

1

The meaning of parallelism is infinity, you can check the code of base_executor :

Class to derive in order to interface with executor-type systems like Celery, Kubernetes, Local, Sequential and the likes.

Parameters parallelism (int) – how many jobs should run at one time. Set to 0 for infinity

Mazlum Tosun
  • 5,761
  • 1
  • 9
  • 23