0

I am deploying Managed Airflow (MWAA) in AWS and getting the below error in the cloudformation. I am giving the log_format same as what mentioned in the airflow documentation but still CFN is giving the error.

Resource handler returned message: "Invalid request provided: Some of the provided configurations do not have the expected format: scheduler, e.g: core.log_format. (Service: Mwaa, Status Code: 400, HandlerErrorCode: InvalidRequest)

Following are the Airflow configuration options I am giving in the cloudformation.

     AirflowConfigurationOptions:
        core:
          parallelism: 64
          default_task_retries: 3
          default_timezone: Australia/Melbourne
          dag_concurrency: 16
          maximum_active_runs_per_dag: 16
          load_examples: False
          load_default_connections: False
          log_format: "[%%(asctime)s] {{%%(filename)s:%%(lineno)d}} %%(levelname)s - %%(message)s"
        webserver:
          default_ui_timezone: Australia/Melbourne
        scheduler:
          catchup_by_default: False
          allow_trigger_in_future: True
        operators:
          default_owner: vulcan
        smart_sensor:
          use_smart_sensor: True
          shards: 8
        logging:
          remote_logging: True
          remote_log_conn_id: s3
          remote_base_log_folder:
            - Fn::ImportValue: xxxxx
Gagan
  • 1,775
  • 5
  • 31
  • 59

1 Answers1

0

The correct way to mention the airflow configurations in CFN is:

  env:
    Type: AWS::MWAA::Environment
    Properties:
      AirflowConfigurationOptions:
        core.parallelism: 64
        core.default_task_retries: 3
        core.default_timezone: Australia/Melbourne

And remote login connection should be first created using AWS Secret manager and than it can be used in the remote logging airflow config in cfn.

Gagan
  • 1,775
  • 5
  • 31
  • 59