3

I've just set up a Cloud Composer Environment on Python 3 and Composer image version composer-1.4.0-airflow-1.10.0. All settings are otherwise "stock"; i.e. no configuration overrides.

I'm trying to test out an extremely simple DAG. It runs without issue on my local Airflow server, but on Cloud Composer, the web server's task info view has the message Dependencies Blocking Task From Getting Scheduled

The dependencies are Unknown, with the following reason:

All dependencies are met but the task instance is not running. In most cases this just means that the task will probably be scheduled soon unless:
- The scheduler is down or under heavy load
- The following configuration values may be limiting the number of queueable processes: parallelism, dag_concurrency, max_active_dag_runs_per_dag, non_pooled_task_slot_count

If this task instance does not start soon please contact your Airflow administrator for assistance.

This happens whether the task is run as scheduled, or when I manually trigger it in the web server (I set all task instances to succeeded before doing this, to avoid delays). I have tried resetting the scheduler in kubernetes as per this answer, but the tasks are still stuck in scheduled.

Also, I noticed that on my local instance (running the server, worker, and scheduler on different Docker containers), the Hostname column in the Task Instances view is populated, but on Cloud Composer, it isn't.

Here is the DAG I am running:

from datetime import datetime, timedelta
import random

from airflow import DAG
from airflow.operators.python_operator import PythonOperator


default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email_on_failure': False,
    'email_on_retry': False,
    'queue': 'airflow',
    'start_date': datetime.today() - timedelta(days=2),
    'schedule_interval': None,
    'retries': 2,
    'retry_delay': timedelta(seconds=15),
    'priority_weight': 10,
}


example_dag = DAG(
    'example_dag',
    default_args=default_args,
    schedule_interval=timedelta(days=1)
)


def always_succeed():
    pass


always_succeed_operator = PythonOperator(
    dag=example_dag,
    python_callable=always_succeed,
    task_id='always_succeed'
)


def might_fail():
    return 1 / random.randint(0, 1)


might_fail_operator = PythonOperator(
    dag=example_dag, python_callable=might_fail, task_id='might_fail'
)


might_fail_operator.set_upstream(always_succeed_operator)
Eric Fulmer
  • 706
  • 2
  • 6
  • 23

1 Answers1

2

Cloud Composer doesn't support multiple celery queues, please remove 'queue' : 'airflow' from the default args. That should fix your problem.

Feng Lu
  • 691
  • 5
  • 6