Airflow DagRunAlreadyExists even after providing the custom run id and execution date

Question

I am Getting DagRunAlreadyExists exception even after providing the custom run id and execution date.

This occurs when there are multiple request within a second.

Here is the MWAA CLI call

def get_unique_key():
    from datetime import datetime
    import random
    import shortuuid
    import string
    
    timestamp = datetime.now().strftime(DT_FMT_HMSf)
    random_str = timestamp + ''.join(random.choice(string.digits + string.ascii_letters) for _ in range(8))
    uuid_str = shortuuid.ShortUUID().random(length=12)
    return '{}{}'.format(uuid_str, random_str)


execution_date = datetime.utcnow().strftime("%Y-%m-%dT%H:%m:%S.%f")
dag_run_id = get_unique_key()    
workflow_id = "my_workflow"    
conf = json.dumps({"foo": "bar"})    
"dags trigger {0} -c '{1}' -r {2} -e {3}".format(workflow_id, conf, dag_run_id, execution_date)

and here is the error log from MWAA CLI. If this can help to debug the issue.

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/airflow/__main__.py", line 48, in main
    args.func(args)
  File "/usr/local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/dag_command.py", line 138, in dag_trigger
    dag_id=args.dag_id, run_id=args.run_id, conf=args.conf, execution_date=args.exec_date
  File "/usr/local/lib/python3.7/site-packages/airflow/api/client/local_client.py", line 30, in trigger_dag
    dag_id=dag_id, run_id=run_id, conf=conf, execution_date=execution_date
  File "/usr/local/lib/python3.7/site-packages/airflow/api/common/experimental/trigger_dag.py", line 125, in trigger_dag
    replace_microseconds=replace_microseconds,
  File "/usr/local/lib/python3.7/site-packages/airflow/api/common/experimental/trigger_dag.py", line 75, in _trigger_dag
    f"A Dag Run already exists for dag id {dag_id} at {execution_date} with run id {run_id}"
airflow.exceptions.DagRunAlreadyExists: A Dag Run already exists for dag id my_workflow at 2022-10-18T06:10:28+00:00 with run id CL4Adauihkvz121928332658Gp6bsTWU

is it that you send dag_run_id but you assigned it to run_id ? I don't see run_id in the command. also, share get_unique_key implementation — ozs, Oct 18 '22 at 11:14

score 0 · Answer 1 · answered Oct 21 '22 at 19:59

0

The problem is that the execution_date resolution is seconds . Airflow ignoring the milliseconds.

You can see in the error that no milliseconds mentioned in the execution_date (2022-10-18T06:10:28)

answered Oct 21 '22 at 19:59

ozs

3,051
1
10
19

Airflow DagRunAlreadyExists even after providing the custom run id and execution date

1 Answers1