In Apache Airflow Tool, DAG wont run due to Duplicate Entry Problem in task_instance table

Question

Today all day i have been getting this error in the scheduler of Airflow.

sqlalchemy.exc.IntegrityError: (_mysql_exceptions.IntegrityError) (1062, "Duplicate entry '%' fir key 'PRIMARY')")

Because of this the Airflow Scheduler would stop and every time i ran this had the same problem

I have found the solution to this problem. This happens if used MySQL DB as metadata store. This problem is a bug in Apache JIRA issues : https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-3045?filter=allopenissues This can be fixed by altering the table and removing the extra `ON UPDATE CURRENT_TIMESTAMP` Hence the fix for this is : alter table `task_instance` change `execution_date` `execution_date` TIMESTAMP(6) not null DEFAULT CURRENT_TIMESTAMP(6); — sumeetgautam, Oct 17 '18 at 16:11
Please *never* post here images of text. text can be copy-paste here and format as a quote or code, or readability. — Nic3500, Oct 17 '18 at 16:35

score 3 · Answer 1 · answered Oct 17 '18 at 16:14

3

This is due to MySQL's ON UPDATE CURRENT_TIMESTAMP and this is posted in JIRA of Airflow :

I fixed this by altering the table to

alter table `task_instance` change `execution_date` `execution_date` TIMESTAMP(6) not null DEFAULT CURRENT_TIMESTAMP(6);

answered Oct 17 '18 at 16:14

sumeetgautam

1 Answers1