40

I'm trying to write our first Airflow DAG, and I'm getting the following error when I try to list the tasks using command airflow list_tasks orderwarehouse:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2038, in resolve_template_files
    setattr(self, attr, env.loader.get_source(env, content)[0])
  File "/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in get_source
    raise TemplateNotFound(template)
TemplateNotFound: ./home/deploy/airflow-server/task_scripts/orderwarehouse/load_warehouse_tables.sh

This DAG is not supposed to use a template. I'm only trying to run the shell script in the specified location per the instructions in the docs. The shell script does exist in that location and is spelled correctly. My DAG looks like this:

from airflow import DAG
from airflow.operators.bash_operator import BashOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2015, 6, 1),
    'email': ['airflow@airflow.com'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
    # 'queue': 'bash_queue',
    # 'pool': 'backfill',
    # 'priority_weight': 10,
    # 'end_date': datetime(2016, 1, 1),
}

orderwarehouse = DAG('orderwarehouse', default_args=default_args)

load_mysql = BashOperator(
    task_id='load_warehouse_mysql',
    bash_command='./home/deploy/airflow-server/task_scripts/orderwarehouse/load_warehouse_tables.sh',
    dag=orderwarehouse)

Not sure why it thinks it needs to look for a Jinja template. Running out of ideas on this one, would appreciate if anyone can point me to where I'm going astray. Thanks.

quaintm
  • 677
  • 1
  • 7
  • 18

3 Answers3

81

This is a pitfall of airflow. Add a space at the end of your bash_command and it should run fine

Source: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=62694614

justinpitts
  • 721
  • 8
  • 11
Rishi Barve
  • 1,133
  • 9
  • 12
  • 2
    Discovered it also works with a newline ahead of the filename! – quaintm Feb 14 '17 at 15:18
  • Does this also apply if the command has args following it (eg. `bash /path/to/script.sh arg1 arg2 ...`)? The docs did not seem to indicate. – lampShadesDrifter Oct 07 '19 at 20:56
  • i think it does, however i haven't tested that. – Rishi Barve Oct 09 '19 at 12:10
  • If you look at the [BashOperator source](https://github.com/apache/airflow/blob/d719e1fd6705a93a0dfefef4b46478ade5e006ea/airflow/operators/bash_operator.py#L68), there's `template_ext = ('.sh', '.bash',)`. Any string ending with either of those values will be interpreted as a Jinja template file. There's also `template_fields = ('bash_command', 'env')` which is the list of fields this behavior applies to. – cyfur01 Nov 06 '19 at 23:32
  • 9
    It's now 2020, how can this bug still be there? I just spent hours tracking this down. :( – user1175849 May 25 '20 at 23:58
  • 7
    Its now 2021, and the bug still there – Mehdi Hadji Feb 11 '21 at 22:52
  • (Putting a note in my agenda for 1-jan-2022, 1 month from now, to make the next comment here...) :-) – BertC Dec 01 '21 at 09:32
  • placing semi colon (;) at the end of command also solve the problem – Pranithan T. Dec 21 '21 at 08:24
  • 5
    it is 2022 and the bug is still there and also applies to wherever you need to pass a file_path as a parameter – Hesam Korki Jul 11 '22 at 13:14
  • 2
    it is 2023 and the bug is still there – Mark Korzhov Jun 15 '23 at 11:53
5

in addition to all the answers provided i had to do something more to get rid of the jinja template not found issue.

we have to add space after the file name in task definition.

enter image description here

Hari Mittapalli
  • 86
  • 1
  • 6
3

You should try with space at the end of filepath. whichever operator you are using you should always follow the same rule.

load_mysql = BashOperator(  
             task_id='load_warehouse_mysql',  
             command='/home/deploy/airflow-server/task_scripts/orderwarehouse/load_warehouse_tables.sh ',  
             dag=orderwarehouse)
Shrikant
  • 31
  • 4