Is there a way to run BashOperator tasks in a virtual environment in airflow?

Question

I am aware of how task decorators are used to decorate python callables to create virtual environments for them to run in. However I have a requirement where I need to run a BashOperator based task in a virtual environment.

with DAG(
    "test_dag_venv",
    default_args=default_args,
    description='Dag to test venv',
    schedule_interval="@once",
    start_date=datetime(2022, 1, 6, 10, 45),
    tags=['testing'],
    concurrency=1,
    is_paused_upon_creation=True,
    catchup=False  # dont run previous and backfill; run only latest
) as dag:
    @task.virtualenv(task_id="print_test", requirements=['numpy'], system_site_packages=False)
    def print_test():
        import numpy as np
        print(np.__version__)
    t1 = print_test()
    t1

The above works as expected.Is there anything close to run BashOperator tasks as well. Something like.

  with DAG(
    "test_dag_venv",
    default_args=default_args,
    description='Dag to test venv',
    schedule_interval="@once",
    start_date=datetime(2022, 1, 6, 10, 45),
    tags=['testing'],
    concurrency=1,
    is_paused_upon_creation=True,
    catchup=False  # dont run previous and backfill; run only latest
) as dag:
    t1 = BashOperator(task_id='t1',
    bash_command="python scripts/my_script.py",
    requirements=['numpy']
    dag=dag)
    t1

score 0 · Answer 1 · answered Oct 11 '22 at 14:22

Just set full path of python virtualenv to PATH variable. Example:

cd /tmp
virtualenv --python python3.10 example
source example/bin/activate
# pip install etc...
which python
# /tmp/example/bin/python

And set path to your prepared virtualenv using env argument

BashOperator(task_id='task1',
    bash_command='python scripts/my_script.py',
    env=dict(PATH='/tmp/example/bin/python'),
    dag=dag,
)

Or you can activate a virtualenv inside bash_command:

bash_command='source /tmp/example/bin/activate && python scripts/my_script.py'

Is there a way to run BashOperator tasks in a virtual environment in airflow?

1 Answers1