0

I wanto run this dag file in airflow 2. But it isn't working with error below. I can run this same command in terminal. But it isn't working dag environment of airflow.

Error

AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-08-16T02:29:00+00:00
[2022-08-16, 20:30:01 KST] {subprocess.py:62} INFO - Tmp dir root location: 
 /tmp
[2022-08-16, 20:30:01 KST] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'conda activate ukjo ']
[2022-08-16, 20:30:01 KST] {subprocess.py:85} INFO - Output:
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - 
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - To initialize your shell, run
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - 
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO -     $ conda init <SHELL_NAME>
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - 
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - Currently supported shells are:
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO -   - bash
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO -   - fish
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO -   - tcsh
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO -   - xonsh
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO -   - zsh
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO -   - powershell
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - 
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - See 'conda init --help' for more information and options.
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - 
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - IMPORTANT: You may need to close and restart your shell after running 'conda init'.
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - 
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - 
[2022-08-16, 20:30:01 KST] {subprocess.py:96} INFO - Command exited with return code 1
[2022-08-16, 20:30:01 KST] {taskinstance.py:1909} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/ai/miniconda3/envs/ukjo/lib/python3.8/site-packages/airflow/operators/bash.py", line 194, in execute
    raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.
[2022-08-16, 20:30:01 KST] {taskinstance.py:1415} INFO - Marking task as FAILED. dag_id=dag_bash, task_id=t2, execution_date=20220816T022900, start_date=20220816T023001, end_date=20220816T023001
[2022-08-16, 20:30:01 KST] {standard_task_runner.py:92} ERROR - Failed to execute job 162361 for task t2 (Bash command failed. The command returned a non-zero exit code 1.; 4041098)
[2022-08-16, 20:30:02 KST] {local_task_job.py:156} INFO - Task exited with return code 1
[2022-08-16, 20:30:02 KST] {local_task_job.py:273} INFO - 0 downstream tasks scheduled from follow-on schedule check

DAG python file

from airflow.decorators import dag
from airflow.operators.bash import BashOperator
from datetime import datetime


@dag(start_date=datetime(2022, 8, 10), schedule_interval='* * * * *', catchup=False)
def dag_bash():

    t1 = BashOperator(task_id='t1', bash_command='source ~/miniconda3/etc/profile.d/conda.sh ')
    t2 = BashOperator(task_id='t2', bash_command='conda activate ukjo ')
    t3 = BashOperator(task_id='t3', bash_command='echo $CONDA_DEFAULT_ENV ')

    t1 >> t2 >> t3

dag = dag_bash()
verystrongjoe
  • 3,831
  • 9
  • 35
  • 66
  • 1
    `bash -c 'conda activate'` makes no sense as a thing to even attempt. Its purpose is to activate a conda environment _inside the current shell_, but that current shell exits when the `bash -c` is finished. The effect of the activate is completely undone by the shell's termination, so why bother in the first place? – Charles Duffy Aug 16 '22 at 02:38
  • Thanks. I wanted it to run this consecutive commands to use local conda environment and run my executable python module in my local server. – verystrongjoe Aug 16 '22 at 02:48

1 Answers1

0

To run a python script with a conda environment, you can just provide the full path of your conda python interpreter:

BashOperator(task_id='execute_python_script', bash_command='~/miniconda3/envs/ukjo/bin/python /path/to/python/script.py').
Hussein Awala
  • 4,285
  • 2
  • 9
  • 23