0

I have a DAG that uses Environment variables. the Environment variables are set in /etc/default/airflow-scheduler

export MY_KEY=1234

but when I echo MY_KEY in the DAG it doesn't print anything.

I checked the airflow scheduler variables and verified that MY_KEY existed. the command I used for verifying that was: 

cat /proc/process_id_of_airflow_scheduler/environ

anyone can advise me on how to solve this problem? Thanks

sia
  • 537
  • 1
  • 6
  • 22
  • Just for clarity, are you actually echoing `MY_KEY` or `KEY` as stated? The latter doesn't match your export. – PirateNinjas Aug 20 '19 at 15:15
  • @PirateNinjas, I edited my question. – sia Aug 20 '19 at 15:20
  • Could you give some more information around how you're using it? Is it something defined at trigger time? If so are you using the web-server to trigger it? If you're using the web-server, does it have access the the environment variable? – PirateNinjas Aug 20 '19 at 15:23
  • I am just printing it in one of tasks in the DAG. it is defined in a /etc/default/airflow-scheduler. I am running a dag from airflow UI. how can I check if web-server has access to it? – sia Aug 20 '19 at 16:16
  • I'm not familiar with setting the environment in `/etc/default/airflow-scheduler`. Is this in the docs somewhere? If you're open to alternative strategies, I would normally have a `.env` file in the project and run `source .env` before starting the scheduler. This will let the scheduler see the environment variable. – PirateNinjas Aug 20 '19 at 18:03
  • You can give [this](https://stackoverflow.com/questions/57550692/airflow-reset-environment-variable-while-running-bashoperator#comment101595723_57550692) a try – y2k-shubham Aug 21 '19 at 03:38

2 Answers2

1

I found the answer. I added the key in the /etc/environment and now it is working.

sia
  • 537
  • 1
  • 6
  • 22
0

If you are using Centos/Redhat, the key should rather be added in /etc/sysconfig/airflow or /etc/default/airflow on Debian/Ubuntu. See this answer.

Tomasz Zielański
  • 443
  • 3
  • 12