3

I have installed airflow 2.0.2 using docker-compose as described under https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html.

I have researched quite some time, but I don't find a way to install python dependencies for my DAGs. I know how to do this using a Dockerfile (COPY requirements.txt /app / RUN pip install -r requirements.txt), but there is no Dockerfile involved here.

  • How to install python dependencies in airflow 2.0.2 / docker-compose?
  • On which of the six containers which are deployed when using the custom docker-compose do I need to install the dependencies?
Requin
  • 467
  • 4
  • 16
  • 2
    Check this [answer](https://stackoverflow.com/a/66701128/10569220), I believe is what you are looking for. – NicoE May 11 '21 at 14:01
  • I have seen this as well, but Apache seems to have changed its documentation, and I cannot find the mentioned content where the link points to. Without this information, I do not understand the reply... – Requin May 11 '21 at 14:12
  • 1
    You were right links were changed, I just updated them. – NicoE May 11 '21 at 14:26
  • glad to heard that! (don't forget to upvote that answer xD) – NicoE May 11 '21 at 16:24

1 Answers1

1

Instead of using

image: apache/airflow:x.x.x

in your docker compose you want to set this to

build:
    context: .
    dockerfile: Dockerfile

and then write this to your Dockerfile

FROM apache/airflow:x.x.x

COPY requirements.txt ./requirements.txt

where your requirements.txt contains the python packages. So you're not using the image but instead build it again with also installing the python packages.

Fabian Wörenkämper
  • 63
  • 4