Installing a package that needs to run gcc in Airflow (python-lzo)

Question

I encounter an error when trying to pip inatall a package which needs root permissions in airflow Dockerfile.

The package is python-lzo. Python 3.7, airflow 2.3.0.

The error I get is a known: "trying to run pip as root", and do acknowledge it is a known issue/fact that according to the Airflow documentations here.

I tried setting the environment variable as this solution stated and it still did not work.

The following example adds lxml python package from PyPI to the image. When adding packages via pip you need to use the airflow user rather than root. Attempts to install pip packages as root will fail with an appropriate error message.

They say you must run pip as the "airflow" user and not the "root".

However, the package I need is called "python-lzo" which needs to run gcc, and gcc only runs as root.

Is there an alternative way to install python-lzo on Airflow, if it needs gcc with root permissions?

Some people suggested I compile the python-lzo library and run gcc in my local computer, then somehow upload it to the Dockerfile, however I am pretty lost there because I don't understand how it's possible, when compiling it locally it means it's dependant on my OS, and I want the DAG to run on Linux (I have Windows).

The dockerfile:

From artifactory.whatever
Copy requirements.txt
Copy pip.ini ~/.pip/pip.conf
Copy mylibrary.whl
Env PIP_CONFIG_FILE ~/.pip/pip.conf
USER root
Run pip install mylibrary

The mylibrary is a .whl I compiled using poetry and in the requirements there is the library python-lzo.

@ozs i updated, the mylibrary is a python library I built and compiled the . whl using "poetry", in the requirements.txt there is the python-lzo library — mimente1200, Aug 23 '23 at 15:45
try to set PIP_USER=False (airflow docker its true) before run pip install — ozs, Aug 23 '23 at 15:52
So the line I add is "SET PIP_USER=False"? After the USER root? Or before? — mimente1200, Aug 23 '23 at 16:11
ry to set PIP_USER=False (airflow docker its true) before USER root — ozs, Aug 23 '23 at 16:54

score 0 · Answer 1 · answered Aug 24 '23 at 17:37

0

You can install gcc and liblzo2-dev as root and then pip install as airflow

USER root

RUN apt-get update
RUN apt-get -y install gcc liblzo2-dev

USER airflow
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" python-lzo --user

answered Aug 24 '23 at 17:37

ozs

3,051
1
10
19

I get unable to locate gcc... – mimente1200 Aug 25 '23 at 07:12
Whick docker image do u use? Also do run apt-get update before install? – ozs Aug 25 '23 at 07:30
Im in another network so i dont have apt-get update because my `sources.list` is not a link i can access and Im using docker image of airflow 2.6.2 – mimente1200 Aug 25 '23 at 09:04

Installing a package that needs to run gcc in Airflow (python-lzo)

1 Answers1