Python.h: No such file or directory on Amazon Linux Lambda Container

Question

I am trying to build this public.ecr.aws/lambda/python:3.6 based Dockerfile with a requirements.txt file that contains some libraries that need gcc/g++ to build. I'm getting an error of a missing Python.h file despite the fact that I installed the python development package and /usr/include/python3.6m/Python.h exists in the file system.

Dockerfile

FROM public.ecr.aws/lambda/python:3.6
RUN yum install -y gcc gcc-c++ python36-devel.x86_64
RUN pip install --upgrade pip && \
    pip install cyquant
COPY app.py ./
CMD ["app.handler"]

When I build this with

docker build -t redux .

I get the following error

cyquant/dimensions.cpp:4:20: fatal error: Python.h: No such file or directory
 #include "Python.h"
                    ^
compilation terminated.
error: command 'gcc' failed with exit status 1

Notice, however, that my Dockerfile yum installs the development package. I have also tried the yum package python36-devel.i686 with no change.

What am I doing wrong?

I ended up just using a python image as my base instead of the lambda specific one. — mmachenry, Jun 06 '21 at 14:51
So any image can be used for lambda, we are not supposed to stick with this? — Abdallah Nasir, Jun 07 '21 at 05:59
Yeah I just used a regular Python image and add ENTRYPOINT ["/usr/local/bin/python", "-m", "awslambdaric"] CMD ["main.handler"] And you need to pip install awslambdaric — mmachenry, Jun 10 '21 at 15:28

score 2 · Answer 1 · answered Feb 18 '21 at 07:58

2

The pip that you're executing lives in /var/lang/bin/pip whereas the python you're installing lives in the /usr prefix

presumably you could use /usr/bin/pip directly to install, but I'm not sure whether that works correctly with the lambda environment

answered Feb 18 '21 at 07:58

anthony sottile

61,815
15
148
207

I think you're on to something here but I'm pretty sure the container is going to run /var/lang/bin/python for the actual Lambda execution. So perhaps I need to find the devel package for that python? It really doesn't seem to exist anywhere in the yum search. – mmachenry Feb 18 '21 at 08:35
there isn't one as far as I can tell -- a workaround would be to build wheels using the system python and then install those. something like `/usr/bin/pip wheel -r requirements.txt -w wheels && pip install -r requirements.txt --no-index --find-links file://$PWD/wheels` – anthony sottile Feb 18 '21 at 16:12

score 1 · Accepted Answer · answered Nov 22 '21 at 02:21

I was able to duplicate the behavior of the AWS Lambda functionality without their Docker image and it works just fine. This is the Dockerfile I am using.

ARG FUNCTION_DIR="/function/"

FROM python:3.6 AS build
ARG FUNCTION_DIR
ARG NETRC_PATH
RUN echo "${NETRC_PATH}" > /root/.netrc
RUN mkdir -p ${FUNCTION_DIR}
COPY requirements.txt ${FUNCTION_DIR}
WORKDIR ${FUNCTION_DIR}
RUN pip install --upgrade pip && \
    pip install --target ${FUNCTION_DIR} awslambdaric && \
    pip install --target ${FUNCTION_DIR} --no-warn-script-location -r requirements.txt

FROM python:3.6
ARG FUNCTION_DIR
WORKDIR ${FUNCTION_DIR}
COPY --from=build ${FUNCTION_DIR} ${FUNCTION_DIR}
COPY main.py ${FUNCTION_DIR}
ENV MPLCONFIGDIR=/tmp/mplconfig
ENTRYPOINT ["/usr/local/bin/python", "-m", "awslambdaric"]
CMD ["main.handler"]

Python.h: No such file or directory on Amazon Linux Lambda Container

2 Answers2