How to locally debug dependencies in a lambda layer?

Question

I'm creating a lambda layer from a dockerfile that installs python packages to a directory and zips the result.

FROM amazonlinux

WORKDIR /
RUN yum update -y

# Install Python 3.7
RUN yum install python3 zip -y

RUN pip3.7 install --upgrade pip

# Install Python packages
RUN mkdir /packages
RUN echo "opencv-python" >> /packages/requirements.txt

RUN mkdir -p /packages/opencv-python-3.7/python/lib/python3.7/site-packages
RUN pip3.7 install -r /packages/requirements.txt -t /packages/opencv-python-3.7/python/lib/python3.7/site-packages


# Create zip files for Lambda Layer deployment
WORKDIR /packages/opencv-python-3.7/
RUN zip -r9 /packages/cv2-python37.zip .
WORKDIR /packages/
RUN rm -rf /packages/opencv-python-3.7/

For this Dockerfile I can successfully deploy.
Now I want to add more libraries¹ but despite successful docker build and upload, there is an error when executing the lambda function (numpy not found). I would like an easier way to debug this than changing the Docker file, building, extracting and uploading the zip file and pressing 'test' in the AWS management console.

I've tried to run the docker container locally and just install the packages there and see if everything can be imported in a python shell but I cannot even recreate the original this way:

bash-4.2# pip3.7 install opencv-python
Collecting opencv-python
  Using cached opencv_python-4.4.0.42-cp37-cp37m-manylinux2014_x86_64.whl (49.4 MB)
Collecting numpy>=1.14.5
  Using cached numpy-1.19.1-cp37-cp37m-manylinux2010_x86_64.whl (14.5 MB)
Installing collected packages: numpy, opencv-python
Successfully installed numpy-1.19.1 opencv-python-4.4.0.42
bash-4.2# python3.7
Python 3.7.8 (default, Jul 24 2020, 20:26:49) 
[GCC 7.3.1 20180712 (Red Hat 7.3.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib64/python3.7/site-packages/cv2/__init__.py", line 5, in <module>
    from .cv2 import *
ImportError: libGL.so.1: cannot open shared object file: No such file or directory

How can I figure out the right dependencies on my local machine?

Update

I made it work with the versions below, but it would still be interesting to know how to test this locally.

¹ specifically I want the following packages:

opencv-python==3.4.3.18
scipy==1.4.1
scikit-learn==0.22.2.post1

@Marcin I need the following libraries: `opencv-python==3.4.3.18`, `scipy==1.4.1`, `scikit-learn==0.22.2.post1` — peer, Aug 22 '20 at 12:12
I created such layer, but its too big. The lowest I could go down with it was about 60MB, which is still too big for a layer (max 50MB). So even if you solve your issues, you may end up with a massive layer. — Marcin, Aug 22 '20 at 13:44

score 1 · Answer 1 · answered Aug 22 '20 at 12:32

The problem you have here is that when opencv is installed its dependencies aren't going to be installed to your -t target location. They're getting installed to the default pip install location of <somewhere>/site-packages/ in the Docker image.

So when you end up zipping up your target location you're missing all of the dependencies. I would solve this by not providing a target to pip when you install opencv. Install it as you would any other package.

From within your Docker image call python -m site --user-site to get the pip install location.

Modify your Docker commands to zip up that entire directory after opencv has been installed, then use that for your zip to Lambda.

score 0 · Answer 2 · answered Dec 03 '20 at 00:35

I was able to use the more recent versions of scikit-learn and cv2 with the following approach that would automate the deployment process, and automatically reduce the size of the packages by removing unnecessary files **/*.py[c|o], **/__pycache__*, **/*.dist-info*:

I had to package both cv2 and scipy, where the package size was a huge problem, and I came to the following solutions in the end.

Using the serverless-python-requirements package on Serverless helped me streamline this whole process and reduce the package size as well. Would definitely recommend checking it out.

This is the guide that I followed

Serverless python-requirements plugin

Make sure to leave the strip flag to false to avoid stripping binaries which leads to the problem "ELF load command address/offset not properly aligned",

This is what my final serverless.yml came out to be which gave me the results I wanted to package sklearn + cv2 as a layer:

custom:
  pythonRequirements:
    dockerizePip: true
    useDownloadCache: true
    useStaticCache: false
    slim: true
    strip: false
    layer:
      name: ${self:provider.stage}-cv2-sklearn
      description: Python requirements lambda layer
      compatibleRuntimes:
        - python3.8
      allowedAccounts:
        - '*'

requirements.txt:

opencv-python-headless==4.4.0.42
scikit-learn==0.23.2

How to locally debug dependencies in a lambda layer?

Update

2 Answers2

Linked