7

I'm trying to create a new lambda layer to import the zip file with psycopg2, because the library made my deployment package get over 3MB, and I can not see the inline code in my lambda function any more.

I created lambda layer for the following 2 cases with Python 3.7:

  • psycopg2_lib.zip (contains psycopg2, psycopg2_binary.libs and psycopg2_binary-2.8.5.dist-info folders)
  • psycopg2_only.zip which contains only the psycopg2 folder.

I added they new created layer into my lambda function. But, in both cases, my lambda_function throws an error as follows: { "errorMessage": "Unable to import module 'lambda_function': No module named 'psycopg2'", "errorType": "Runtime.ImportModuleError" }

The error seems as if something went wrong with my zip file that they are not recognized. But when it works well in my deployment package.

Any help or reason would be much appriciated. Thanks!

Khue Nguyen
  • 73
  • 1
  • 2
  • 5

6 Answers6

17

not sure if the OP found a solution to this but in case others land here. I resolved this using the following steps:

  1. download the code/clone the git from: https://github.com/jkehler/awslambda-psycopg2
  2. create the following directory tree, if building for python3.7, otherwise replace 'python3.7' with the version choice: mkdir -p python/lib/python3.7/site-packages/psycopg2
  3. choose the python version of interest and copy the files from the folders downloaded in step 1. to the directory tree in step 2. e.g. if building a layer for python 3.7: cp psycopg2-3.7/* python/lib/python3.7/site-packages/psycopg2
  4. create the zip file for the layer. e.g.: zip -r9 psycopg2-py37.zip python
  5. create a layer in the console or cli and upload the zip
Dharman
  • 30,962
  • 25
  • 85
  • 135
Matt
  • 369
  • 3
  • 6
9

If you end up on this page in >= 2022 year. Use official psycopg2-binary https://pypi.org/project/psycopg2-binary/

Works well for me. Just

pip install --target ./python psycopg2-binary

zip -r python.zip python

none none
  • 91
  • 1
  • 1
  • 6
    The documentation says `The binary package is a practical choice for development and testing but in production it is advised to use the package built from sources.` although they do not explain the reasoning for that. – Vlad Miller Aug 19 '22 at 12:39
  • 1
    This doesn't really answer the question about getting it to run on AWS Lambda, though – leo Dec 19 '22 at 19:57
  • @leo you can create a Lambda layer using the zip file (for example in the console) – LoMaPh Jun 13 '23 at 18:57
6

Maintainers of psycopg2 do not recommend using psycopg2-binary because it comes with linked libpq and libssl and others that may cause issues in production under certain circumstances.

I may imagine this being an issue when upgrading postgresql server while bundled libpq is incompatible. I also had issues w/ psycopg2-binary on AWS Lambda running in arm64 environment.

I've resorted to building postgresql and psycopg in Docker running on linux/arm64 platform using public.ecr.aws/lambda/python:3.9 as the base image.

FROM public.ecr.aws/lambda/python:3.9

RUN yum -y update && \
    yum -y upgrade && \
    yum -y install libffi-devel postgresql-devel postgresql-libs zip rsync wget openssl openssl-devel && \
    yum -y groupinstall "development tools" && \
    pip install pipenv

ENTRYPOINT ["/bin/bash"]

The build script is the following and valid for aarch64 platform. Just change path to x86_64 version on Prepare psycopg2 step.

#!/usr/bin/env bash

set -e

PG_VERSION="14.5"

cd "$TERRAFORM_ROOT"
if [ ! -f "postgresql-$PG_VERSION.tar.bz2" ]; then
  wget "https://ftp.postgresql.org/pub/source/v$PG_VERSION/postgresql-$PG_VERSION.tar.bz2"
  tar -xf "$(pwd)/postgresql-$PG_VERSION.tar.bz2"
fi

if [ ! -d "psycopg2" ]; then
  git clone https://github.com/psycopg/psycopg2.git
fi

# Build postgres
cd "$TERRAFORM_ROOT/postgresql-$PG_VERSION"
./configure --without-readline --without-zlib
make
make install

# Build psycopg2
cd "$TERRAFORM_ROOT/psycopg2"
make clean
python setup.py build_ext \
  --pg-config "$TERRAFORM_ROOT/postgresql-14.5/src/bin/pg_config/pg_config"

# Prepare psycopg2
cd build/lib.linux-aarch64-3.9
mkdir -p python/
cp -r psycopg2 python/
zip -9 -r "$BUNDLE" ./python

# Prepare libpq
cd "$TERRAFORM_ROOT/postgresql-$PG_VERSION/src/interfaces/libpq/"
mkdir -p lib/
cp libpq.so.5 lib/
zip -9 -r "$BUNDLE" ./lib

where $BUNDLE is the path to already existing .zip file.

I also tried to statically build psycopg2 binary and link libpq.a, however, I have had quite a lot of issues with missing symbols.

Vlad Miller
  • 2,255
  • 1
  • 20
  • 38
2

From AWS post How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda?:

To create a Lambda deployment package or layer that's compatible with Lambda Python runtimes when using pip outside of Linux operating system, run the pip install command with manylinux2014 as the value for the --platform parameter.

pip install \
    --platform manylinux2014_x86_64 \
    --target=my-lambda-function \
    --implementation cp \
    --python 3.9 \
    --only-binary=:all: --upgrade \
    psycopg2-binary

You can then zip the content of directory my-lambda-function

MaFF
  • 9,551
  • 2
  • 32
  • 41
1

Answer from Maff worked for me but to clarify I had to:

  • put the 3 directories created by pip install into ./python/lib/python3.9/site-packages/
  • rename the .so file in psycopg2 directory to _psycopg.so

Then zip the content of ./python directory

cndsv
  • 11
  • 3
0

I just wrote up my solution to this on Medium, MaFF's answer is correct but problematic if you're running MacOS - pip won't let you target a platform architecture on MacOS! - EDIT: article updated with a retraction on that point, you can indeed build on MacOS.

In short, you need to follow the AWS Guide to build the psycopg-binary2 (or pyscopg-binary, if you're using sqlalchemy v1).

The two supported manylinux architectures for Lambda Layers are manylinux2014_x86_64 for AMD64 and manylinux2014_aarch64 for ARM64.

therightstuff
  • 833
  • 1
  • 16
  • 21