No module named 'psycopg2._psycopg': ModuleNotFoundError in AWS Lambda

Question

I have created a deployment package for AWS Lambda with my python file and the dependencies including sqlalchemy and psycopg2. The code works perfectly in accessing the DB locally. But when I imported this zip file, I am getting the following error.

No module named 'psycopg2._psycopg': ModuleNotFoundError

The stack trace of the error is,

{
  "errorMessage": "No module named 'psycopg2._psycopg'",
  "errorType": "ModuleNotFoundError",
  "stackTrace": [
    [
      "/var/task/DBAccessLamdaHandler.py",
      50,
      "lambda_handler",
      "engine = create_engine(rds_host)"
    ],
    [
      "/var/task/sqlalchemy/engine/__init__.py",
      387,
      "create_engine",
      "return strategy.create(*args, **kwargs)"
    ],
    [
      "/var/task/sqlalchemy/engine/strategies.py",
      80,
      "create",
      "dbapi = dialect_cls.dbapi(**dbapi_args)"
    ],
    [
      "/var/task/sqlalchemy/dialects/postgresql/psycopg2.py",
      554,
      "dbapi",
      "import psycopg2"
    ],
    [
      "/var/task/psycopg2/__init__.py",
      50,
      "<module>",
      "from psycopg2._psycopg import (                     # noqa"
    ]
  ]
}

Any help is appreciable

You don't need to build it from scratch. The repo includes a pre-built package in the psycopg2 folder that you can simply include in the ZIP file that you're going to upload to Lambda (allegedly, I haven't actually used it). — jarmod, Jun 30 '17 at 22:51

jarmod · Accepted Answer · 2018-11-29T15:44:11.407

37

The AWS Lambda runtime environment doesn't include the PostgreSQL libraries so you need to include them within your AWS Lambda upload.

One way to do this is to get them from the jkehler/awslambda-psycopg2 repo at GitHub. Note that you don't need to build this project from scratch as the repo includes a pre-built package in the psycopg2 folder that you can simply include in your Lambda upload.

edited Nov 29 '18 at 15:44

answered Jul 01 '17 at 14:18

jarmod

71,565
16
115
122

This helps. Thank you – Hrushikesh Patel Sep 22 '17 at 13:25
2

How would this solution work with the serverless framework where packages are installed from the requirmets.txt file? – johnny May 18 '18 at 12:43
1

@johnny Take a look at https://github.com/UnitedIncome/serverless-python-requirements – jarmod May 18 '18 at 13:14
3

I've included this library like so however it is still not working. Something I'm going wrong? – Jack Mar 27 '19 at 22:36
@johnny were you able to solve this I am facing the same issue. – ankush reddy Dec 26 '19 at 17:57
@ankushreddy not yet.. we run it in an EC2 for now but planning to move it to Lambda soon... after I figure this issue out. – johnny Dec 27 '19 at 17:16
4

For anyone working with CloudFormation + CodeBuild the following two lines in `buildspec.yml` worked like a charm: `- git clone https://github.com/a-j/awslambda-psycopg2.git && cd awslambda-psycopg2 && git checkout python-3.8-libraries` and `- cp -r awslambda-psycopg2/psycopg2-3.8 ./src && mv ./src/psycopg2-3.8 ./src/psycopg2` – Stefano Messina Jul 31 '20 at 07:52
If you are going to psycopg2-3.6 (- 3.9), do not forget to rename `_psycopg.cpython-39-x86_64-linux-gnu.so` to `_psycopg.so` – Anish Jan 20 '23 at 16:30

score 17 · Answer 2 · edited Jan 22 '20 at 04:51

17

The psycopg2 build library from jkehler/awslambda-psycopg2 was built for python 3.6 and make sure that while uploading your code to AWS lambda, select Python Runtime environment as 3.6, and it should work. I banged my head on this for a full day and then when i changed to 3.6, the import error just vanished.

If you are going to attempt to build it yourself, remember that you must build on a machine or VM with the same architecture as your target at AWS.

edited Jan 22 '20 at 04:51

New Alexandria

6,951
4
57
77

answered May 07 '19 at 15:19

Nikhil Ravindran

392
3
12

Thanks Nikhil, but I tried this and it didn't work on p3.6, could you please help in detail? – GD- Ganesh Deshmukh Oct 11 '19 at 20:08
@ganeshdeshmukh It work with me on python 3.7. I simply copied the whole folder for python 3.7, rename it to `psycopg2`, and zip it with my lambda function file. After uploading the zipped file to aws lambda, it worked as expected. I suppose the procedure for python 3.6 should be similar. – Fanchen Bao Oct 21 '19 at 20:17
It work with me on python 3.6. I have followed the same procedure as mentioned by @FanchenBao – Mahan Feb 24 '20 at 05:26

score 10 · Answer 3 · answered Apr 01 '22 at 13:58

10

Like other answers, psycopg2-binary worked fine for python3.9 (it looks like other package awslambda-psycopg2 was only available for python3.6).

But, if you run on MacOs before sending to aws lambda, you must specify the platform to your pip install like this:

pip3.9 install --platform=manylinux1_x86_64 --only-binary=:all: psycopg2-binary

answered Apr 01 '22 at 13:58

Vincent J

4,968
4
40
50

1

This is the answer. Many thanks. Your answer and @viru s answer make really sense. – aykcandem May 31 '22 at 09:22
This didn't work for me, as pip doesn't appear to support the platform option on MacOS. I just wrote up my solution [on Medium](https://therightstuff.medium.com/building-binary-python-dependencies-for-amd64-and-arm64-lambda-layers-efc6f17e4bf6) You need to follow the [AWS Guide](https://repost.aws/knowledge-center/lambda-python-package-compatible) to build the `psycopg-binary2`, and do so in a Docker container The two supported [manylinux](https://github.com/pypa/manylinux) architectures for Lambda Layers are `manylinux2014_x86_64` for AMD64 and `manylinux2014_aarch64` for ARM64. – therightstuff Jul 27 '23 at 05:53
1

@therightstuff: it looks like the --target=. option is now mandatory, but it works on mac – Vincent J Jul 27 '23 at 06:39
@VincentJ I'm on Ventura 13.4.1, and it doesn't work for me. I wouldn't have ever suspected that to be the problem, but then I read "Note: macOS --platform tags don't work" in the [AWS Guide](https://repost.aws/knowledge-center/lambda-python-package-compatible). When I used exactly the same command I'd been trying from within a Docker container, it did the job. – therightstuff Jul 27 '23 at 08:44
1

@therightstuff aws note is ambiguous but I understand that only manylinux1_x86_64 tag work, not macos nor windows tags. It clearly worked before, but I cannot retest right now I don't have this env anymore. – Vincent J Jul 27 '23 at 11:04
@VincentJ You're absolutely right! I guess i need to post a retraction - I don't know what I was doing differently before but I just got it built and deployed successfully without Docker. Thanks! – therightstuff Jul 27 '23 at 17:51

score 6 · Answer 4 · edited Feb 04 '22 at 18:58

6

Latest as of 26/MAR/2020

I was skeptical about depending on a third-party library for my production code. On research the following works,

The issue happens only when the packages are built from MAC OS.

I can confirm today that the issue is fixed when I build the package from Centos 7 ( AWS AMI )

The following is my approach

requirement.txt

psycopg2-binary==2.8.4

Build process

pip install -r requirements.txt --target .

Lambda code is in the root directory

+-- lambda_function.py
+-- psycopg2
    +-- psycopg2 files

Zip the directory and test the code in lambda works.

The only additional step is to build the package in Linux env instead of macOS using the Docker container. An example can be found here: Deploy AWS Amplify Python Lambda from macOS with Docker

edited Feb 04 '22 at 18:58

Abdullah Khawer

4,461
4
29
66

answered Mar 26 '20 at 20:52

viru

2,081
19
23

1

How is this related to the AWS Lambda runtime? Packages are installed by `pip` on the `aws/codebuild/standard:2.0` image – Stefano Messina Jul 31 '20 at 07:07
2

Python runtime checks for the local library first. Give this a shot - works for me in production till date – viru Jul 31 '20 at 19:14

score 4 · Answer 5 · answered Apr 12 '23 at 07:33

The psycopg2 python package requires Postgres libraries which is missing in AMI used by the AWS Lambda function.

The easiest solution is to use a pre-compiled package aws-psycopg2 which has the required libraries. The package is present in the PyPi repository.

pip install aws-psycopg2

After this, just do the import of psycopg2 as usual in your program. Sample import is given below.

import psycopg2 With this package, my Lambda function started working perfectly.

Thank you I did not know `aws-psycopg2` existed. I swear I hate working with Amazon Linux sometimes — Amon, Aug 22 '23 at 03:53

score 3 · Answer 6 · answered May 23 '22 at 17:12

3

Thanks to AWS Lambda Layers we can include a ready compiled psycopg2 layer for our selected Python versions directly to our Lambda function. Please use the version you need from this Github repo. Be careful to use the correct Python version and Region in Lambda and layer creation.

answered May 23 '22 at 17:12

alexopoulos7

794
7
27

This makes my life easier.. LOL ! Thanks !! – Yvette Lau Nov 03 '22 at 17:47

score 2 · Answer 7 · edited Jun 20 '20 at 09:12

2

For Windows (Looks like in any OS which is not build on Amazon AMI or Centos), the easiest fix is to use

psycopg2 Python Library for AWS Lambda

which you can find https://github.com/jkehler/awslambda-psycopg2

edited Jun 20 '20 at 09:12

Community

1
1

answered May 14 '20 at 13:28

Subrata Fouzdar

724
5
17

score 2 · Answer 8 · answered Sep 20 '22 at 11:47

I have faced a similar issue. I tried to create the layers using AWS Cloud shell it worked with Python3.8. But failed in Python3.9 though.

Please change the region name as per your AWS Region

sudo amazon-linux-extras install python3.8

curl -O https://bootstrap.pypa.io/get-pip.py

python3.8 get-pip.py --user

mkdir python

python3.8 -m pip install --platform=manylinux1_x86_64 --only-binary=:all: pandas numpy pymysql psycopg2-binary SQLAlchemy -t python/

zip -r layer.zip python

aws lambda publish-layer-version \
 --layer-name mics-layer \
 --description "Pandas Numpy psycopg2-binary SQLAlchemy pymysql" \
 --zip-file fileb://layer.zip \
 --compatible-runtimes python3.8 python3.9 \
 --region eu-west-1

Ziggity · Answer 9 · 2022-05-07T11:08:43.667

1

I've had issues getting SQLAlchemy to run on AWS Lambda despite the fact that I have tried multiple psycopg2 versions, but what eventually solved it for me was to use an older Python version. I went from Python 3.9 to 3.7 and it was finally able to run (Using psycopg2-binary 2.8.4, but I didn't try other versions or the none binary version with 3.7)

edited May 07 '22 at 11:08

answered May 07 '22 at 10:19

Ziggity

23
5

score 0 · Answer 10 · edited Sep 10 '20 at 08:16

instead of using psycopg2, try using pg8000 in command prompt go the directory that you are currently working(example F:\example) pip install pg8000 -t. (-t. is for installing pg8000 in the directory that you are working)

// handler.py import pg8000

database=''
host=''
port=''
user=''
password=''
conn = pg8000.connect(database=database, host=host, port=port, user=user, 
password=password)

def lambda_function(event, context):
   .
   .
   .

after writing your code zip the code and upload in your aws lambda.

It worked for me!!!

lmiguelmh · Answer 11 · 2023-01-30T14:50:44.450

If you are using CDK you can use SQLAlchemy on AWS Lambda by bundling with Docker and providing the right pip install command:

Python 3.8.

    sqlachemy_layer = lambda_.LayerVersion(
        scope=self,
        id=LAYERS_SQLALCHEMY_LAYER,
        layer_version_name=LAYERS_SQLALCHEMY_LAYER,
        code=lambda_.Code.from_asset(
            str(pathlib.Path(__file__).parent.joinpath("dependencies").resolve()),
            bundling=BundlingOptions(
                image=lambda_.Runtime.PYTHON_3_8.bundling_image,
                command=[
                    "bash", "-c",
                    "pip install -r requirements-sqlalchemy.txt -t /asset-output/python && cp -au . /asset-output/python",
                ]
            )
        ),
        compatible_runtimes=[
            lambda_.Runtime.PYTHON_3_8,
        ],
    )

Python 3.9

    sqlachemy_layer = lambda_.LayerVersion(
        scope=self,
        id=LAYERS_SQLALCHEMY_LAYER,
        layer_version_name=LAYERS_SQLALCHEMY_LAYER,
        code=lambda_.Code.from_asset(
            str(pathlib.Path(__file__).parent.joinpath("dependencies").resolve()),
            bundling=BundlingOptions(
                image=lambda_.Runtime.PYTHON_3_9.bundling_image,
                command=[
                    "bash", "-c",
                    "pip3.9 install --platform=manylinux1_x86_64 --only-binary=:all: -r requirements-sqlalchemy.txt -t /asset-output/python && cp -au . /asset-output/python",
                ]
            )
        ),
        compatible_runtimes=[
            lambda_.Runtime.PYTHON_3_9,
        ],
    )

The requirements-sqlalchemy.txt:

psycopg2-binary
sqlalchemy

And the file structure:

dependencies/
- requirements-sqlalchemy.txt
my_stack.py

therightstuff · Answer 12 · 2023-07-27T18:16:56.187

I just wrote up my solution to this on Medium, a lot of the suggestions here didn't work for me ~~and one of the main reasons is that pip won't let you target a platform architecture on MacOS!~~ - EDIT: article updated with a retraction on that point, you can indeed build on MacOS.

In short, the most straightforward method (in my opinion) is to follow the AWS Guide to build the psycopg-binary2 (or pyscopg-binary, if you're using sqlalchemy v1)~~, doing so in a Docker container if you're using a Mac~~.

The two supported manylinux architectures for Lambda Layers are manylinux2014_x86_64 for AMD64 and manylinux2014_aarch64 for ARM64.

No module named 'psycopg2._psycopg': ModuleNotFoundError in AWS Lambda

12 Answers12

Linked