4

I've resisted writing this post for over a day because I was determined to find a solution from one of the many posts out there with answers to similar issues but none of them seem to work.

My issue...

When I run sam build && sam local invoke, it works just fine. But, when I run a python -m pytest tests/ -v, I get the following error.

Traceback:
../../../.pyenv/versions/3.8.6/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/unit/test_handler.py:3: in <module>
    from get_devices import app
get_devices/app.py:4: in <module>
    import aurora
E   ModuleNotFoundError: No module named 'aurora'

The aurora module is one I wrote that allows me to connect with AWS Aurora. It's in a layer so I can share it across my various lambdas.

Here is what I've tried

I've added empty __init__.py files. I've also tried adding setup.py and conftest.py to my root. I've tried playing around sys.path and tirelessly tried importing my aurora module with relative and absolute paths to no avail. I found the sam-python-with-layers project but when I run pytest on it, I get the same error (just with different module name).

My question...

Is SAM or the combo of SAM + PyEnv preventing pytest from finding the aurora module? How can I get around this? Any assistance would be greatly appreciated.

Project/Code...

# project tree

├── __init__.py
├── get_devices
│   ├── __init__.py
│   ├── app.py
│   ├── requirements.txt
├── template.yaml
├── tests
│   ├── __init__.py
│   └── unit
│       ├── __init__.py
│       └── test_handler.py
└── utils_layer
    ├── __init__.py
    ├── python
    │   ├── __init__.py
    │   ├── aurora.py
    │   ├── pg8000

This is my aurora.py file...

# aurora.py

import boto3
import os
import pg8000
import ssl

__location__ = os.path.realpath(os.path.join(os.getcwd(), os.path.dirname(__file__)))

def make_conn():   
    conn = None
    try:                                          
        conn = pg8000.connect(
            database=db_name, 
            user=db_user, 
            password=db_password, 
            host=db_host,
            port=db_port,
            ssl_context=ssl_context                   
            )  
    except Exception as e:
        print(f'Connection error: {e}')
    return conn

This is my app.py where I import the aurora module...

# app.py

import aurora

def lambda_handler(event, context):
    conn = aurora.make_conn()
    if conn:
        print('connected')

This is my template.yaml file...

# template.yaml

Globals:
  Function:
    Runtime: python3.8
    MemorySize: 256
    Timeout: 60
    Layers:
      - !Ref UtilsLayer

Resources:
  GetDevicesFunction:
    Type: AWS::Serverless::Function
    Properties:      
      CodeUri: get_devices/
      Handler: app.lambda_handler                
      Policies:
        -
          Version:  '2012-10-17'
          Statement:                              
            Effect: Allow 
            Action: rds-db:connect
            Resource: !FindInMap [ResourcesName, !Ref MyEnvironment, databaseResourceId]                             
      Events:
        GetDevicesApiEvent:
          Type: Api
          Properties:         
            Path: /devices
            Method: GET   
            
  UtilsLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      Description: Utils layer
      ContentUri: utils_layer/
      CompatibleRuntimes:        
        - python3.8
      RetentionPolicy: Delete
vvvvv
  • 25,404
  • 19
  • 49
  • 81
hugo
  • 1,175
  • 1
  • 11
  • 25
  • Hello, i am also facing this problem. can you please tell me what is in this this pg8000 in your layer . because in my case third party library not found in layers like my project ModuleNotFoundError: No module named 'oracledb' . – Mahmudul Hassan Dec 15 '22 at 06:37

2 Answers2

3

I tried making your project run pytest from root dir, 3 options each one can do the work,

  • remove __init__ from root dir. from utils_layer.python.aurora import aurora
  • export PYTHONPATH=$PWD:$PYTHONPATH. from utils_layer.python.aurora import aurora
  • add to the import path the full path from the root dir. from <rootdir>.utils_layer.python.aurora import aurora
Avihay Tsayeg
  • 444
  • 4
  • 10
  • Thanks for the response, Avihay. I tried all of your suggestions and I'm still not finding success. But, I'm not sure I totally understand so I have some further questions for you. In all your suggestions, you have ```utils_layer.utils_layer```...why do you have ```utils_layer``` twice? Second, in your last suggestion you have `````` but my root has a dash in it (e.g. my-root) which ends up failing. Any further suggestions. – hugo Dec 12 '20 at 15:28
  • https://stackoverflow.com/users/1264589/hugo The duplication is a mistake, I created similar project as yours and got the error, All of these 3 options fixed that, as well opened new ASM project and it worked just fine. The other project you mentioned I download it and adding the specific folder of the file to python path do the job, If you want you can send me copy o the project I will try to do it as well. – Avihay Tsayeg Dec 12 '20 at 15:40
  • Thanks Avihay Tsayeg! I got it working and I updated the original post to include the solution. Hope this helps someone else. – hugo Dec 12 '20 at 22:13
0

Thanks to Avihay Tsayeg, I found that the solution was to add the path to my utils_layer/python directory to my PYTHONPATH, like this:

export PYTHONPATH=$PWD/utils_layer/python

This answer was posted as an edit to the question SAM app works but running pytest throws...ModuleNotFoundError: No module named 'aurora' by the OP hugo under CC BY-SA 4.0.

vvvvv
  • 25,404
  • 19
  • 49
  • 81