4

I have a python package which looks like the following:

package/
├── __init__.py
├── PyMySQL-0.7.6-py2.7.egg
├── pymysql
├── PyMySQL-0.7.x.pth  
└── tests.py

The folder structure cannot be changed because it is from a third party library.

The contents of the .pth file are 

import sys; sys.__plen = len(sys.path)
./PyMySQL-0.7.6-py2.7.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

Whats the best way to include pymysql in tests.py

I obviously can't use from PyMySQL-0.7.6-py2.7.egg because the folder name contains dots.

P.S. absolute paths are not known because this code is supposed to be deployed to AWS lambda

Alexis.Rolland
  • 5,724
  • 6
  • 50
  • 77
vineet sinha
  • 317
  • 1
  • 4
  • 12
  • I would try to use the `importlib.import_module` function. But this will not work, you will need to copy the importlib (name it something like `myimportlib`) package and change the path splitting algorithm of `_bootstrap._find_and_load_unlocked`. Sounds a cool project but I would not like to maintain the code! I paste the url for the source code, be aware that this is on Python tip and needs python 3.6 to run. https://github.com/python/cpython/tree/master/Lib/importlib – user983716 Sep 19 '16 at 01:25
  • yeah definitely not what I was going for :D – vineet sinha Sep 26 '16 at 22:34
  • But apparently it is the only solution if you cannot avoid the problem. Personally I believe it would be cool to do and maybe useful for others. Keep me informed if you go with an implementation, I would like to see how to test such a feature. – user983716 Sep 28 '16 at 12:24

1 Answers1

5

The problem is that the module's .pth file is not being used. Amazon's build instructions for Lambda have you install your dependencies into the same root directory as your Lambda function file. This directory is not configured as a "site directory" (e.g. like lib/pythonX.Y/site-packages/ in a virtualenv, or for global installations, /usr/lib/pythonX.Y/site-packages (CentOS) or /usr/local/lib/pythonX.Y/dist-packages (Ubuntu)), and so .pth files are not evaluated.

You can configure the current working directory as a site directory. The site module documentation describes creating a usercustomize.py as the convention for adding custom site directories, but Lambda presents another hiccup: to be evaluated automatically, the file is expected to be found in ~/.local/lib/pythonX.Y/site-packages. I created a usercustomize.py in the root of my Lambda .zip file and imported it first in my Lambda function handler file. I was then able to import modules that use .pth files to configure dot notation.

usercustomize.py

import site
import os

site.addsitedir(os.getcwd())

lambda_handler.py

import usercustomize
from foo.bar import baz

Using this method, you could install the dependencies into a site-packages dir in the root of your distribution zip file (os.getcwd() becomes os.path.join(os.getcwd(), 'site-packages')), making the structure a little cleaner, if that matters to you.

Hijinks
  • 311
  • 2
  • 5