Adding a Geo Django project to AWS Lambda using Zappa. Getting OSError: cannot open shared object file: No such file or directory

Question

I am trying to deploy my Geo-Django app to Zappa 1st I got

django.core.exceptions.ImproperlyConfigured: Could not find the GDAL library 
(tried "gdal", "GDAL", "gdal2.2.0", "gdal2.1.0", "gdal2.0.0", "gdal1.11.0", 
"gdal1.10.0", "gdal1.9.0"). Is GDAL installed? If it is, try setting 
GDAL_LIBRARY_PATH in your settings.

Then I followed this link and added the below

I set these environment variables in my AWS Lambda console:

"LD_LIBRARY_PATH": "/tmp/code/lib/",
"PROJ_LIB": "/tmp/code/lib/proj4/",

and in my (Django) app's settings file, I set:

GDAL_LIBRARY_PATH = "/tmp/code/lib/libgdal.so.20.1.3"
GEOS_LIBRARY_PATH = "/tmp/code/lib/libgeos_c.so.1"

Now I am getting the error

OSError: /tmp/code/lib/libgdal.so.20.1.3: cannot open shared object file: No such file or directory

How can I fix this ?

Summary of what I have done

$ pip install zappa
$ zappa init
$ zappa deploy prod

Below is my zappa_settings.json

{
    "prod": {
        "aws_region": "us-east-1",
        "django_settings": "Cool.settings",
        "profile_name": "default",
        "project_name": "cool",
        "runtime": "python3.6",
        "s3_bucket": "coolplaces-t47c5adgt",
        "extra_permissions": [{
            "Effect": "Allow",
            "Action": ["rekognition:*"],
            "Resource": "*"
        }]
    }
}

score 1 · Answer 1 · answered Jan 22 '19 at 14:35

1

I'm assuming you've bundled the two required libraries to your Lambda deployment package.

In the Lambda container, that gets extracted inside the /var/task directory. That directory is already in the LD_LIBRARY_PATH. Try setting the other necessary ENVVARS to /var/task as well.

answered Jan 22 '19 at 14:35

Milan Cermak

7,476
3
44
59

Thank you so much for your answer. Could you dumb it down for me a bit. Also can you explain what you mean by "I'm assuming you've bundled the two required libraries to your Lambda deployment package." I have added more info in the question above under Summary of what I have done – Samir Tendulkar Jan 23 '19 at 03:13
The `libgdal` and `libgeos_c` need to be compiled for a 64-bit Linux system. That's the Lambda execution environment. Then they need to be a part of the deployment package (which is just a zip archive). I'm not overly familiar with zappa, but when you do zappa deploy, it creates this package for you and uploads it to S3, from where Lambda downloads, unarchives and runs it. So you need to make sure the two required libraries get zipped into this zappa-built archive. If you're not sure they are part of it, you can always just download the zip file that zappa uploads to your S3 bucket and check. – Milan Cermak Jan 23 '19 at 13:32

score 1 · Accepted Answer · answered Jan 24 '19 at 06:33

Ok I think I almost got it This is what I did

zappa undeploy prod
pip uninstall zappa
delete the zappa_settings.json file

Step 1)

$ pip install git+git://github.com/bahoo/Zappa.git#egg=zappa

Step 2) then type zappa init then you will see it automatically creates a file called zappa_settings.json

Add to your zappa_settings.json:

"project_directory": "/tmp/code", (Copy this as is no "," if this is the last statement) "slim_handler": true (Use this if it gives you a error saying. Your file is too big. Which I am sure it will as the lib file size is 107.1 MB. Also no "," since this was my last statement in my zappa_settings.json no quotes for true)

Step 3) made a directory called lib in my root directory and copied the files to it. (Copy these files) See images below https://i.stack.imgur.com/yOLsK.jpg

Step4) In your AWS lambda console.

"LD_LIBRARY_PATH": "/tmp/code/lib/",
"PROJ_LIB": "/tmp/code/lib/proj4/",

Remember do not replace code keep it as is https://i.stack.imgur.com/Gxkki.jpg

Step5) add these to your Django settings.py: (Do not replace code with your path keep it as is)

GDAL_LIBRARY_PATH = "/tmp/code/lib/libgdal.so.20.1.3"
GEOS_LIBRARY_PATH = "/tmp/code/lib/libgeos_c.so.1"

Step 6) Finally, zappa deploy dev or zappa deploy prod whatever stage you want

Step 7) If it gives you errors do zappa tail it will give you all logs and tell you what the error is fix them and do zappa update

This was successful. Thank you bahoo so much for your help and taking the time to dumb it down for me. Also Thank you so much for making geodjango work on zappa

It gave me a error saying bad request told me to add a long amazon link to my allowed host. Did that. Now the next error was to add my data_base. I am doing that. But I feel I got it

For more details refer to https://github.com/Miserlou/Zappa/issues/985

Have you managed to find a more space-efficient way of doing this? a 100MB lambda file seems very large, I'm having trouble just getting the lambda to respond before the 30 second API Gateway cutout. — BenDog, Jul 08 '19 at 09:57
@ In AWS there is a thing called slim handler that helps you reduce the size of the files Do `"slim_handler": true` check the link above that helps. The large file has no bad effects on performance — Samir Tendulkar, Aug 23 '19 at 10:03

Adding a Geo Django project to AWS Lambda using Zappa. Getting OSError: cannot open shared object file: No such file or directory

2 Answers2