How to create a layer in lambda function

Question

Steps followed to create

1. Create a virtual environment and activate it
2. pip install elasticsearch
3. Zip the folder inside site-packages.zip

4.Create Layer in AWS (say name is elastic)

5. Add the code below in lambda_handler and add the Layer elastic

Below is code

import json
from elasticsearch import Elasticsearch, RequestsHttpConnection
def lambda_handler(event, context):
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

Still I got "errorMessage": "Unable to import module 'lambda_function': No module named 'elasticsearch'",

Answer 1

If I may, I would like to recommend an alternative technique which has never failed me. The technique includes docker tool described in the recent AWS blog:

• How do I create a Lambda layer using a simulated Lambda environment with Docker?

Thus for this question, I verified it using elasticsearch as follows:

1. Create empty folder, e.g. mylayer.

2. Go to the folder and create requirements.txt file with the content of

elasticsearch

3. Run the following docker command (may adjust python version to your needs):

docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"

4. Create layer as zip:

zip -r elastic.zip python > /dev/null

5. Create lambda layer based on elastic.zip in the AWS Console. Don't forget to specify Compatible runtimes to python3.8.

6. Test the layer in lambda using the following lambda function:

import json

from elasticsearch import Elasticsearch, RequestsHttpConnection

def lambda_handler(event, context):
    # TODO implement
    
    print(dir(Elasticsearch))
    
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

The function executes correctly:

['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__enter__', '__eq__', '__exit__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'bulk', 'clear_scroll', 'close', 'count', 'create', 'delete', 'delete_by_query', 'delete_by_query_rethrottle', 'delete_script', 'exists', 'exists_source', 'explain', 'field_caps', 'get', 'get_script', 'get_script_context', 'get_script_languages', 'get_source', 'index', 'info', 'mget', 'msearch', 'msearch_template', 'mtermvectors', 'ping', 'put_script', 'rank_eval', 'reindex', 'reindex_rethrottle', 'render_search_template', 'scripts_painless_execute', 'scroll', 'search', 'search_shards', 'search_template', 'termvectors', 'update', 'update_by_query', 'update_by_query_rethrottle']

Answer 2

As one option is already mentioned by @Marcin which is required Docker to be installed in the target machine. If you want to skip docker then you can use below script to create and publish layer to AWS.

All you need

./creater_layer.sh <package_name> <layer_name>

./creater_layer.sh elasticsearch my-layer

script creater_layer.sh

path="app"
package="${1}"
layername="${2}"
mkdir -p $path
pip3 install "${package}" --target "${path}/python/lib/python3.8/site-packages/"
cd $path && zip -r ../lambdalayer.zip .
aws lambda publish-layer-version --layer-name "${layername}" --description "My layer" --license-info "MIT" --zip-file "fileb://../lambdalayer.zip" --compatible-runtimes python3.8