Python at AWS Lambda: `requests` from botocore.vendored deprecated, but `requests` not available

Question

I've got a Python script for an AWS Lambda function that does HTTP POST requests to another endpoint. Since Python's urllib2.request, https://docs.python.org/2/library/urllib2.html, can only handle data in the standard application/x-www-form-urlencoded format and I want to post JSON data, I used the Requests library, https://pypi.org/project/requests/2.7.0/.

That Requests library wasn't available at AWS Lambda in the Python runtime environment, so had to be imported via from botocore.vendored import requests. So far, so good.

Today, I get a deprecation warning on that:

DeprecationWarning: You are using the post() function from 'botocore.vendored.requests'.
This is not a public API in botocore and will be removed in the future.
Additionally, this version of requests is out of date. We recommend you install the
requests package, 'import requests' directly, and use the requests.post() function instead.

This was mentioned in this blog post from AWS too: https://aws.amazon.com/blogs/developer/removing-the-vendored-version-of-requests-from-botocore/.

Unfortunately, changing from botocore.vendored import requests into import requests results in the following error:

No module named 'requests'

Why is requests not available for the Python runtime at AWS Lambda? And how can I use / import it?

Are you using serverless framework? – Yugandhar Chaudhari Oct 18 '19 at 07:35 — Yugandhar Chaudhari, Oct 18 '19 at 07:35
That's a surprising, but excellent guess. Yes, I am. – Jochem Schulenklopper Oct 18 '19 at 07:36 — Jochem Schulenklopper, Oct 18 '19 at 07:36

score 32 · Answer 1 · answered Oct 18 '19 at 10:34

32

I succeeded sending HTTP POST requests using the urllib3 library, which is available at AWS Lambda without the requirements for additional installation instructions.

import urllib3

http = urllib3.PoolManager()

response = http.request('POST',
                        url,
                        body = json.dumps(some_data_structure),
                        headers = {'Content-Type': 'application/json'},
                        retries = False)

answered Oct 18 '19 at 10:34

Jochem Schulenklopper

6,452
4
44
62

I'm getting 403 error when i use this method. Can't figure out what is the problem. Any solution? – IvanAK Mar 10 '20 at 15:31
2

You get 403 errors on which URL? And that error isn't present with for example `curl -X POST ` or `curl --data "pi:3.14159265" `? 403 means the server understood the request, but is refusing to authorize it... so it's more likely that the problem is with your request against that endpoint than with the `urllib3` library. – Jochem Schulenklopper Mar 13 '20 at 09:39
1

The vendored versions of requests and urllib3 are no longer being used and have been replaced with a direct dependency on upstream urllib3 and requests is no longer a dependency of botocore. While these vendored dependencies are still in the botocore package they should not be used as they will be removed in the future. Any code that imports from botocore.vendored.requests.* should be updated accordingly. https://botocore.amazonaws.com/v1/documentation/api/latest/index.html – Yash P Shah Feb 12 '21 at 13:42

score 9 · Answer 2 · answered Mar 31 '20 at 02:10

9

Check out the instructions here: https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-dependencies

All you need to do is download the requests module locally, then include it in your Lambda function deployment package (ZIP archive).

Example (if all your Lambda function consisted of was a single Python module + requests module):

$ pip install --target ./package requests
$ cd package
$ zip -r9 ${OLDPWD}/function.zip .
$ cd $OLDPWD
$ zip -g function.zip lambda_function.py
$ aws lambda update-function-code --function-name my-function --zip-file fileb://function.zip

answered Mar 31 '20 at 02:10

Mark

829
11
22

1

More info: https://aws.amazon.com/premiumsupport/knowledge-center/build-python-lambda-deployment-package/ – Mark Mar 31 '20 at 02:16
This is by far the easiest and most straightforward solution, and it's the one recommended by AWS Lambda in their documentation. With three short commands, you've got your ZIP file ready for upload, which you can also upload manually if you like. – Sean McCarthy Dec 22 '20 at 16:41
What if you are working on a windows machine ? :( – SimonB May 18 '22 at 20:02
@SimonB try [Cygwin](https://cygwin.com/index.html) or just look up the Command Prompt or PowerShell counterparts to the commands above. – Mark May 19 '22 at 01:57

score 9 · Answer 3 · answered Jun 19 '20 at 00:12

9

Answer 2020-06-18

I found a nice and easy way to use requests inside AWS Lambda functions!

Open this link and find the region that your function is using:
https://github.com/keithrozario/Klayers/tree/master/deployments/python3.8/arns

Open the .csv related to your region and search for the requests row.
This is the ARN related to requests library:
arn:aws:lambda:us-east-1:770693421928:layer:Klayers-python38-requests:6

So now in your lambda function, add a layer using the ARN found.
Obs.: make sure your Python lambda function runtime is python3.8.

answered Jun 19 '20 at 00:12

igorkf

3,159
2
22
31

6

As owner of the repo, I approve of this comment :) – keithRozario Jun 19 '20 at 11:21
4

OK, that is using an AWS Lambda layer that happens to have the `requests` package installed. Instead of using a layer from an 'unknown' source, it'd be better to install the package yourself, or use an AWS-distributed layer containing that package. – Jochem Schulenklopper Jun 24 '20 at 20:45

Yugandhar Chaudhari · Accepted Answer · 2020-01-17T05:29:38.913

5

If you are using serverless framework

Specify the plugin in serverless.yml

plugins:
- serverless-python-requirements

At the directory root create file requirements.txt

requirements.txt

requests==2.22.0

This will install the requests and packages mentioned.

edited Jan 17 '20 at 05:29

answered Oct 18 '19 at 07:44

Yugandhar Chaudhari

3,831
3
24
40

It should be clarified that this only works with the Serverless framework. – Esteban Jan 17 '20 at 01:27
@Esteban ok will do it – Yugandhar Chaudhari Jan 17 '20 at 05:28

score 2 · Answer 5 · answered Oct 18 '19 at 07:39

2

requests is NOT part of core python. See https://docs.aws.amazon.com/en_pv/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html about packaging a Lambda having external dependencies (in your case the requests library)

answered Oct 18 '19 at 07:39

balderman

22,927
7
34
52

I know Requests isn't _in_ core Python, and thus not 'standard' available in an AWS Lambda runtime. After following AWS's advice, "We recommend you install the requests package, 'import requests' directly, and ..." I ran into the (stated) problem that apparently the package isn't also available _for_ the Python runtime of AWS Lambda (at least not at the moment of writing). – Jochem Schulenklopper Nov 14 '19 at 19:39

score 1 · Answer 6 · answered Jan 17 '20 at 01:35

Amazon's Serverless Application Model (SAM) provides a build command that can bundle arbitrary python dependencies into the deployment artifact.

To be able to use the requests package in your code, add the dependency to your requirements.txt file:

requests==2.22.0

then run sam build to get an artifact that vendors requests. By default, your artifacts will be saved to the .aws-sam/build directory but another destination directory can be specified with the --build-dir option.

Consult SAM's documentation for more info.

score 0 · Answer 7 · answered Apr 06 '21 at 09:33

Here's my redneck solution that works with any library, using an AWS Lambda Layer:

This has the advantage that you don't have to trust any 3rd party layers, because you can easily make it yourself.

Go to your local python's Lib/site-packages (python install location or your venv)
Copy whichever libraries you need (e.g. "requests") into a folder named "python"
Zip this folder
Create an AWS Lambda Layer, and upload the zip into it
Add this layer in your lambda function
Import your libraries as usual, and keep coding as if nothing happened

score -9 · Answer 8 · edited Oct 18 '19 at 08:21

-9

pip install requests
and then
import requests
to use.

edited Oct 18 '19 at 08:21

Moon

4,014
3
30
66

answered Oct 18 '19 at 07:39

vimin

1

4

Thanks. `pip install requests` installs the library locally, but not automatically when deploying the script to AWS Lambda. So, this solution works locally, but not in AWS. – Jochem Schulenklopper Oct 18 '19 at 09:11

Python at AWS Lambda: `requests` from botocore.vendored deprecated, but `requests` not available

8 Answers8

Linked