30

What I want to achieve

To scrape an website using AWS Lambda and save the data on S3.

The issues I'm having

When I execute Lambda, the following error message appears.

{ "errorMessage": "Unable to import module 'lambda_function': cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (/opt/python/urllib3/util/ssl_.py)", "errorType": "Runtime.ImportModuleError", "requestId": "fb66bea9-cbad-4bd3-bd4d-6125454e21be", "stackTrace": [] }

Code

The minimum Lambda code is as follows.

import requests
import boto3 

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    upload_res = s3.put_object(Bucket='horserace-dx', Key='/raw/a.html', Body='testtext')
    
    return event

An layer was added to the Lambda. Files were save in python folder using the commands below , frozen in a zip file, then uploaded to AWS Lambda as a layer.

!pip install requests -t ./python --no-user
!pip install pandas -t ./python --no-user
!pip install beautifulsoup4 -t ./python --no-user
  • The bucket horserace-dx exists
  • The folder raw exists
  • The role of the Lambda is properly set. It can read from and write to S3
  • The runtime of the Lambda is Python 3.9. The python version of the local computer is 3.9.13.

What I did so far

I google "cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'" and found some suggestions. I made the layer with the following code and tried again in vain.

!pip install requests -t ./python --no-user
!pip install pandas -t ./python --no-user
!pip install beautifulsoup4 -t ./python --no-user
!pip install urllib3==1.26.15 -t ./python --no-user

So what should I do to achieve what I want to achieve? Any suggestions would be greatly appreciated.

dixhom
  • 2,419
  • 4
  • 20
  • 36

6 Answers6

27

cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'

You are encountering this issue because you’re using botocore which does not support urllib3 2.0 yet.

Since you’re deploying to AWS Lambda, you’ll need to explicitly pin to urllib3<2 in your project to ensure urllib3 2.0 isn’t brought into your environment.
(Source)

urllib3<2

Follow this guide for how to deploy Python Lambda functions with .zip file archives.

If you cannot get it to work via the .zip file, consider deploying via a container image instead by following this guide.

Daniel T
  • 511
  • 2
  • 6
16

In my case I just specified requests version (runtime python3.9) -

requests==2.28.2

and it worked.

ashraf minhaj
  • 504
  • 2
  • 14
4

The resolution that worked for me was to update the AWS Lambda runtime to 3.10 and if applicable any AWS Lambda layers to python 3.10 and then ensure you have packaged the latest release of requests. If you are using botocore or boto3 you will also need to ensure that you have included them in the code package or in an AWS Lambda Layer including versions boto3>=1.26.153 and botocore>=1.29.153.

ashraf minhaj
  • 504
  • 2
  • 14
chrispy
  • 41
  • 1
  • 1
    That's actually a little misleading, using previous stable version of libraries did solve the issue while using `python3.9` as runtime. – ashraf minhaj Jun 18 '23 at 07:41
3

  1. Execute the following commands.

    pip install requests==2.25.0 -t ./python --no-user pip install beautifulsoup4 -t ./python --no-user pip install pytz -t ./python --no-user

  2. On PyPI, download the following whl files from the pages of numpy and pandas

  • numpy-1.24.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
  • pandas-2.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

  1. Unzip the files and move the contents to the python folder.

  2. Zip the python folder and upload it to AWS Lambda Layer.

  3. Set the layer to the Lambda.

  4. Then the code runs without errors.

dixhom
  • 2,419
  • 4
  • 20
  • 36
1

You can solve this adding in your deployment.txt:

urllib3==1.16

William Faria
  • 11
  • 1
  • 1
    This does not work, you will get error 'is_ipaddress' from 'urllib3.util.ssl_' (/var/task/urllib3/util/ssl_.py)", 1.26.16 works fine – Philip Jay Fry Jun 23 '23 at 06:56
0

I solved this problem by using python 3.11 when setting up my lambda function in AWS, the lower versions tends to throw this error the more

Anamsken
  • 1
  • 1
  • 1
    Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Aug 11 '23 at 03:20
  • 1
    upgrading lambda environment wasn't enough in my case--still had to rebuild package with correct `requests` version per @ashraf minhad – varontron Aug 16 '23 at 10:51