s3fs timeout issue on an AWS Lambda function within a VPN

Question

s3fs seems to fail from time to time when reading from an S3 bucket using an AWS Lambda function within a VPN. I am using s3fs==0.4.0 and pandas==1.0.1.

import s3fs
import pandas as pd


def lambda_handler(event, context):
    bucket = event['Records'][0]['s3']['bucket']['name']
    s3_file = event['Records'][0]['s3']['object']['key']
    s3fs.S3FileSystem.connect_timeout = 1800
    s3fs.S3FileSystem.read_timeout = 1800
    with s3fs.S3FileSystem(anon=False).open(f"s3://{bucket}/{s3_file}", 'rb') as f:
        self.data = pd.read_json(f, **kwargs)

The stacktrace is the following:

Traceback (most recent call last):
File "/var/task/urllib3/connection.py", line 157, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/var/task/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/var/task/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/task/botocore/httpsession.py", line 263, in send
chunked=self._chunked(request.headers),
File "/var/task/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/var/task/urllib3/util/retry.py", line 376, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/var/task/urllib3/packages/six.py", line 735, in reraise
raise value
File "/var/task/urllib3/connectionpool.py", line 672, in urlopen
chunked=chunked,
File "/var/task/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/var/task/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/var/task/urllib3/connection.py", line 300, in connect
conn = self._new_conn()
File "/var/task/urllib3/connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x7f4d578e3ed0>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/task/botocore/endpoint.py", line 200, in _do_get_response
http_response = self._send(request)
File "/var/task/botocore/endpoint.py", line 244, in _send
return self.http_session.send(request)
File "/var/task/botocore/httpsession.py", line 283, in send
raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://my_bucket.s3.eu-west-1.amazonaws.com/?list-type=2&prefix=my_folder%2Fsomething%2F&delimiter=%2F&encoding-type=url"

Has someone faced this same issue? Why would it fail only sometimes? Is there a s3fs configuration that could help for this specific issue?

**Side-note:** It is not recommended to use `s3fs` in a production environment. Amazon S3 is an object storage system, not a filesystem. Frankly, I have never heard of anyone trying s3fs from an AWS Lambda function. I would recommend that you instead use an AWS SDK to communicate with Amazon S3. — John Rotenstein, Apr 24 '20 at 08:08

score 2 · Accepted Answer · answered Apr 27 '20 at 12:31

Actually there was no problem at all with s3fs. Seems like we were using a Lambda function with two Subnets within the VPC and one was working normally but the other one wasn't allowed to access S3 resources, therefore when a Lambda was spawned using the second network it wouldn't be able to connect at all.

Fixing this issue was as easy as removing the second subnet.

score 0 · Answer 2 · edited Jun 21 '21 at 06:36

You could also use boto3 which is supported by AWS, in order to get json from S3.

import json
import boto3

def lambda_handler(event, context):
   bucket = event['Records'][0]['s3']['bucket']['name']
   key = event['Records'][0]['s3']['object']['key']

   s3 = boto3.resource('s3')
   file_object = s3_resource.Object(bucket, key)
   json_content = json.loads(file_object.get()['Body'].read())

s3fs timeout issue on an AWS Lambda function within a VPN

2 Answers2