Boto3 timeout troubleshooting

Question

I am trying to troubleshoot a situation.

I am initiating a Boto3 client like this:

s3_client = boto3.client('s3')

Then I am iterating over a number of files and uploading them using:

s3_client.upload_file()

My problem is that every now and then I see it pausing for 60 seconds and then continuing normally.

I've tried reducing the timeout value and the error I seem to be getting is:

botocore.exceptions.ConnectTimeoutError

or

urllib3.exceptions.ConnectTimeoutError

If I leave it at 60, eventually it succeeds and I don't get any errors.

My question is, this error means that when executing upload_file, it tried to send a put request to the server and the server didn't respond to that request?

When initiating the client, is any connection established that could be lost or that's simply only storing the credentials and it's irrelevant to this issue?

Many thanks.

Update: Adding more detailed logs:

CRITICAL Connect timeout

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
TimeoutError: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/botocore/httpsession.py", line 439, in send
    urllib_response = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1012, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 174, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError

During handling of the above exception, another exception occurred:

  File "/home/user/.local/lib/python3.10/site-packages/boto3/s3/inject.py", line 143, in upload_file
    return transfer.upload_file(
  File "/home/user/.local/lib/python3.10/site-packages/boto3/s3/transfer.py", line 288, in upload_file
    future.result()
  File "/home/user/.local/lib/python3.10/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/home/user/.local/lib/python3.10/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/home/user/.local/lib/python3.10/site-packages/s3transfer/tasks.py", line 139, in __call__
    return self._execute_main(kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/s3transfer/tasks.py", line 162, in _execute_main
    return_value = self._main(**kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/s3transfer/upload.py", line 758, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/client.py", line 415, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/client.py", line 731, in _make_api_call
    http, parsed_response = self._make_request(
  File "/home/user/.local/lib/python3.10/site-packages/botocore/client.py", line 751, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/endpoint.py", line 107, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/endpoint.py", line 183, in _send_request
    while self._needs_retry(attempts, operation_model, request_dict,
  File "/home/user/.local/lib/python3.10/site-packages/botocore/endpoint.py", line 305, in _needs_retry
    responses = self._event_emitter.emit(
  File "/home/user/.local/lib/python3.10/site-packages/botocore/hooks.py", line 358, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/hooks.py", line 229, in emit
    return self._emit(event_name, kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/hooks.py", line 212, in _emit
    response = handler(**kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/retryhandler.py", line 194, in __call__
    if self._checker(**checker_kwargs):
  File "/home/user/.local/lib/python3.10/site-packages/botocore/retryhandler.py", line 267, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/home/user/.local/lib/python3.10/site-packages/botocore/retryhandler.py", line 294, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/retryhandler.py", line 333, in __call__
    checker_response = checker(attempt_number, response,
  File "/home/user/.local/lib/python3.10/site-packages/botocore/retryhandler.py", line 233, in __call__
    return self._check_caught_exception(
  File "/home/user/.local/lib/python3.10/site-packages/botocore/retryhandler.py", line 376, in _check_caught_exception
    raise caught_exception
  File "/home/user/.local/lib/python3.10/site-packages/botocore/endpoint.py", line 249, in _do_get_response
    http_response = self._send(request)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/endpoint.py", line 321, in _send
    return self.http_session.send(request)
  File "/home/user/.local/lib/python3.10/site-packages/botocore/httpsession.py", line 472, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)

Is this a single-threaded Python app? Is it an up to date boto3 library? If you run your app with boto3 debug level logs, does it repro and do the logs tell you anything useful? — jarmod, Jun 14 '22 at 19:24
It is single threaded and boto3 was recently installed. I only have a few logs, let me update the issue and add some of them. — mangotango, Jun 14 '22 at 19:26
Those are standard Python logs when handling exceptions, which don't provide boto3 internal logging. What I mean by "debug level logs" is to modify your app to enable additional level of logging (see example in code [here](https://github.com/boto/boto3/issues/2292)). Also, are you reusing the same client object for each upload? And approximately how many files are uploaded before this fails? — jarmod, Jun 14 '22 at 19:45
Yes, I haven't had those logs enabled and not sure I can easily reproduce it but I will try. Yes, the same client object is used for all uploads. Is that a bad practice? It's probably hundreds of files until I noticed it. — mangotango, Jun 14 '22 at 19:47
No, I don't think it's a bad practice using the same client object. I was thinking the opposite (that if you *were* creating hundreds of client objects, that could become a problem). Also, see [this answer](https://stackoverflow.com/a/48265766/271415). — jarmod, Jun 14 '22 at 20:10
Ah, yes I saw that question before. By initiating a client object is any form of communication established that could be relevant to this issue or that's just to manage the credentials and it all comes down to the individual upload/download or whatever request one makes, in order to get a timeout? — mangotango, Jun 14 '22 at 20:13
Creating a client object will not initiate a connection of any kind, afaik. How boto3 manages connections to AWS services once you make API calls is not clearly documented. — jarmod, Jun 14 '22 at 20:16

score 3 · Accepted Answer · answered Jul 26 '22 at 11:53

The default boto3 retry mode (Legacy) doesn't properly handle a number of errors/exceptions, including RequestThrottled, PriorRequestNotComplete, ConnectionError and BandwidthLimitExceeded. It can manifest as very intermittent problems with long delays and repeated connection failures. You'll likely be better off using the Standard retry mode instead, which also has an exponential backoff.

import boto3
from botocore.client import Config as BotoConfig

TIMEOUT = 3
config = BotoConfig(connect_timeout=TIMEOUT, retries={"mode": "standard"})
client = boto3.client("s3", config=config)

the standard mode retries sets max_attempts==3, the default connect_timeout==60. https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html — j_d_b, Jan 11 '23 at 20:34

Boto3 timeout troubleshooting

1 Answers1