2

I have a few lambda functions which is making mutliple AWS Elastic beanstalk API Call written in python. It was working fine. but since last few days we are getting Throttling error. After discussion with AWS, they have told to add exponential back-off logic in code. So if it's throttle, will retry same API Call on incremental interval. I got what they are saying and how it works, but i don't understand how to add in my code. They have documentation for CLI but they don't have for APIs as follow,http://docs.aws.amazon.com/general/latest/gr/api-retries.html

can someone please give me simple example how can we map response of API Call and retry if it's throttle like my one api call i am using in my code as below,

import boto3

conn = boto3.client('elasticbeanstalk')

response = conn.describe_environments(EnvironmentNames=["xyz"])

return response

I know simple way to do it with if condition, by checking response is "Rate exceeded" using while do i think i can achieve this. but i want to check as provided in example of CLI, how can i do similar for API?

Any help would be appreciated!

Willem Van Onsem
  • 443,496
  • 30
  • 428
  • 555
Meet101
  • 711
  • 4
  • 18
  • 35

1 Answers1

5

You can use a proxy object to wrap around any AWS client object and add some retrying logic to the proxy object:

from botocore.exceptions import ClientError
import retrying
import wrapt

class RetriedClient(wrapt.ObjectProxy):
"""Add retry logic to a boto3 client.

Wait 2^x * 1000 milliseconds between each retry, up to 10 seconds,
then 10 seconds afterwards.

"""

    def is_throttling_error(exception):
        """Botocore throws just a generic ClientError exception."""
        return isinstance(exception, ClientError) \
            and "RequestLimitExceeded" in exception.response

    @retrying.retry(
        wait_exponential_multiplier=1000,
        wait_exponential_max=10000, 
        retry_on_exception=is_throttling_error)
    def __getattr__(self, name):
        return getattr(self.__wrapped__, name)


# Create a boto3 client to Cloudformation
cf_client = boto3.client('cloudformation')

# Add exponential backoff retries to all client methods
wrapped_cf_client = RetriedClient(cf_client)

Then you can just use wrapped_cf_client as you would normally use boto3's built-in client:

resp = wrapped_cf_client.describe_stacks()

DEPRECATION NOTE:

In newer versions of botocore there is a better way of configuring the retry logic of the boto3 SDKs. This works starting from version 1.6.0 of botocore:

from botocore.config import Config

config = Config(
    retries = dict(
        max_attempts = 10
    )
)

ec2 = boto3.client('ec2', config=config)
German Gomez Herrero
  • 531
  • 5
  • 5
  • but, you are retrying even if the exception is not `ThrottlingException`. This is problematic in some scenarios... – confiq May 31 '18 at 14:04
  • @confiq you are completely right of course. I should have said that the code is just a proof of concept. I modified it so that only Throttling errors are retried, but the code is still fragile since botocore doesn't have a specific exception for throttling errors, but it just throws a generic `ClientError`. Also I added a deprecation note since this is not necessary in newer versions of botocore. – German Gomez Herrero Jun 01 '18 at 07:39
  • your answer indeed deserves +1! How do you know it's version 1.10.4 version of botocore? – confiq Jun 03 '18 at 13:41
  • 1
    I just checked the value of `botocore.__version__` I had in the environment where I tested the snippet, but I just checked in the release notes from botocore that it was added in `1.6.0`. I'll update the answer. – German Gomez Herrero Jun 06 '18 at 19:56