9

Several days ago all went well, but now, not just one, but three of our servers are throwing these exceptions:

{ "Error" => { "Message" => "Rate exceeded", "Code" => "Throttling", "Type" => "Sender" } }

This happens when retrieving data from Amazon, while calling DescribeJobFlows on Amazon::Coral::ElasticMapReduceClient, and shows as NFO Exception Retriable invalid response.

Eliran Malka
  • 15,821
  • 6
  • 77
  • 100
Pily Kevin Hao
  • 91
  • 1
  • 1
  • 3
  • You must be hitting the server too hard it looks like. Try these links with suggestions or other information and then please let us know what works for you. http://sesblog.amazon.com/post/TxKR75VKOYDS60/How-to-handle-a-quot-Throttling-Maximum-sending-rate-exceeded-quot-error https://forums.aws.amazon.com/thread.jspa?messageID=366822# – Joseph Myers Aug 24 '13 at 04:09
  • 1
    Thank you very much.Now ,I have know the reason without the links.My colleages told me .Thanks any way.The reason is that there are so many tasks.We have 5 servers and aws must receive our data during 5 hours.But it is not enough to upload one server task to AWS in an hour.So time is out. I will read the links and then tell u which one can solve my problem.Thanks. – Pily Kevin Hao Aug 24 '13 at 04:57

2 Answers2

3

Every AWS API has dynamic throttling protection. It's kinda sad but best way it to retry/sleep between API calls :(

There is a configuration for the boto3 client to change max_retries before throwing the exception. There is also the general reference on how to use retries.

Good luck!

Eliran Malka
  • 15,821
  • 6
  • 77
  • 100
confiq
  • 2,795
  • 1
  • 26
  • 43
0

You should add some retry logic to your code. See this other Stackoverflow question for some ways of implementing that in Python:

Need advice to add exponenital back-off logic in AWS API in python

Community
  • 1
  • 1
German Gomez Herrero
  • 531
  • 5
  • 5