32

Traceback (the last output from console):

File "batchpy.py", line 61, in <module>
obj.batch_w1()
File "batchpy.py", line 49, in batch_w1
batch.put_item(data=item)
File "/usr/local/lib/python2.7/dist-packages/boto/dynamodb2/table.py",   line 1641, in __exit__
self.resend_unprocessed()
File "/usr/local/lib/python2.7/dist-packages/boto/dynamodb2/table.py", line 1718, in resend_unprocessed
resp = self.table.connection.batch_write_item(batch_data)
File "/usr/local/lib/python2.7/dist-packages/boto/dynamodb2/layer1.py", line 420, in batch_write_item
body=json.dumps(params))
File "/usr/local/lib/python2.7/dist-packages/boto/dynamodb2/layer1.py", line 2842, in make_request
retry_handler=self._retry_handler)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 954, in _mexe
status = retry_handler(response, i, next_sleep)
File "/usr/local/lib/python2.7/dist-packages/boto/dynamodb2/layer1.py", line 2876, in _retry_handler
response.status, response.reason, data)
boto.dynamodb2.exceptions.ProvisionedThroughputExceededException:    ProvisionedThroughputExceededException: 400 Bad Request
{u'message': u'The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API', u'__type': u'com.amazonaws.dynamodb.v20120810#ProvisionedThroughputExceededException'}
Taran
  • 12,822
  • 3
  • 43
  • 47
Onkar Kundargi
  • 341
  • 1
  • 3
  • 5

2 Answers2

44

There are 2 ways you can handle this problem:

  1. Increase level of throughput (for this option you have to pay more).

  2. The way that normally we have to do at some points is that we need to implement the logic at application level. For instance, call dynamoDB to check for an exception. If throughput is exceeded, sleep for some seconds and call the same query again (this is what we have implemented in our app).

Jamal
  • 763
  • 7
  • 22
  • 32
Harshal Bulsara
  • 7,898
  • 4
  • 43
  • 61
  • #2 Is a nice solution, but what about running this approach inside a lambda function that has limited timeout max to 300 seconds? – user2976753 Dec 15 '16 at 19:02
  • @user2976753, this might not be a good solution when it is added with lambda function. This logic needs to be implemented before calling it. – Harshal Bulsara Dec 16 '16 at 04:43
  • 1
    This might be interesting https://aws.amazon.com/blogs/aws/auto-scale-dynamodb-with-dynamic-dynamodb/ – Harshal Bulsara Dec 16 '16 at 04:46
42

DynamoDB uses a provisioned throughput model for both reads and writes. That means your application will receive errors if it tries to perform more reads or writes than you have allocated to the table.

AWS has done a number of things to help out with this:

  • The AWS SDK clients will automatically retry the request several times before you see a ProvisionedThroughputExceededException
  • Up to five minutes of unused capacity can be consumed in bursts to accommodate spikes in requests
  • You can increase the provisioned read/writes on a table an unlimited number of times each day (to perform more reads/writes per second)
  • You can decrease the provisioned throughput on a table 9 times a day (4 times in the first 4 hours, and 1 each in the rest 4 hours window) (to save money)
  • You can use Dynamic DynamoDB, a 3rd party app, to automatically manage scaling up and down the provisioned throughput

Depending on the type of app you are creating there are several things you can do to deal with these errors:

  • Batch or long-running applications can back off and retry these requests if they occur to limit the table usage
  • When bulk loading or bulk reading from a table you can manually scale up the throughput and scale down after you are done
  • For a transactional application you can ensure you have your provisioned throughput above the level required to run the application
  • Use Dynamic DynamoDB or script changes to table throughput yourself
Muhammad Dyas Yaskur
  • 6,914
  • 10
  • 48
  • 73
JaredHatfield
  • 6,381
  • 2
  • 29
  • 32
  • 3
    Specifically for the BatchWrite API, you also want to check the response to the batch call because in there, there's a field called UnprocessedItems, which are items that are throttled. Ideally you want to just re batch the Unprocessed items, and keep looping until it's empty. The difference between this and what is happening now is that your entire batchwrite call is being throttled. I would recommend dialing up your throughput, although this will depend on how much WCU (KB/sec) you plan on consuming. – Raymond Lin Jul 17 '15 at 22:00
  • 2
    You can decrease the throughput 9 times a day (4 decreases in the first 4 hours, and 1 decrease for each of the subsequent 4 hour windows in a day). Edited the answer without logging in to SO. :( – Sahil Lakhwani Jul 29 '17 at 14:37