Amazon Dynamodb find All in grails

Question

I have a grails application. I'm using amazon dynamodb for a specific requirement which is accessed, and entries are added by a different application. Now I need to get all the information from the dynamodb table to a postgreSQL table. There are over 10000 records in the dynamodb but the throughput is

Read capacity units : 100
Write capacity units : 100

In BuildConfig.groovy I have defined the plugin

compile ":dynamodb:0.1.1"

In config.groovy I have the following configuration

grails {
  dynamodb {
    accessKey = '***'
    secretKey = '***'
    disableDrop = true
    dbCreate = 'create'
  }
}

The code I have looks something similar to this

class book {

  Long id
  String author
  String name
  date publishedDate

  static constraints = {
  }

  static mapWith = "dynamodb"
  static mapping = {
    table 'book'
    throughput read:100
  }
}

When I try something like book.findAll() I get the following error

AmazonClientException: Unable to unmarshall response (Connection reset)

And when I tried to reduce the number of records by trying something like book.findAllByAuthor() (which also wud have above 1000's of records) I get the following error

Caused by ProvisionedThroughputExceededException: Status Code: 400, AWS Service: AmazonDynamoDB, AWS Request ID: ***, AWS Error Code: ProvisionedThroughputExceededException, AWS Error Message: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API.

I have the need to get all the records in dynamodb despite the throughput restriction and save it in a postgres table. Is there a way to do so?

I'm very new to this area, thanks in advance for the help.

After some research I came Across Google Guava. But even to use Guava RateLimiter, there wont be a fixed number of times I would need to send the request of how long it would take. So I'm looking for a solution which will suit the requirement I have

score 0 · Answer 1 · edited May 23 '17 at 11:54

0

Probably your issue is not connected with grails at all. The returned error message claims: The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API

So you should consider increasing level of throughput (for this option you have to pay more) or adjust your queries to obey actual limits.

Check out also this answer: https://stackoverflow.com/a/31484168/2166188

edited May 23 '17 at 11:54

Community

1
1

answered Dec 16 '16 at 11:33

Michal_Szulc

4,097
6
32
59

I'm looking for a solution which means I do not have to increase the throuhput, since it means I have to pay more. After a bit more researching, Throttling could be a solution for this, which wont exceed the limit just would take longer for task to complete. I have not done that before, so help from that end would also be helpful. – Visahan Dec 16 '16 at 12:59
First you should check out what are the limits which you are violating. For example you could create a job which will be executed every 15 minutes until it finish full task. – Michal_Szulc Dec 16 '16 at 13:35
As I have described in the above sample, I require to get all the records from dynamodb to be processed. So I am violating the read capacity units. I'm looking for a solution where I would be able to use throttling to achieve my goal, which would take longer but will finish the task. – Visahan Dec 20 '16 at 05:24

Amazon Dynamodb find All in grails

1 Answers1