Pagination in Dynamo db while scan operation

Question

I want to scan my Dynamo db table with pagination applied to it. In my request I want to send the number from where I want pagination to get start. Say, e.g. I am sending request with start = 3 and limit = 10, where start is I want scan to start with third item in the table and limit is upto 10 items. Limit however I can implement with .withLimit() method(I am using java). I followed this aws document. Following is the code of what I want to achieve:

<Map<String, AttributeValue>> mapList = new ArrayList<>();
      AmazonDynamoDB client =AmazonDynamoDBClientBuilder.standard().build();

      Gson gson = new GsonBuilder().serializeNulls().create();

      Map<String, AttributeValue> expressionAttributeValues = new 
      HashMap<String,AttributeValue>(); 
      expressionAttributeValues.put(":name",
      newAttributeValue().withS(name));

      List<ResponseDomain> domainList = new ArrayList<>(); 
      ResponseDomain responseDomain = null;

      //lastKeyEvaluated = start
      Map<String, AttributeValue> lastKeyEvaluated = null; 
      do { 
      ScanRequest scanRequest = new 
      ScanRequest().withTableName(STUDENT_TABLE)
      .withProjectionExpression("studentId, studentName")
      .withFilterExpression("begins_with(studentName, :name)")
      .withExpressionAttributeValues(expressionAttributeValues).
      withExclusiveStartKey(lastKeyEvaluated);

      ScanResult result = client.scan(scanRequest);

      for (Map<String, AttributeValue> item : result.getItems()) { 
      responseDomain = gson.fromJson(gson.toJson(item), 
      ResponseDomain.class); 
      domainList.add(responseDomain);


      } lastKeyEvaluated = result.getLastEvaluatedKey(); 
      } while (lastKeyEvaluated!= null); 
      //lastKeyEvaluated = size

      return responseDomain;

In the above code I am stuck at 3 places:

How can I set lastKeyEvaluated as my start value i.e 3
In the while condition how can I specify my limit i.e 10
When I try to map item from Json to my domain class, I encounter error.

Am I misinterpreting the concept of pagination in dynamodb or doing something wrong in the code. Any guidance will be highly appreciated as I am a newbie.

score 6 · Accepted Answer · answered Dec 12 '17 at 09:05

You can only start reading from some place by the ExclusiveStartKey. This key is the primary key of your table. If you know your item key, you can use it like this (e.g. your table primary key is studentId):
```
Map<String, AttributeValue> lastKeyEvaluated = new HashMap<String,AttributeValue>();
lastKeyEvaluated.put("studentId", new AttributeValue(STUDENTID));
```
When you specify the limit = N in dynamo you are setting that it should only read N items from the table. Any filtering is applied after the N items have been read. See Limiting the Number of Items in the Result Set. That might leave you with less results than expected. So you could create a variable in your code to send requests until you hit your expected limit and cutoff the extra results.
```
int N = 10;
List<Map<String, AttributeValue>> itemsList = new ArrayList<>();
do {
   // scanRequest.withLimit(N)

   ...

   itemList.addAll(result.getItems());

   if(itemsList.size() >= N) {
       itemsList = itemsList.subList(0, N);
       break;
   }
} while (lastKeyEvaluated != null && itemsList.size() < N);

// process the itemsList
```

Dynamo uses it’s own json structure. See Dynamo response syntax. You can get the value of an attribute the way you stored it in dynamo. If studentId is a string then it could be something like this:

for (Map<String, AttributeValue> item : result.getItems()) {
    responseDomain = new ResponseDomain();
    responseDomain.setId(item.get("studentId").getS());
    domainList.add(responseDomain);
}

score 2 · Answer 2 · answered Dec 11 '17 at 10:27

Pagination doesn't quite work the way you are thinking.

Use ScanRequest.withLimit(X) to choose the number of items in each page of results. For example, setting ScanRequest.withLimit(10) means that each page of results you get will have 10 items in it.

The lastKeyEvaluated is not a number of a page, it is the actual key of an item in the table. Specifically it is the key of the last item in the last set of results you retrieved.

Think about it like this. Imagine your results are:

Dog
Chicken
Cat
Cow
Rhino
Buffalo

Now lets say I did ScanRequest.withLimit(2) and lastKeyEvaluated = null, so each page of results has 2 items and I will retrive the first page of results. My first scan returns

Dog
Chicken

And

lastKeyEvaluated = result.getLastEvaluatedKey();

Returns

Chicken

And now to get the next page of results I would use ScanRequest.withExclusiveStartKey(Chicken). And the next set of results would be

Cat
Cow

Your code above uses a do/while loop to retrieve every page of results and print it out. Most likely you will want to remove that do/while loop so that you can handle one page at a time, then retrieve the next page when you are ready.

What in my use case where I have to specify from where to start scanning table. For that do I have to scan whole table first and then apply start by using for loop on the received items? — Pallavi Kaushik, Dec 11 '17 at 10:57
Scan always evaluates every item in a table, there is no way to set a starting point. You can use a filterexpression to return a subset of results. For example if you have item attributes that go 1,2,3,4,5 you can set a filterexpression to only return results where that attribute has values 2-4. — F_SO_K, Dec 11 '17 at 11:58

Pagination in Dynamo db while scan operation

2 Answers2

Linked