6

I have a list of documents that are stored in DynamoDB. There are more than 10,000 documents being stored in the db. I have set the limit to 100 when requesting. DynamoDB is doing a good job by returning me the first 100 documents and the LastEvaluatedKey to get the next 100 documents.

The problem here is I also want the DynamoDB to return me the total number of pages for pagination purpose. In this case since i have 10,000 documents it should return 100 (the number of pagination)

For now what I have done is that I have been counting manually by looping the queries until it doesn't return me the LastEvaluatedKey. Add up how many looping has been done to get the total page. But I believe there are better approach.

Ibrahim
  • 336
  • 2
  • 13
  • You can Write one more query for counting a total number of rows and this helps you about work on calculating on the front end. . – Samson Jul 04 '19 at 07:37

3 Answers3

10

As the other answer has correctly explained, there is no efficient way to get total result counts for DynamoDB query or scan operations. As such there is no efficient way to get total number of pages.

However, what I wanted to call out is that modern UIs have been moving away from classic pagination towards an infinite scroll design pattern. Where the “next page” of results is loaded on demand as the List is scrolled. This can be achieved with DynamoDB. You can still show discrete pages but cannot show, apriori, how many results or how many pages there are.. It’s a current shortcoming of DynamoDB.

Mike Dinescu
  • 54,171
  • 16
  • 118
  • 151
  • Since DynamoDB act differently than MySql, I have come to a conclusion to do it in infinite scroll way. Anyway thanks for the suggestions. – Ibrahim Jul 05 '19 at 03:21
  • 1
    but sometimes , lets take example of chat app, when you want to go to middle post by searching something, you would not do all fetching upto that point, instead fetch that middle page, then if user moves upward , fetch upper page , else if user moves below, fetch below page, how to do such scenario ? – Yusuf May 31 '21 at 18:28
7

Neither "Scan" (to read the entire table) nor "Query" (to read all the items with the same partition key) operations return an estimate on how many total results there are. In some cases (e.g., when a FilterExpression is provided), there is no efficient way for DynamoDB to do this estimation. In other cases there may be a way for DynamoDB to provide this information, but it doesn't.

If I understand you correctly, you want to Scan the entire table, without a filter. Like I said, Scan itself doesn't give you the number of items in the table. But you can find this number using DescribeTable, which returns, among other things, an "ItemCount" attribute, which is an estimate on the total number of items in the table, which may not be completely up-to-date but perhaps is good enough for your needs (if you want an estimate for some sort of progress report, that doesn't need to be 100% accurate).

If you really need accurate and up-to-the-moment counts of items in a partition or an entire table, you can always try to maintain such counters as separate data. Doing this correctly is not trivial, and will have performance implications, but in some use cases (e.g., rare writes and a lot of reads) may be useful.

Nadav Har'El
  • 11,785
  • 1
  • 24
  • 45
  • 1
    But even with the help of an additional table, how I am going to get, say, 5th page in a call? Also the page sizes should be customizable. Can you suggest me a popular NoSQL DB that can do this because I see even google using this sort of paging in their NewsAPI. – guneetgstar Jan 10 '21 at 08:34
2

You can maintain your own counts using DynamoDB streams - basically you create a Lambda function that watches for items being created/deleted and then write back to a counter item in DynamoDB that stores the current count of items.

mixja
  • 6,977
  • 3
  • 32
  • 34