0

I am using DynamoDBEnchancedAsyncClient to query DynamoDB using GSI and pagination. Below is the code that I am using to achieve the same. I am trying to limit the number of pages emitted to the client using limit() below in the SdkPublisher<Page<Customer>>

Flux.from(PagePublisher.create(query.limit(5)).items())

I am planning to provide the next subsequent pages to the client back using the lastEvaluatedKey and exclusiveStartKey mentioned at Docs > My doubt is when we limit the number of pages using limit(), what exactly happens wrt the number of pages actually fetched from the DynamoDB?

  1. Are all the pages that can accomodate the 1MB per queryresponse limit are fetched from DynamoDB and then only the pages that are permitted after limit() are sent back?
  2. Only the number of pages permitted by limit() are fetched from the DynamoDB?

if its #1. Then wouldn't it be overhead or additional redundant effort of fetching pages again from the DDB, since the earlier pages would have been already fetched and we are just again fetching them using exclusiveStartKey?

ghostrider
  • 2,046
  • 3
  • 23
  • 46

1 Answers1

0

Limit is evaluated inside DynamoDB and you are not charged for access to items beyond the specified limit.

If you need to get multiple pages of data from DynamoDB, it is your job to track the total items returned, limit only applies to a single request. If you have a limit of 500, and the first page of data returns 400 items, when making the second request you should reduce your limit to 100.

If you want to implement paging for your end user and want to supply a specific page size, you can use a limit and continue using that limit value for subsequent requests. You will get a number of items up to that limit for each request.

Ross Williams
  • 507
  • 1
  • 9
  • Ok. My question was how does the Dynamo db fetch the pages i have specified a limit to the number of pages to be emitted? Will it just fetch only those limited pages or will fetch all of them and return only the limited pages? – ghostrider May 30 '22 at 08:50
  • I'm not 100% sure I understand your question, but I have updated my answer. "pages I have specified" - I don't understand what you mean. "fetch only those limited pages" - I don't understand. – Ross Williams May 30 '22 at 09:04
  • Ok Got it thanks. Any idea about this question? https://stackoverflow.com/questions/72431044/scanindexforward-not-working-as-expected-while-doing-pagination-in-dynamodb – ghostrider May 30 '22 at 09:41