I have a problem whit query/scan operation in DynamoDb. I'm using the dynamodb-data-mapper whit aws-data-mapper-annotations in order to do operations whit the database. I have a Lambda function which have to get all the items from a table (in this case I have a table of tags) and I want to paginate the result, whit a 'limit' and a 'start key' specified by the client, because the tags table could contain hundred of thousands records. So I write something like this:
import dbContext from '@dbModel/dbContext';
import Tag from '@dbModel/tables/tags';
import { ScanOptions } from '@aws/dynamodb-data-mapper';
const scanTag = async (filter: any): Promise<{
items: Tag[],
lastKey: Partial<Tag>
}> => {
let items: Tag[] = [];
let lastKey: Partial<Tag>;
const dbFilter: ScanOptions = {
limit: filter.limit,
startKey: {
id: filter?.startKey,
},
readConsistency: 'strong'
};
// use paginator because of the scan limit of 1MB
const paginator = dbContext.scan(Tag, dbFilter).pages();
for await (const page of paginator) {
items = items.concat(page);
lastKey = paginator.lastEvaluatedKey;
}
return Promise.resolve({
items: items,
lastKey: lastKey
});
};
export default scanTag;
This is the problem: the scan operations returns a non-sorted list of items, so how I can ensure that the second call by the client (so the client has already received the first 'n' items and he want to get the next 'm' items) will return a distinct set of items? For example: the first call return the result: [T5, T8, T1, T4], I want to do a second call providing the key of T4 as startKey, and I want to be sure the there's no T5, T8, T1 or T4 in my second response.
Thank you all in advance!