Using GetRemaining() in dynamodb c# not able to load the data faster?

Question

I’m Not able to load data in dynamo DB using C#. we have used Scan and GetRemaining, which makes the process very slow. is there any alternative option to load a list of data faster from?

Table peopleTable = Table.LoadTable(client, "User_Activities_Log");
//get all records
ScanFilter scanFilter = new ScanFilter();
List<Document> allItems = peopleTable.Scan(scanFilter).GetRemaining();

Takes 5-6 mins to load 900 000 records. it is very much slow than SQL

score 0 · Answer 1 · answered Sep 20 '19 at 13:52

A scan on a document DB is a very slow process. Rather than doing a scan you should look to use a Query which will return data much quicker as it will be indexed.

Think of a scan like having to a thumb through every page in a book to find the infomration. Rather than a Query where the book had a table of contents to help you find the data. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LowLevelDotNetQuerying.html

AmazonDynamoDBClient client = new AmazonDynamoDBClient();

var request = new QueryRequest
{
    TableName = "Reply",
    KeyConditionExpression = "Id = :v_Id",
    ExpressionAttributeValues = new Dictionary<string, AttributeValue> {
        {":v_Id", new AttributeValue { S =  "Amazon DynamoDB#DynamoDB Thread 1" }}}
};

var response = client.Query(request);

foreach (Dictionary<string, AttributeValue> item in response.Items)
{
    // Process the result.
    PrintItem(item);
}

thanx I have tired its working fine, but still, im looking to fetch data for all the Id, rather than filtering it — sunera hanif, Sep 27 '19 at 06:33

score 0 · Answer 2 · answered Sep 23 '19 at 17:29

You can perform a segmented/parallel scan to scan the table in parallel. Each segment will have it's own iterator and you can load the data much faster.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html#Scan.ParallelScan

If you are using provisioned capacity you should make sure you are not being throttled.

Using GetRemaining() in dynamodb c# not able to load the data faster?

2 Answers2