Parallel Querying Azure Storage

Question

I currently have a queury which looks along the lines of:

TableQuery<CloudTableEntity> query = new TableQuery<CloudTableEntity().Where(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, PK));

foreach (CloudTableEntity entity in table.ExecuteQuery(query))
{
    //Logic
}

I been researching about parallels, however, I cannot find any good code examples on how to use it. I want to be able to query thousands of partition keys like

CloudTableEntity().Where(PartitionKey == "11" || PartitionKey == "22")

Where I can have around 40000 Partition keys. Is there a good way to do this?

see this answer http://stackoverflow.com/a/3878378/142904 – Kristian Damian Sep 23 '14 at 14:55 — Kristian Damian, Sep 23 '14 at 14:55

score 3 · Answer 1 · answered Jul 21 '15 at 07:38

The following sample code will issue multiple partition key queries in parallel:

     CloudTable table = tableClient.GetTableReference("xyztable");
     List<string> pkList = new List<string>(); // Partition keys to query
     pkList.Add("1");
     pkList.Add("2");
     pkList.Add("3");
     Parallel.ForEach(
        pkList,
        //new ParallelOptions { MaxDegreeOfParallelism = 128 }, // optional: limit threads
        pk => { ProcessQuery(table, pk); }
     );

Where ProcessQuery is defined as:

  static void ProcessQuery(CloudTable table, string pk)
  {
     string pkFilter = TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, pk);
     TableQuery<TableEntity> query = new TableQuery<TableEntity>().Where(pkFilter);
     var list = table.ExecuteQuery(query).ToList();
     foreach (TableEntity entity in list)
     {
        // Process Entities
     }
  }

Note that ORing two partition keys in the same query as you listed above will result in a full table scan. To avoid a full table scan, execute individual queries with one partition key per query as the sample code above demonstrates.

For more details on query construction please see http://blogs.msdn.com/b/windowsazurestorage/archive/2010/11/06/how-to-get-most-out-of-windows-azure-tables.aspx

score 0 · Answer 2 · answered Jun 27 '17 at 13:58

0

Using table.ExecuteQuerySegmentedAsync will provide better performance

answered Jun 27 '17 at 13:58

Maayan Hope

1,482
17
32

Parallel Querying Azure Storage

2 Answers2