0

I am trying to understand how to implement Partitioner<T>.

My case:

We read from a file and we insert to the Azure table storage. We use tasks in order to speed up the process. The file has nearly 10,000,000 lines.

I tried to implement a more elegant solution using Parallel execution and partitioning. I would like to partition this large group of 9,000,000 into a group of partition sets and distribute them in 30-50 threads.

I would think that I need a dynamic implementation of Partitioner<List<List<TableEntity>>>. I have few examples, the one with the StringPartitioner, but it's not really helpful for my case as I need an IEnumerator for List<List<TableEntity>>.

If all this is confusing, please let me know so I can clarify.

The problem is how to implement a dynamic Partitioner in List<List<TableEntity>>.

svick
  • 236,525
  • 50
  • 385
  • 514
Veronica_Zotali
  • 253
  • 1
  • 3
  • 13
  • 1
    Please have a loot at this: http://msdn.microsoft.com/en-us/library/dd997416(v=vs.110).aspx – Reza Sadr Nov 25 '14 at 00:18
  • Why do you think you need a custom partitioner? How exactly are the entities supposed to be processed? (Is each `TableEntity` independent, or do you need to process a whole `List` at once?) – svick Nov 27 '14 at 13:54

0 Answers0