Select every 1000 items from a list and pass it to a method

Question

I have a list and currently we are passing on a single item at a time to another method

foreach (var contact in tracker.Parse())

basically we are selecting contacts from Azure Blob Storage and importing them into Dynamics CRM. tracker.Parse() returns is a list of contacts.

I want to select every 1000 and then wait until they are completed in the other method before I pass in the next 1000.

Need guidance on how to do this.

Appreciate the assistance!

Show us what code you have already (and then Google `Take()` and `Skip()`) — DavidG, Mar 20 '17 at 22:39
This is almost impossible to answer without some more context. What is tracker.Parse()? — Luke Benting, Mar 20 '17 at 22:43
Possible duplicate of [Split an IEnumerable into fixed-sized chunks (return an IEnumerable> where the inner sequences are of fixed length)](http://stackoverflow.com/questions/13709626/split-an-ienumerablet-into-fixed-sized-chunks-return-an-ienumerableienumerab) — Chris, Mar 20 '17 at 23:16
Your sample shows neither a `List` nor passing a single item to a method. You should use a bare minimum example of the problem. You say `tracker` is a list, but a `List` does not have a `Parse()` method, so it's a little confusing. Presumably `Parse` returns a `List`? — Rufus L, Mar 20 '17 at 23:24

Nkosi · Answer 1 · 2017-03-21T00:05:58.737

Group the data from the data source into the desired group size and process them.

Using Linq it can be achieved with Select and GroupBy extensions

int groupSize = 1000;

//Batch the data
var batches = tracker.Parse()
  .Select((contact, index) => new { contact, index })
  .GroupBy(_ => _.index / groupSize, _ => _.contact);

//Process the batches.
foreach (var batch in batches) {
    //Each batch would be IEnumerable<TContact>
    ProcessBatch(batch);
}

This can be converted a reusable generic method if needed

public static void ProcessInBatches<TItem>(this IEnumerble<TItem> items, int batchSize, Action<IEnumerable<TItem>> processBatch) {

    //Batch the data
    var batches = items
      .Select((item, index) => new { item, index })
      .GroupBy(_ => _.index / batchSize, _ => _.item);

    //Process the batches.
    foreach (var batch in batches) {
        //Each batch would be IEnumerable<TItem>
        processBatch(batch);
    }
}

Rufus L · Answer 2 · 2017-03-20T23:14:03.107

Maybe something like this...

static void Main()
{
    const int batchSize = 1000;

    // Populate array with 5841 items of data
    var lotsOfItems = new List<int>();
    for (int i = 0; i < 5841; i++)
    {
        lotsOfItems.Add(i);
    }

    // Process items in batches, waiting for each batch to complete before the next
    int indexOfLastItemTaken = 0;
    while (indexOfLastItemTaken < lotsOfItems.Count - 1)
    {
        var itemsTaken = lotsOfItems.Skip(indexOfLastItemTaken).Take(batchSize).ToList();
        ProcessItems(itemsTaken);
        indexOfLastItemTaken += itemsTaken.Count();
    }

    Console.Write("Done. Press any key to quit...");
    Console.ReadKey();
}

static void ProcessItems(IEnumerable<int> input)
{
    // do something with input
    Console.WriteLine(new string('-', 15));
    Console.WriteLine($"Processing a new batch of {input.Count()} items:");
    Console.WriteLine(string.Join(",", input));
}

Select every 1000 items from a list and pass it to a method

2 Answers2