using the Take inside a while loop

Question

i have a collection with thousands of elements, i do a for loop to process 100 elements at a time and then upload them and pause for 5 seconds

while (EntriesList.Count > 0)
            {
                service.InsertEntries(EntriesList.Take(100).ToList<Entry>());
                System.Threading.Thread.Sleep(5000);
            }

service.InsertEntries takes a List<Entry> as a parameter, also is there a better way to achive such thing?

normally i go with a for loop with a % operator and a temp list, is the above the same?

I usually write myself a `AsChunked(int batchSize)` method that converts a list into a sequence of batches. — usr, Nov 06 '13 at 12:21
ps: "Take" function does not remove elements from the list => Your 'while' will never remove anything from EntriesList. Normal ? — Olivier, Nov 06 '13 at 12:23
unless there is another thread modifying it... but I can see kind of a conccurency issue :) — Olivier, Nov 06 '13 at 12:26
It that your own Take or from `Enumerable.Take` extensions? Because if its the latter your loop won't terminate, Take does not remove elements from the source list. Also, why do you need to sleep exactly? It may have a bearing on the best way to proceed. — James World, Nov 06 '13 at 12:26
@JamesWorld i just figured out that Enumerable.Take doesnt remove,i dont want to modify original list just copty elements from it and process them and then dump the temp list, the pause is because i dont own the servers and the block fast uploads so it has to be there — FPGA, Nov 06 '13 at 12:31

score 3 · Answer 1 · answered Nov 06 '13 at 12:25

This will insert the first 100 entries again and again.

Keep track of where you are and use use Skip to move forward:

int current = 0;
int batchSize = 100;

while (current < EntriesList.Count)
{
    service.InsertEntries(EntriesList.Skip(current).Take(batchSize).ToList());
    current += batchSize;
    System.Threading.Thread.Sleep(5000);
}

score 2 · Answer 2 · answered Nov 06 '13 at 12:24

EntriesList.Take(100).ToList() will create a new list of 100 entries, you will not remove the 100 entries from your EntriesList.

You can do something like:

List<TYPE> objectsforupload = EntriesList.ToList();
while (objectsforupload.Count > 0)
{
        List<TYPE> uploadlist = objectsforupload.Take(100).ToList();
        objectsforupload = objectsforupload.Where(x => !uploadlist.Contains(x)).ToList();
        service.InsertEntries(objectsforupload);
        System.Threading.Thread.Sleep(5000);
}

score 2 · Answer 3 · answered Nov 06 '13 at 12:26

Call this after the .Take()

EntriesList.RemoveRange(0, Math.Min(EntriesList.Count, 100));

It will remove the first 100 elements.

EDIT: more complete example:

while (EntriesList.Count > 0)
{
    var poppedEntries = EntriesList.Take(100).ToList<Entry>();
    EntriesList.RemoveRange(0, poppedEntries.Count);
    service.InsertEntries(poppedEntries);
    System.Threading.Thread.Sleep(5000);
}

score 2 · Accepted Answer · edited May 23 '17 at 12:29

For this kind of thing you should write a Chunkify extension method. One good example you can get from Marc here. But due to the question it always returns an array with a fixed size and maybe null entries.

So maybe this implementation matches better your requirements:

/// <summary>
/// Divides an enumeration into smaller (same-sized) chunks.
/// </summary>
/// <typeparam name="T">The type of the elements within the sequence.</typeparam>
/// <param name="source">The sequence which should be breaked into chunks.</param>
/// <param name="size">The size of each chunk.</param>
/// <returns>An IEnumerable&lt;T&gt; that contains an IEnumerable&lt;T&gt; for each chunk.</returns>
public static IEnumerable<IEnumerable<T>> Chunkify<T>(this IEnumerable<T> source, int size)
{
    if (source == null)
    {
        throw new ArgumentNullException("source");
    }

    if (size < 1)
    {
        throw new ArgumentOutOfRangeException("size");
    }

    return ChunkifyImpl(source, size);
}

private static IEnumerable<IEnumerable<T>> ChunkifyImpl<T>(IEnumerable<T> source, int size)
{
    using (var iter = source.GetEnumerator())
    {
        while (iter.MoveNext())
        {
            var chunk = new List<T>(size);
            chunk.Add(iter.Current);

            for (int i = 1; i < size && iter.MoveNext(); i++)
            {
                chunk.Add(iter.Current);
            }

            yield return chunk;
        }
    }
}

Now you can use this extension as follows:

foreach(var chunk in EntriesList.Chunkify(100))
{
    service.InsertEntries(chunk);
    Thread.Sleep(5000);
}

score 1 · Answer 5 · answered Nov 06 '13 at 12:28

As it stands, your code does not work, you should also Skip by the same amount and

        while (entries.Count > 0)
        {
            service.InsertEntries(entries.Take(100).ToList<Entry>());
            entries = entries.Skip(100);
            System.Threading.Thread.Sleep(5000);
        }

However, this performs very badly. the IEnumable<> you get back from Skip() will add overhead every time it is called.

So the best advice for this scenario is to use a for loop.

using the Take inside a while loop

5 Answers5