Am I iterating this enumerable twice here?

Question

Consider the following block of code:

using (FileStream fs = new FileStream(@"C:\bad_records.txt", 
                                      FileMode.Create, 
                                      FileAccess.Write))
{
    var badEnumerable = _cache.Where(kvp => !kvp.Value.Item1);

    fs.WriteLine(string.Format("BAD  RECORDS ({0})", badEnumerable.Count()));
    fs.WriteLine("==========");

    foreach (var item in badEnumerable)
    {
        fs.WriteLine(string.Format("{0}: {1}", item.Key, item.Value.Item2));
    }
}

where _cache is defined like this:

static Dictionary<string, Tuple<bool, string, string>> _cache;

am I iterating this enumerable twice? Once with Count() and once with the foreach?

Answer 1

Yes, you are iterating the enumerable twice.

An easy way to test this is to use a helper method such as this:

private static int count = 0;
public static IEnumerable<T> CountIterations<T>(IEnumerable<T> sequence)
{
    count++;
    //or some other debug type logging
    Console.WriteLine("Iterated {0} times.", count);
    foreach(var item in sequence)
        yield return item;
}

Answer 2

You use your dictionary _cache as IEnumerable of KeyValuePair.
Where method has deferred execution.
So you enumerate it twice: in Count and in foreach.

You can change it as:
var badEnumerable = _cache.Where(kvp => !kvp.Value.Item1).ToArray();
and fs.WriteLine(string.Format("BAD RECORDS ({0})", badEnumerable.Length));.

Answer 3

Yes, the .Count() and the foreach will both cause _cache to be enumerated twice and verified against the predicate in the Where cause.

As to whether this is a problem or not depends on a number of things:

1. Are all the values in _cache in memory already or is it querying an underlying source such as a database.
2. How many items are in the queried collection and what is the cost of the comparison.
3. Can the source be enumerated safely multiple times.

For example, if values in _cache are already in memory and the predicate is a simple boolean property comparison then enumerating the cache twice is probably more efficient and will add no extra memory overhead whereas adding a .ToList() will still result in 2 enumerations (one of _cache and one of the list) however the predicate check will only happen once (in the .ToList() call) and the foreach will have less objects to enumerate but you will have added the extra memory overhead of the additional list.

If the cache comes from a database then the memory overhead of adding a .ToList() after the .Where() will almost certainly be better than performing 2 individual queries against the database.

Answer 4

The short answer is yes.

Depending on the underlying type of badEnumerable, it might be enumerated twice. This is due to what is called "deferred execution". Deferred execution means that your LINQ query is not actually executed "until the query variable is iterated over in a foreach or For Each loop" (MSDN). Your foreach statement is obviously an iteration over the variable, and Enumerable.Count() also performs an iteration (in this case).

In some cases though, this will not cause two iterations. This occurs when badEnumerable is actually a subclass of ICollection. In that case, calling .Count() actually references the underlying .Count property and does not enumerate it.

Since badEnumerable is a Dictionary<TKey, TValue>, and also because the call to Enumerable.Where() returns a generic IEnumerable (one that is not an ICollection), your specific case does not run into this situation and will iterate twice.

Answer 5

Yes, to avoid iterating twice, use a List You may print the count after the initial iteration. Another solution is to save the text written in the loop and print it after you print the count.

Edit Corrected:

using (FileStream fs = new FileStream(@"C:\bad_records.txt", FileMode.Create, FileAccess.Write))
{
    var badEnumerable = _cache.Where(kvp => !kvp.Value.Item1);

    int count = 0;
    foreach (var item in badEnumerable)
    {
        count++;
        Console.WriteLine(string.Format("{0}: {1}", item.Key, item.Value.Item2));
    }

    Console.WriteLine("==========");
    Console.WriteLine(string.Format("BAD  RECORDS ({0})", count));
}