Multiple enumeration of IEnumerable - StackOverflowException

Question

An interesting problem with IEnumerable and Linq.

static void Main(string[] args)
{
    var subject = "#Accountancy #premier #Agile #Apache #automation #Automation #banking #Banking #bankIngs #AutoMation";
    var hashtags = subject.Split('#').Select(hashtag => hashtag.Trim().ToUpper()).Distinct();

    var plurals = hashtags.Where((hashtag) =>
    {
        return hashtags.Contains($"{hashtag.ToUpper()}S");
    }).Select(h => $"{h.ToUpper()}S");      //.ToList(); (1) - will not break

    //filter hashtags
    hashtags = hashtags.Except(plurals);    //.ToList(); (2) - will not break

    //if iterate, would break with:
    //System.StackOverflowException was unhandled Message: An unhandled exception of type 'System.StackOverflowException' occurred in mscorlib.dll
    foreach (var hashtag in hashtags)
    {
        Console.WriteLine(hashtag);
    }

    Console.Read();
}

Curious on how would one explain why overflow exception occurs?

Use a new variable at the filter to avoid changing `hashtags` before fully iteration over it. — wimh, Apr 07 '16 at 13:33
Read about **Linq and deferred execution** and combine this readings with the other two answers. Then you'll see why your code breaks. Deferred execution can lead to some pitfalls. — derpirscher, Apr 07 '16 at 13:44

Rob · Accepted Answer · 2016-04-07T13:57:02.377

Walk through it step by step.

Plurals is every word in hashtags which also has the same word ending in s
Hashtags is every word except plurals.

To execute 2, you must execute 1. However, hashtags is continually changing, so plurals is trying to execute not on the original collection, but on the result of 2 (which again, relies on the result of 1).

Your query will attempt to be:

hashtags = hashtags.Except(plurals);

Replacing plurals

hashtags = hashtags.Except(
            hashtags.Where(hashtag => { return hashtags.Contains($"{hashtag.ToUpper()}S"); })
                    .Select(h => $"{h.ToUpper()}S")
           );

But hashtags is hashtags.Except(plurals);

hashtags.Except(
            hashtags.Except(plurals).Where(hashtag => { return hashtags.Contains($"{hashtag.ToUpper()}S"); })
                    .Select(h => $"{h.ToUpper()}S")
           );

And then we need to replace plurals again.. and so on.

Your fixes (adding .ToList()) is the logical way to fix it.

score 3 · Answer 2 · answered Apr 07 '16 at 13:41

3

You are re-assigning hashtags (which has not been evaluated) to another evaluation which is resulting in an infinite loop. If you put the second evaluation in another variable, it will work:

var hashtags2 = hashtags.Except(plurals);

foreach (var hashtag in hashtags2)
{
    Console.WriteLine(hashtag);
}

answered Apr 07 '16 at 13:41

Josh Knack

405
2
8

1

I don't think this is the complete answer - `hashtags = hashtags.Except(hashtags.Where(h => h.StartsWith("B")));` doesn't throw a stack overflow exception. – stuartd Apr 07 '16 at 13:43

Multiple enumeration of IEnumerable - StackOverflowException

2 Answers2