Deferred execution vs ToList gives different results

Question

I'm in the middle of unit testing a function I wrote to search a queryable of items. I just assert that I get 1 item back, which I should get if the method works. But I get 0 items back. In my method I use deferred execution and just use ToList before I return it. But if I instead change the method to work directly with a list and repeatedly call ToList, I get the correct results.

Am I correct to say that it is not safe to assume that deferred execution produces the same results as Immediate execution?

This is a small app to demonstrate that it returns 0 items

   class Program
    {

        static void Main(string[] args)
        {
            Dictionary<string, string> values = new Dictionary<string, string>()
            {
                 {
                     "Prop1",
                     "*Value*"
                 },
                 {
                     "Prop2",
                     "2*"
                 }
            };
            List<InputItem> items =new List<InputItem>()
            {
                new InputItem()
            };
            Console.WriteLine(Helper.SearchInputItems(items.AsQueryable(), values).Count);
            Console.ReadLine();
        }
    }

    public class InputItem
    {
        public Dictionary<string, string> MappedValues = new Dictionary<string, string>()
        {
            {
                     "Prop1",
                     "This is a value that should be found"
                 },
                 {
                     "Prop2",
                     "2 everything that begins with 2 should be found"
                 }
        };
    }
    public static class Helper
    {
        delegate bool Searcher(string input, string searchString);
        /// <summary>
        /// Searches the added input items.
        /// </summary>
        /// <param name="values">A dictionary of field names and the search pattern for that field.</param>
        /// <returns>List of found InputItems.</returns>
        public static List<InputItem> SearchInputItems(IQueryable<InputItem> inputItems, Dictionary<string, string> values)
        {
            foreach (var value in values)
            {
                string searchString = value.Value;
                Searcher searcher;
                if (searchString.StartsWith("*") && searchString.EndsWith("*"))
                {
                    searcher = new Searcher(StringHelpers.Contains);
                    searchString = searchString.Substring(1);
                    searchString = searchString.Remove(searchString.Length - 1);
                }
                else if (searchString.EndsWith("*"))
                {
                    searcher = new Searcher(StringHelpers.StartsWith);
                    searchString = searchString.Remove(searchString.Length - 1);
                }
                else
                {
                    searcher = new Searcher(StringHelpers.Exact);
                }
                inputItems = inputItems.Where(c =>
                    c.MappedValues.Any(x => x.Key == value.Key) &&
                    searcher(c.MappedValues.First(x => x.Key == value.Key).Value, searchString)
                    );

            }
            return inputItems.ToList();
        }
    }
    public static class StringHelpers
    {
        public static bool Contains(string input, string searchString)
        {
            return input.ToUpperInvariant().Contains(searchString.ToUpperInvariant());
        }
        public static bool StartsWith(string input, string searchString)
        {
            return input.ToUpperInvariant().StartsWith(searchString.ToUpperInvariant());
        }
        public static bool Exact(string input, string searchString)
        {
            return input.ToUpperInvariant() == searchString.ToUpperInvariant();
        }
    }

If I set a breakpoint I can actually see that it checks if 2 everything that begins with 2 should be found contains Value, which it doesn't and returns false. So it seems like the FirstOrDefault in the where clause selects the wrong item

Am I right in saying the code you've posted is the *working* code? How does it look if it's broken? Because the method signature says it returns a `List` - it's hard to see how you can make that deferred... or are you talking about calling `ToList` on the *input* data? — Jon Skeet, Aug 11 '11 at 09:03
@Jon No, the posted code is not working as intended. When it works, I change the first line to be `var inputItems = InputItemRepository.GetInputItems().ToList;` — Oskar Kjellin, Aug 11 '11 at 09:06
Right. That wasn't entirely clear. Now, what is InputItemRepository actually returning? Is this *really* hooked up to a database, or what? You really should be able to reduce the amount of code you've got in order to demonstrate this - do you really need both StartsWith and EndsWith to make it fail? Do you need all the mappings and filters? — Jon Skeet, Aug 11 '11 at 09:07
@Jon in my unit test it is hooked up to a TestInputItemRepository which has a List in the background. Just adding and removing to is using the methods and the get method returns `InputItems.AsQueryable` — Oskar Kjellin, Aug 11 '11 at 09:09
@Jon I will try to throw together a small console app to demonstrate it — Oskar Kjellin, Aug 11 '11 at 09:10
That would be perfect. My *suspicion* is that in the process of putting together the app, you'll find the problem. — Jon Skeet, Aug 11 '11 at 09:16
@Jon please see the new code. If you change the `SearchInputItems` to accept a `List` instead it will work (given that you add `ToList` on the where clause to make it compile) — Oskar Kjellin, Aug 11 '11 at 09:18

score 3 · Accepted Answer · answered Aug 11 '11 at 09:37

Okay, I've got it. It's the old problem of capturing the loop variable.

Here:

        foreach (var value in values)
        {
            ...
            inputItems = inputItems.Where(c =>
                c.MappedValues.Any(x => x.Key == value.Key) &&
                searcher(c.MappedValues.First(x => x.Key == value.Key).Value, 
                                              searchString)
                );

        }

you're using value within a lambda expression, which means it will use "the current value of value" when that executes... and that value changes as the loop iterates.

Simply use:

foreach (var valueIterationVariable in values)
{
    var value = valueIterationVariable;
    // code as before
}

and I believe it will be okay. (I would question your use of the name "value", by the way, but that's a different matter.)

I haven't looked deeply into why it worked with IEnumerable<T> but not IQueryable<T>, but I suspect the extra deferral was to blame.

That's awesome. I had no idea. The article clearly describes why this does not work, and it's acutally quite logical — Oskar Kjellin, Aug 11 '11 at 09:49

Deferred execution vs ToList gives different results

1 Answers1