Getting Count() with deferred execution and lazy loading

Question

I have some code that uses deferred execution and lazy loading:

    public static IEnumerable<XElement> GetStreamElementP(string fileId, ListProgressEventHandler progressHandler, int total)
    {
        var filePath = Utility.GetEContentFilePath(fileId);
        using (var reader = XmlReader.Create(filePath, new XmlReaderSettings { IgnoreWhitespace = true, }))
        {
            var cnt = 0;
            reader.MoveToContent();
            // Parse the file and display each of the p nodes.
            reader.Read();
            while (reader.NodeType == XmlNodeType.Element && reader.Name == "p")
            {
                cnt++;
                var returnedValue = XElement.ReadFrom(reader) as XElement;

                int rem = cnt % _streamElementCallBackSize;
                if (progressHandler != null && rem == 0)
                {
                    progressHandler(null, new ListProgressEventArgs { ItemsProcessed = cnt, TotalItemsToProcess = total, });
                }
                yield return returnedValue;
            }
            reader.Close();
        }

    }

I'm looking to get a simple count on the number of elements. The current code we are using is:

    public static int FileElementsCount(string fileId)
    {
        var cnt = 0;
        foreach (XElement e in GetStreamElementP(fileId))
        {
            cnt++;
        }
        return cnt;
    }

Can I improve this to?

    public static int FileElementsCount(string fileId)
    {
        return GetStreamElementP(fileId).Count<XElement>();
    }

Or will this cause more memory to be used when getting the count? We are dealing with very large files in some cases and attempting to keep memory usage to a minimum where possible.

I have tried to find a concrete example that explains how the memory is used in each case without any success.

Thanks in advance for any help.

score 2 · Answer 1 · answered Jun 28 '11 at 21:09

It doesnt really matter. Both your method and the count method internally perform a direct loop (no lazy stuff here) over the result of GetStreamElementP. There is no caching or whatsoever involved.

If you want this to be faster, you either have to find a smart way of caching / pre-calculating the result of GetStreamElementP- or have a variant on GetStreamElementP which does a smarter count on the file directly

score 0 · Accepted Answer · answered Jun 28 '11 at 22:33

In your case, both ways of computing the count are will do the same.

The memory consumption of this function should be proportional the size of the <p> elements. So, if there is lots of small elements, it shouldn't consume large amounts of memory. If you have relatively few huge elements, this could consume quite a lot of memory, because you're creating an XElement out of each of them. If this was the case, the memory consumption could be made much smaller by not creating them at all.

Getting Count() with deferred execution and lazy loading

2 Answers2