An unhandled exception of type 'System.StackOverflowException' occurred in EntityFramework dll

Question

I get this exception when I try to process 270k records. It fails at 12k. Can someone explain to me what I am missing?

The database is SQL and I am using EF 6. I am using predicate builder to build my where clause. The idea being select * from table where ((a = 'v1' and b = 'v2') or (a = 'v11' and b = 'v21') or (a = 'v12' and b = 'v22') ..) I don't see anywhere that I still hold reference to my object that represents EF class. I am creating a POCO for the result I want to send back to view. Any ideas?

Also I am using CommandTimeout of 10000 and the point where it fails, when I run the query with same paramters in sql management studio, it returns 400 rows.

When I ran profiler, I noticed a few seconds before I got the error, memory usage shot up to 1GB+

Thanks

    public List<SearchResult> SearchDocuments(List<SearchCriteria> searchCriterias)
{
            List<SearchResult> results = new List<SearchResult>();
            var fieldSettings = GetData() ;// make a call to database to get this data
            using (var context = CreateContext())
            {
                var theQuery = PredicateBuilder.False<ViewInSqlDatabase>();
                int skipCount = 0;
                const int recordsToProcessInOneBatch = 100;
                while (searchCriterias.Skip(skipCount).Any())
                {
                    var searchCriteriasBatched = searchCriterias.Skip(skipCount).Take(recordsToProcessInOneBatch);
                    foreach (var searchCriteria in searchCriteriasBatched)
                    {
                        var queryBuilder = PredicateBuilder.True<ViewInSqlDatabase>();
                        // theQuery
                        if (searchCriteria.State.HasValue)
                            queryBuilder = queryBuilder.And(a => a.State == searchCriteria.State.Value);
                        if (!string.IsNullOrWhiteSpace(searchCriteria.StateFullName))
                            queryBuilder = queryBuilder.And(a => a.StateName.Equals(searchCriteria.StateFullName, StringComparison.CurrentCultureIgnoreCase));
                        if (searchCriteria.County.HasValue)
                            queryBuilder = queryBuilder.And(a => a.County == searchCriteria.County.Value);
                        if (!string.IsNullOrWhiteSpace(searchCriteria.CountyFullName))
                            queryBuilder = queryBuilder.And(a => a.CountyName.Equals(searchCriteria.CountyFullName, StringComparison.CurrentCultureIgnoreCase));
                        if (!string.IsNullOrWhiteSpace(searchCriteria.Township))
                            queryBuilder = queryBuilder.And(a => a.Township == searchCriteria.Township);
                        // and so on...for another 10 parameters
                        theQuery = theQuery.Or(queryBuilder.Expand());
                    }
                    // this is where I get error after 12k to 15k criterias have been processed
                    var searchQuery = context.ViewInSqlDatabase.AsExpandable().Where(theQuery).Distinct().ToList();
                    foreach (var query in searchQuery)
                    {
                        var newResultItem = SearchResult.Create(query, fieldSettings); // POCO object with no relation to database
                        if (!results.Contains(newResultItem))
                            results.Add(newResultItem);
                    }

                    skipCount += recordsToProcessInOneBatch;
                }
            }

            return results.Distinct().OrderBy(a => a.State).ThenBy(a => a.County).ThenBy(a => a.Township).ToList();
}

You could just do `context.SearchResults.Where(x => ((x.a == 'v1' &&x.b == 'v2') || (x.a = 'v11' &&x.b = 'v21') || (x.a = 'v12' && x.b = 'v22')).Distinct().OrderBy(a => a.State).ThenBy(a => a.County).ThenBy(a => a.Township).ToList();` — Fourat, Feb 23 '16 at 17:11
Build that in a for loop? Also at any given point, I am pulling only upto 4000 rows. What reference is not getting cleaned up that I get this error? I am guessing the same error would show up when I use the format suggested above. All this does is replace predicate with direct parameters. right? — Deepak, Feb 23 '16 at 18:38
that replaces the foreach loop ! please read more about Linq and EF online — Fourat, Feb 25 '16 at 09:01

score 0 · Answer 1 · answered Feb 23 '16 at 18:42

0

Fourat is correct that you can modify your query to context.SearchResults.Where(x => ((x.a == 'v1' &&x.b == 'v2') || (x.a = 'v11' &&x.b = 'v21') || (x.a = 'v12' && x.b = 'v22')).Distinct().OrderBy(a => a.State).ThenBy(a => a.County).ThenBy(a => a.Township).ToList(); What this do with make the database do the heavy lifting for you and you

I would also suggest that you use lazy evaluation instead of forcing it into a list if you can.

answered Feb 23 '16 at 18:42

William Karnesky

86
7

I assume that doing one call to be is akin to lazy evaluation since the query is not executed against db until that point and only one call is executed. Am I missing something? var searchQuery = context.ViewInSqlDatabase.AsExpandable().Where(theQuery).Distinct().ToList(); – Deepak Feb 23 '16 at 19:17
also in not the foreach (var searchCriteria in searchCriteriasBatched) building the same where clause? – Deepak Feb 23 '16 at 19:21
That is not lazy evaluation. As you are calling the .ToList() method you are forcing all results to be turned immediately. if you leave that off and parse it through an enumerator it will only return a few records at a time, letting .Net and SQL optimize how many records it returns from the database, but it won't try to force them all in at once and thus you will have a much lower memory footprint. – William Karnesky Feb 23 '16 at 19:34
You can also simplify your code further and just use something along the lines of `context.SearchResults.Where(x => ((x.a == 'v1' &&x.b == 'v2') || (x.a = 'v11' &&x.b = 'v21') || (x.a = 'v12' && x.b = 'v22')).Distinct().OrderBy(a => a.State).ThenBy(a => a.County).ThenBy(a => a.Township).Select(item=> SearchResult.Create(item, fieldSettings);` this will give you a lazy evaluation that you can enumerate through a set of searchResults. – William Karnesky Feb 23 '16 at 19:41
Removed the "tolist". There is about 15 parameters...so I have gone with the other option. Your suggestion to simplify code is good when there is only few parameters. Will mark it answer if this works. thanks. – Deepak Feb 23 '16 at 19:44
This didn't return anything..I have been in waiting mode for over 20 min. Earlier, with ToList(), this returned in upto 3 min. Missing something? var searchQuery = context.TheTable.AsExpandable().Where(theQuery); foreach (var query in searchQuery) { var newResultItem = SearchResult.CreatePoco(query); if (!results.Contains(newResultItem)) results.Add(newResultItem); } – Deepak Feb 23 '16 at 20:29

An unhandled exception of type 'System.StackOverflowException' occurred in EntityFramework dll

1 Answers1