Entity Framework c#: Bug when AddRange in loop

Question

I have 2 millions of "pb" object instances. I want to add it to my database. Because of memory i splitted list of my objects into a collections of lists and in "for" loop I am trying to add 10'000 of my object instances to database per iteration.

Here is my code

 for (int i = 0; i < splittedList.Count; i++)                    
 {                         
    using(MyEntities db = new MyEntities()) //DB context                            
    {                         
    List<pb> outputPBs = splittedList[i];                        
    db.pb.AddRange(outputPBs);
    db.SaveChanges();
    }
}

Unfortunately first iteration add 10'000 instances, but next one add 10'370 instances instead of 10'000. This bug propagate till the last iteration.

I've made a long debug: I add proper list with AddRange, however db.pb.Local contains 10'370 objects after AddRange execution.

This additional 370 objects are instances from the previous iteration.

Could You help me?

Here is a code, which I used to split a list:

public static List<List<object>> Split(List<object> source)
{
    return  source
        .Select((x, i) => new { Index = i, Value = x })
        .GroupBy(x => x.Index / 3)
        .Select(x => x.Select(v => v.Value).ToList())
        .ToList();
}

edit: completed

I solved an issue. Problem was connected with one to many relation. I created tree data structure, and added root in different context than branches and leafs

Is this happening beyond a certain threshold? With less instances it doesn't do that? Your splittedlist contains 10.000 objects each, right? — rene, Apr 25 '14 at 14:36
I tried to split a list into packs of 100 instances and it works this way: 1st iteration: 100 elements added, table contains 100 elements 2nd iteration: 200 elements added (100 proper and 100 additional), table contains 300 elements 3rd iteration: 300 elements added (100 proper and 200 additional), table contains 600 elements Those additional elements in this case are copy of elements from previous insertion. — painacle, Apr 25 '14 at 16:19

score 0 · Answer 1 · answered Apr 25 '14 at 14:56

0

As you are creating a new on every iteration it doesn't make sense how there could be a 'left over' 370 objects from the previous iteration, with regards to the context that is.

Are you completely sure your split is working correctly?

With

List<pb> outputPBs = splittedList[i];

Are you saying that outputPBs.Count = 10,000 ?

Are the additional 370 objects actually PBs or do any of the PBs contain relationships to other objects that are possibly being added to the object count?

answered Apr 25 '14 at 14:56

Paul Zahra

9,522
8
54
76

I've attached to the main point splitting function which I used to split list. Yes - outputPBs.Count = 10000; Additional 370 objects are PBs, I think that there are a copy of previously added PBs. PBs has got a relation with FN (many PBs to one FN). It can be a root cause? – painacle Apr 25 '14 at 16:25
Before db.pb.AddRange(outputPBs); check the number of PBs in the context, if it's 370 then you are right and they are PBs... check the number of PBs that have a relationship (out of the 10,000 that turn into 10,370)... try using pb.Connection.Open() and pb.Connection.Close() to manage it's connection of the context a little better and prevent any pooling. – Paul Zahra Apr 26 '14 at 08:51
that should be db.Connection.Open() and db.Connection.Close() – Paul Zahra Apr 26 '14 at 09:10

Entity Framework c#: Bug when AddRange in loop

1 Answers1