c# BlockingCollection causing out of memory

Question

I am coding a Forex Trading robot, and I am running an outOfMemory exception after some time (around 2 hours) using BlockingCollection. I basically have 1 queue pair Trade chart, that are added into a dict:

        private Dictionary<string, BlockingCollection<tick>> tickQueues = new Dictionary<string, BlockingCollection<tick>>();

I check the memory dump after one hour, and I can see the following items are piling up:

                                     Count Size(bytes)   Inclusive Size
ThreadPoolWorkQueue+QueueSegment    22,951  24,236,256  40,316,868
QueueUserWorkItemCallback   689,838 13,796,760  16,081,272
TimerQueueTimer 11,160  713,772 2,355,736

I have a timer that is responsible to add data to the Queue:

   private void TickTimer_tick(object source, ElapsedEventArgs e) {
            if (Monitor.TryEnter(LockTimerTick, GlobalSettings.APISleepDelayMSTick)) {
                updateLockFailCount = 0;
                try {
                     tick t = new tick(DateTime.Now, d.bid, d.ask);    
                            lastBid = d.bid;
                            lastAsk = d.ask;
                            t.pair = Inst.pair;
                            //myTickQueue.TryAdd(t);
                            if (!myTickQueue.TryAdd(t)) {
                                functions.Logger.log("Error when adding Tick on Queue for " + Inst.pair+ " Maybe Queue is full", "SHMAPI", LOGLEVEL.WARN);
                            } 

                } catch (Exception E) {
                    functions.Logger.log("Error happened when refreshing tick data: " + E.Message, "SHMAPI", LOGLEVEL.ERROR);
                } finally {
                    Monitor.Exit(LockTimerTick);
                }
            } else {
                updateLockFailCount++;
                int sev = LOGLEVEL.TRACE;
                if (updateLockFailCount == 10) { sev = LOGLEVEL.DEBUG; }
                if (updateLockFailCount==50) { sev = LOGLEVEL.WARN;  }
                if (updateLockFailCount % 100 == 0 && updateLockFailCount>=100) { sev = LOGLEVEL.ERROR; }
                functions.Logger.log("Could not get lock to refresh tick data for symbol "+Symbol, "SHMAPI", sev);
            }
        }

And finally, my task that checks the Q:

public void startQueueTask(string Pair) {
            if (!tickQueues.ContainsKey(Pair.ToUpper())) {
                tickQueues.Add(Pair.ToUpper(), new BlockingCollection<tick>(GlobalSettings.tickQueueSize));
                if (!MTAPIs.ContainsKey(Pair.ToUpper())) {
                    throw new Exception("API for pair " + Pair + " Should be initialized !!");
                }
                MTAPIs[Pair.ToUpper()].setTickQueue(tickQueues[Pair.ToUpper()]);
                functions.Logger.log("Starting " + Pair + " Queue Task", "TICKPROCESSING", LOGLEVEL.DEBUG);

                Task.Run(() => {
                    foreach (tick tick in tickQueues[Pair.ToUpper()].GetConsumingEnumerable()) {     
                        try {
                            onTick(tick);
                        } catch (Exception E) {
                            functions.Logger.log("Error processing tick for symbol " + tick.pair + " " + E.Message, "TICKPROCESSING", LOGLEVEL.ERROR);
                            functions.printException(E);
                        }

                    }
                    functions.Logger.log("Exiting Queue Task", "TICKPROCESSING", LOGLEVEL.ERROR);
                });

            } else {
                functions.Logger.log("Skipping " + Pair + " Queue Task because already exists", "TICKPROCESSING", LOGLEVEL.DEBUG);
            }
        }

I am not really sure why I am getting OOM, but it looks similar to: http://blogs.microsoft.co.il/bnaya/2012/02/26/real-life-story-blocking-collection/ But I am not using parallel here... My Queues are empty though since week end market is closed. Would using another timer with TryDequeue a better approach ? Any advice would be welcome !

It looks like you are using System.Timers.Timer? If so, what is the interval? How long does the timer tick methods take? What is the value of AutoReset on the timer? — Mike Barry, May 01 '16 at 11:34
I don't see obvious problem, but I didn't hear about the problem you post link to. But... if it is similar, try to start the task using TaskFactory.StartNew and set the TaskCreationOptions to LongRunning Also: 1) your naming convention is horrible, you cannot see a different from class, local variable, method... try to change it to more C# like :P 2) get rid of ToUpper() by setting additional constructor parameter to StringComparer.InvariantCultureIgnoreCase :) it will be cleaner and faster — Carnifex, May 01 '16 at 12:08
Hey, I've the timer firing up every 50ms... I was actually thinking my issue might be due to timer events piling up... I'll try manual reset and let you guys know. FYI, timer definition: TickTimer = new System.Timers.Timer(); TickTimer.Elapsed += new ElapsedEventHandler(TickTimer_tick); TickTimer.Interval = GlobalSettings.APISleepDelayMSTick; //50 MS TickTimer.AutoReset = true; // Have the timer fire repeated events (true is the default) TickTimer.Enabled = true; — bmigette, May 01 '16 at 20:37
And regarding my naming conventions, it is as my programing skills :) I never really learnt programing, I just manage to do stuff when I need to ;) — bmigette, May 01 '16 at 20:46

score 0 · Answer 1 · answered May 02 '16 at 10:11

I switched timer to manual like this:

private void TickTimer_tick(object source, ElapsedEventArgs e) {
           try {
            //...
            } finally {
                TickTimer.Start();
            }

        }

And it seemed to have resolved my issue. I am also making sure to send ticks in the Q, and they are filtered by duplicates by the Receiver, just so the Queing thread is never pending for too long as well. Thanks for the pointer !

c# BlockingCollection causing out of memory

1 Answers1