Sporadic memory bloat using Toub's thread pool for long running tasks?

Question

I have read the Toub's thread pool is a good solution for longer running tasks, so I implemented it in the following code. I'm not even sure if my implementation is a good one because I seem to have sporadic memory bloat. The process runs around 50 MB most of the time then will spike to almost a GB and stay there.

The thread pool implementation is as follows (should I even be doing this?):

private void Run()
    {
        while (!_stop)
        {
            // Create new threads if we have room in the pool
            while (ManagedThreadPool.ActiveThreads < _runningMax)
            {
                ManagedThreadPool.QueueUserWorkItem(new WaitCallback(FindWork));
            }

            // Pause for a second so we don't run the CPU to death
            Thread.Sleep(1000);
        }
    }

The method FindWork looks like this:

private void FindWork(object stateInfo)
    {
        bool result = false;
        bool process = false;
        bool queueResult = false;
        Work_Work work = null;

        try
        {
            using (Queue workQueue = new Queue(_workQueue))
            {
                // Look for work on the work queue
                workQueue.Open(Queue.Mode.Consume);
                work = workQueue.ConsumeWithBlocking<Work_Work>();

                // Do some work with the message from the queue ...

                return;

The ConsumeWithBlocking method blocks if there is nothing in the queue. Then we call return to exit the thread if we successfully retrieve a message and process it.

Typically we run 10 threads with them typically in the blocking state (WaitSleepJoin). The whole point of this is to have 10 threads running at all times.

Am I going about this all wrong?

I might be misssing something, but why aren't you using the standard thread pool for this? — Kirk Woll, May 27 '14 at 14:53
From what I understand it is typically used for quick running tasks and I read that Toub's was better for longer running tasks for whatever reason. What would the implementation look like using the standard thread pool? — Evil August, May 27 '14 at 14:55
Ten threads can allocate 10 times more memory than one thread. It is a special number only to humans, computers don't have 10 fingers and toes so there's little reason to assume it is a happy number. Use a memory profiler if you have no idea what is swallowing the memory. And keep in mind that Toub's code dates from .NET 1.0, favor TaskCreationOption.LongRunning today. — Hans Passant, May 27 '14 at 16:21
Thank you Hans. I see your point. Can I get an example using TaskCreationOption.LongRunning? Thanks again. — Evil August, May 27 '14 at 16:31
Furthermore, I don't see how I can do this with tasks because using a cancellation token won't work since the threads need to be stopped sometimes when they are in a blocking state (WaitSleepJoin). — Evil August, May 27 '14 at 20:16

Sporadic memory bloat using Toub's thread pool for long running tasks?

0 Answers0