On a Windows7 quadcore with hyperthreading enabled, I have an hundred of threads comunicating via BlockingCollection<T> (all initialized with the default constructor, thus using ConcurrentQueue<T> internally).
All threads receives 10-100 messages each seconds, except one that receives just 4-20 messages each day.
My issue is related to this last consumer: most of time, it's blocked waiting for new messages, but when a message is ready to be consumed it should be handled it as soon as possible, possibly instantly.
The problem is that when a message is added to the BlockingCollection dedicated to this consumer, it is received after a few seconds (the Take() returns from 3 to 12 seconds after the message was enqueued).
I guess the problem is related to how windows schedule this thread.
I've tried to increase the ThreadPriority of this consumer without any improvements. Then I've tried to set its processor affinity to a dedicated core, and I've changed all other threads's affinity to use the others cores, but still without any improvements.
How can I address this issue? What's the actual problem?
This is the thread loop (commands is the BlockingCollection):
while (!commands.IsCompleted)
{
_log.Debug("Waiting for command.");
command = commands.Take();
_log.DebugFormat("Received: {0}", command);
command.ApplyTo(target);
_log.InfoFormat("Dispatched: {0}", command);
}
Should I directly use a ConcurrentQueue, sleeping for something like 50 milliseconds (that would be an acceptable delay) when no messages are pending?
Note that
No CPU use more that 50%.
As for the delay: logs show that (most of times) a few seconds passes between "Dispatched: ..." and "Waiting for command." (aka on commands.IsCompleted) and a few others between "Waiting for command." and "Received: ..." (aka on commands.Take())
All "sane" threads are I/O bound but they are cooperating: believe me I can't change their design. However they work pretty well: the only missbehaving one is the low load thread that does a different kind of work. I know that thread priority are evil but I didn't find a better solution.
A test that (almost) replicate the issue
Here is a test that almost replicate the issue, on the server. "Almost" because the test stress the CPUs (all go to 100%), while in the actual application all the CPUs are under 50%:
public class ThreadTest
{
class TimedInt
{
public int Value;
public DateTime Time;
}
[Test]
public void BlockingCollection_consumedWithLowLoad_delaySomeTimes()
{
// arrange:
int toComplete = 0;
BlockingCollection<KeyValuePair<DateTime, TimedInt>> results = new BlockingCollection<KeyValuePair<DateTime, TimedInt>>();
BlockingCollection<TimedInt> queue = new BlockingCollection<TimedInt>();
Action<int> producer = a =>
{
int i = 1;
int x = Convert.ToInt32(Math.Pow(a, 7));
while (i < 200000000)
{
if (i % x == 0)
{
queue.Add(new TimedInt { Time = DateTime.Now, Value = i });
Thread.SpinWait(100); // just to simulate a bit of actual work here
queue.Add(new TimedInt { Time = DateTime.Now, Value = i + 1 });
}
i++;
}
Interlocked.Decrement(ref toComplete);
};
Action consumer = () =>
{
Thread.CurrentThread.Priority = ThreadPriority.Highest; // Note that the consumer has an higher priority
Thread.CurrentThread.Name = "Consumer";
while (toComplete > 0)
{
TimedInt v;
if (queue.TryTake(out v, 1000))
{
DateTime now = DateTime.Now;
results.Add(new KeyValuePair<DateTime, TimedInt>(now, v));
}
}
};
// act:
List<Thread> threads = new List<Thread>();
threads.Add(new Thread(new ThreadStart(consumer)));
for (int i = 0; i < 200; i++)
{
var t = new Thread(new ThreadStart(() => producer(7 + (i % 3))));
t.Name = "Producer " + i.ToString();
threads.Add(t);
toComplete++;
}
threads.ForEach(t => t.Start());
threads.ForEach(t => t.Join());
// assert:
Assert.AreEqual(0, results.Where(kvp => (kvp.Key - kvp.Value.Time).TotalMilliseconds > 1000).Count());
}
}
Any ideas?
