Heartbeat implementation with Thread.Sleep()

Question

in my application I have an "heartbeat" functionality that is currently implemented in a long running thread in the following way (pseudocode):

while (shouldBeRunning)
{
    Thread.Sleep(smallInterval);

    if (DateTime.UtcNow - lastHeartbeat > heartbeatInterval)
    {
        sendHeartbeat();
        lastHeartbeat = DateTime.UtcNow;
    } 
}

Now, it happens that when my application is going through some intensive CPU time (several minutes of heavy calculations in which the CPU is > 90% occupied), the heartbeats get delayed, even if smallInterval << heartbeatInterval.

To crunch some numbers: heartbeatInterval is 60 seconds, lastHeartbeat is 0.1 seconds and the reported delay can be up to 15s. So, in my understanding, that means that a Sleep(10) can last like a Sleep(15000) when the CPU is very busy.

I have already tried setting the thread priority as AboveNormal - how can I improve my design to avoid such problems?

Poor design, mostly :) I am surely moving to a more reliable solution (and timers are the most obvious thing), but at this point I am curious about the behavior. — Andrea, Feb 22 '12 at 17:30
@Andrea - Fix your design. The problem you encountered is the reason the Timer class was created. Please note you will still be at the mercy of the .NET Scheduler. — Security Hound, Feb 22 '12 at 17:32

score 4 · Answer 1 · answered Feb 22 '12 at 17:31

Is there any reason you can't use a Timer for this? There are three sorts you can use and I usually go for System.Timers.Timer. The following article discusses the differences though:

http://msdn.microsoft.com/en-us/magazine/cc164015.aspx

Essentially timers will allow you to set up a timer with a periodic interval and fire an event whenever that period ticks past. You can then subscribe to the event with a delegate that calls sendHeartbeat().

Timers should serve you better since they won't be affected by the CPU load in the same way as your sleeping thread. It has the advantage of being a bit neater in terms of code (the timer set up is very simple and readable) and you won't have a spare thread lying around.

score 3 · Answer 2 · answered Feb 22 '12 at 17:33

You seem to be trying to reinvent one of the timer classes.

How about using System.Timers.Timer for example?

var timer = new System.Timers.Timer(smallInterval);
timer.Elapsed += (s, a) => sendHeartbeat;
timer.Enabled = true;

One of the issues here may be, at a guess, how often your thread gets scheduled when the CPU is under load. Your timer implementation is inherently single threaded and blocks. A move to one of the framework timers should alleviate this as (taking the above timer as an example) the elapsed event is raised on a thread pool thread, of which there are many.

score 2 · Accepted Answer · answered Feb 22 '12 at 17:43

Unfortunately, Windows is not a Real Time OS and so there are few guarantees about when threads are executed. The Thread.Sleep () only schedules the earliest time when the thread should be woken up next, it is up to the OS to wake up the thread when there's a free time slice. The exact criteria for waking up a sleeping thread is probably not documented so that the Window's kernel team can change the implementation as they see fit.

I'm not sure that Timer objects will solve this as the heartbeat thread still needs to be activated after the timer has expired.

One solution is to elevate the priority of the heartbeat thread so that it gets a chance of executing more often.

However, heartbeats are usually used to determine if a sub-system has got stuck in an infinite loop for example, so they are generally low priority. When you have a CPU intensive section, do a Thread.Sleep (0) at key points to allow lower priority threads a chance to execute.

Heartbeat implementation with Thread.Sleep()

3 Answers3

Linked