What is the best way to create periodic Linux threads in C

Question

For my application I have the requirement of accurate periodic threads with relative low cycle times (500 µs).
In particular the application is a run time system of a PLC. It's purpose is to run an application developed by the PLC user. Such applications are organised in programs and periodic tasks - each task with it's own cycle time and priority.
Usually the application runs on systems with real time OSs (eg. vxWorks or Linux with RT patch).

Currently the periodic tasks are implemented via clock_nanosleep. Unfortunately the actual sleep time of clock_nanosleep is disturbed by other threads - even with lower priority. Once every second, the sleep time is exceeded by about 50 ms. I've observed this on Debian 9.5, on RaspberryPi and also on an ARM-Linux with Preemt-RT.

Here's a sample, which shows this behavior:

#include <pthread.h>
#include <unistd.h>
#include <stdint.h>
#include <stdio.h>

typedef void* ThreadFun(void* param);

#define SCHEDULER_POLICY    SCHED_FIFO
#define CLOCK CLOCK_MONOTONIC
#define INTERVAL_NS       (10 * 1000 * 1000)

static long tickCnt = 0;

static long calcTimeDiff(struct timespec const* t1, struct timespec const* t2)
{
  long diff = t1->tv_nsec - t2->tv_nsec;
  diff += 1000000000 * (t1->tv_sec - t2->tv_sec);
  return diff;
}

static void updateWakeTime(struct timespec* time)
{
  uint64_t nanoSec = time->tv_nsec;
  struct timespec currentTime;

  clock_gettime(CLOCK, &currentTime);
  while (calcTimeDiff(time, &currentTime) <= 0)
  {
    nanoSec = time->tv_nsec;
    nanoSec += INTERVAL_NS;
    time->tv_nsec = nanoSec % 1000000000;
    time->tv_sec += nanoSec / 1000000000;
  }
}

static void* tickThread(void *param)
{
  struct timespec sleepStart;
  struct timespec currentTime;
  struct timespec wakeTime;
  long sleepTime;
  long wakeDelay;

  clock_gettime(CLOCK, &wakeTime);
  wakeTime.tv_sec += 2;
  wakeTime.tv_nsec = 0;

  while (1)
  {
    clock_gettime(CLOCK, &sleepStart);
    clock_nanosleep(CLOCK, TIMER_ABSTIME, &wakeTime, NULL);
    clock_gettime(CLOCK, &currentTime);
    sleepTime = calcTimeDiff(&currentTime, &sleepStart);
    wakeDelay = calcTimeDiff(&currentTime, &wakeTime);
    if (wakeDelay > INTERVAL_NS)
    {
      printf("sleep req=%-ld.%-ld start=%-ld.%-ld curr=%-ld.%-ld sleep=%-ld delay=%-ld\n",
          (long) wakeTime.tv_sec, (long) wakeTime.tv_nsec,
          (long) sleepStart.tv_sec, (long) sleepStart.tv_nsec,
          (long) currentTime.tv_sec, (long) currentTime.tv_nsec,
          sleepTime, wakeDelay);
    }
    tickCnt += 1;
    updateWakeTime(&wakeTime);
  }
}

static void* workerThread(void *param)
{
  while (1)
  {
  }
}

static int createThread(char const* funcName, ThreadFun* func, int prio)
{
  pthread_t tid = 0;
  pthread_attr_t threadAttr;
  struct sched_param schedParam;

  printf("thread create func=%s prio=%d\n", funcName, prio);

  pthread_attr_init(&threadAttr);
  pthread_attr_setschedpolicy(&threadAttr, SCHEDULER_POLICY);
  pthread_attr_setinheritsched(&threadAttr, PTHREAD_EXPLICIT_SCHED);
  schedParam.sched_priority = prio;
  pthread_attr_setschedparam(&threadAttr, &schedParam);

  if (pthread_create(&tid, &threadAttr, func, NULL) != 0)
  {
    return -1;
  }

  printf("thread created func=%s prio=%d\n", funcName, prio);
  return 0;
}
#define CREATE_THREAD(func,prio)  createThread(#func,func,prio)


int main(int argc, char*argv[])
{
  int minPrio = sched_get_priority_min(SCHEDULER_POLICY);
  int maxPrio = sched_get_priority_max(SCHEDULER_POLICY);
  int prioRange = maxPrio - minPrio;

  CREATE_THREAD(tickThread, maxPrio);
  CREATE_THREAD(workerThread, minPrio + prioRange / 4);
  sleep(10);
  printf("%ld ticks\n", tickCnt);
}

Is something wrong in my code sample?

Is there a better (more reliable) way to create periodic threads?

Answer 1

For my application I have the requirement of accurate periodic threads with relative low cycle times (500 µs)

Probably too strong requirement. Linux is not a hard real-time OS.

I would suggest to have fewer threads (perhaps a small fixed set -only 2 or 3, organized in a thread pool; see this for an explanation, remembering that a RasberryPi3B+ has only 4 cores). You might prefer a single thread (think of a design around an event loop, inspired by continuation-passing style).

You probably don't need periodic threads. You need some periodic activity. They all might happen in the same thread. (the kernel is rescheduling tasks perhaps every 50 or 100 ms, even if it is capable of sleeping a smaller time, and if tasks get rescheduled very frequently -e.g. every millisecond- , their scheduling has a cost).

So read carefully time(7).

Consider using timer_create(2), or even better timerfd_create(2) used in an event loop around poll(2).

On a RaspberryPi, you won't have guaranteed 500µs delays. This is probably impossible (the hardware might not be powerful enough, and the Linux OS is not hard real-time). I feel your expectations are not reasonable.