3

How to improve scheduler and interrupt latency:

Background:

  • Embedded system based on 10 cores mips64 processor
  • 9 cores run SMP linux. kernel version 2.6.32.27
  • We have realtime performance required process which has to complete certain tasks within 1ms. At maximum load conditions it may take 800uS.
  • This process starts the processing after receiving GPIO interrupt (1ms interrupt provided by FPGA. implemented as a kernel driver). Till then it will make a icotl call to gpio driver and will be put to sleep by the virtue of wake_up_interruptible system call
  • The GPIO ISR will wake_up() this process
  • To prevent other processes hogging CPU for this process, we run this process on an "isolcpus" core.
  • We have set priority to be highest among user thread for this process as below: Priority: 80, Scheduling type:SCHED_FIFO threadSetRtPriority(SCHED_FIFO, 80);
  • All /proc/sys/kernel/sched_ parameter values are default. We haven't fine tuned them

Problem:

  • Sometimes we see that ISR has called wake_up, but the process is scheduled only after 350uS. This is a big time since our processor is running at 1.25GHz. This big number for scheduling latency, is puzzling us, as we have already isolated the core exclusively for this process by using "isolcpus"
  • We profile the max CPU cycle count between consecutive 1ms GPIO ISR calls. This max time is more than 1.5ms. This big number for interrupt latency is too a concern for us, as this will eat up into the time available for the process to do its processing within 1ms boundary.

Please help us with inputs to reduce the interrupt and scheduling latency numbers

guy
  • 51
  • 7
  • Is `CONFIG_PREEMPT` option enabled for kernel? Not clear: is that process in kernel space or user space? – LPs Dec 13 '16 at 13:04
  • we havent enabled "CONFIG_PREEMPT" option. The process that i am referring is in user space. – guy Dec 13 '16 at 13:36
  • So you need at least to enable preemption or to apply RT patch already explained by @Clifford – LPs Dec 13 '16 at 13:40

2 Answers2

3

The standard Linux kernel does not provide real-time scheduling. A level of real-time determinism can be achieved with the RT_Preempt patch. It still requires careful design, and is no substitute for an RTOS for critical real-time requirements.

Clifford
  • 88,407
  • 13
  • 85
  • 165
  • yes, i agree with you. let me try applying this patch and try compiling our mips arch – guy Dec 13 '16 at 13:40
2

I have been working on linux kernel 4.8 preempt-rt which has the RT_Preempt patch applied from this repo: linux kernel 4.8 preempt-rt and have some promising results!

I have benchmarked both preempt-rt and non-preempt-rt linux kernels by running rt-benchmark cyclictests and found that the Max Latency in case of preempt-rt linux kernel has come down to 61 us as against 2025 us when using non-preempt linux kernel, which might as well help your case.

The results have clearly tempted me to use the prempt-rt kernel as there is an overwhelming difference in Max Latency between the two. I have documented the results here: sachin-mokashi-linux-preempt-rt, in case if it might be of help to you!

Community
  • 1
  • 1
Sachin Mokashi
  • 415
  • 5
  • 17