4

Following kernel parameters are showing very different behaviour from R6 to R7 and we are not able to figure out why. Any help appreciated.

kernel.sched_min_granularity_ns kernel.sched_wakeup_granularity_ns

Background :

  • Application already running on RHEL6
  • Low latency requirement.

  • Application equipped with robustness feature i.e once the latency starts to increase more then acceptable threshold levels(pre-defined) or CPU usage is more then 85% then it stops processing new request(s) in order to avoid overloading.

  • We are now trying to deploy in on RHEL7 virtual environment and not able to utilize the CPU to the extent we could in RHEL6. We hardly can reach up to 55-60% and observe the latency spikes beyond acceptable threshold.

Notes :

  • Application version is identical on both cases(R6/R7).
  • Database and its configuration is also identical
  • Memory and CPU setting are identical as well

In R7, we were using tuned profile which changed the following kernel parameters impacting the behaviour : kernel.sched_min_granularity_ns = 10000000 kernel.sched_wakeup_granularity_ns = 15000000

If we change these values to R6 default(kernel.sched_min_granularity_ns = 4000000 kernel.sched_wakeup_granularity_ns = 4000000), then we do get the cpu usage to R6 levels(>85%).

However, when we put the same values in R6(kernel.sched_min_granularity_ns = 10000000 kernel.sched_wakeup_granularity_ns = 15000000), we don't see any adverse impact and CPU still scales up to 85-90% as earlier.**

So the non-default values of the above two parameters are R6 & R7 shows opposite impact.

So we are looking for reason why the same parameter behaves very differently compared to RHEL6 & RHEL7?

Thanks in advance.

Chota Bheem
  • 161
  • 7
  • "then we do cpu usage to R6 levels." This is murky. Please clarify. – Jeter-work Aug 10 '18 at 14:58
  • Oh yeah. Correct. I meant that we do get the CPU usage of 85-90% in RHEL6 as well with non-default values. R7 on the other hand same values does not give same output. – Chota Bheem Aug 11 '18 at 04:21
  • If when using RH6 defaults on your RH7 box all is working properly, why do not simply use the old default values? – shodanshok Aug 11 '18 at 06:51
  • Yeah that's an option. But since these parameter are by default given in one of the tuned profiles(throughput-performance) by RHEL, we wanted to understand its impact. But those seem to given and opposite effect. So we are trying to find the kernel behaviour or scheduler changes in regards to these parameter between RHEL6 & RHEL7. – Chota Bheem Aug 11 '18 at 06:54

1 Answers1

2

Finally, we have been able to achieve our target of scaling application CPU usage to 80%+ in RHEL7. It is now very closer to how application performed on RHEL6.

Following are our observations and the reason which drove our conclusion :

"vmstat" showed "runnable" count > total number of CPUs. But at the same time CPU usage was just 50%. Another indicator was the number of "context switches" between RHEL6 & RHEL7. RHEL7 showed lot lesser context switches for the same application load compared to RHEL6.

This meant that the processing is not slow rather the tasks are runnable tasks were not assigned to CPU cores. After a bit reserach we found kernel parameter "kernel.sched_migration_cost_ns" which basically specifies the amount of time before which the runnable task would be migrated to another CPU.

kernel.sched_migration_cost

Amount of time after the last execution that a task is considered to be “cache hot” in migration decisions. A “hot” task is less likely to be migrated, so increasing this variable reduces task migrations. The default value is 500000 (ns). If the CPU idle time is higher than expected when there are runnable processes, try reducing this value. If tasks bounce between CPUs or nodes too often, try increasing it.

Decreasing the value of kernel.sched_migration_cost to around 250nanosecond(default : 500ns) help us accelerate the CPU usage to 80%.

Chota Bheem
  • 161
  • 7
  • This is interesting. Did you end up talking to Red Hat support? – ewwhite Oct 10 '18 at 11:47
  • Nope. We had this finding on our own. Obviously we refereed lot of RHEL docs related to tuning guide and we performed lot of testing cycles to conclude on optimized value of the `kernel.sched_migration_cost` parameter. – Chota Bheem Oct 11 '18 at 04:13