We have developed a kernel module that acts basically like a wired communication traffic (eth, ...) - WiFi bridge. It forwards periodically incoming wired data to WiFi and vice versa. The system consists of two devices running the same kernel module. Under certain conditions we are experiencing undesired latency.
The module starts a kernel thread which basically performs:
while(1) {
schedule()
do_bridge_work()
usleep_range(200, 500)
}
During normal operation, top command displays:
CPU: 0% usr 19% sys 0% nic 60% idle 0% io 0% irq 19% sirq
Load average: 1.03 1.06 1.01 1/54 491
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
491 2 root DW 0 0% 29% [bridge_thread]
3 2 root SW 0 0% 10% [ksoftirqd/0]
When diagnostics is activated the thread also sends out additional diagnostic wired eth packets. In this mode, ping of devices on one side to the other side goes from 5-6ms to 45-900 ms. top then shows:
CPU: 0% usr 25% sys 0% nic 50% idle 0% io 0% irq 24% sirq
Load average: 1.06 1.07 1.01 1/54 491
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
491 2 root DW 0 0% 35% [bridge_thread]
3 2 root SW 0 0% 14% [ksoftirqd/0]
If an additional schedule() is inserted before usleep_range(), or if the sleep time is increased, it drastically reduces the latency. I think I have narrowed it down to conclude that the RX softirq does not get the scheduling time it needs to process the incoming traffic (NET_RX_SOFTIRQ). This is based on the fact the timing of transmitted packets are perfectly fine.
My questions are:
1: Why does the ping time increase due to more work in do_bridge_work() (more processing and additional transmitting of packets). Is NET_RX_SOFTIRQ likely starved?
2: Why does the ping time decrease when additional schedule() is inserted, or when sleep time is increased? Does that mean that 200-500 us is not enough to process all pending softirqs work, and the lower softirqs are starved? Is the effect of adding additional schedule() the same as letting other do more work by increasing the sleep time?
Kernel version is 4.1.38, configured with NO_HZ, PREEMPT = y.
