Reading a file kills NIC interrupts

Question

I have a very strange problem with a board based on a MIPS processor and Linux 2.6. There are NIC interrupts for all incoming Ethernet packets. If I send 10.000 packets I could see that 10.000 NIC interrupts occurred.

START SYSTEM

SEND 10k PACKETS

/mnt/system # cat /proc/interrupts
          CPU0       
24:      10000         MIPS  NIC
29:       7192         MIPS  timer
30:          0         MIPS  UART1
31:       3092         MIPS  serial

ERR:          0

However, after I open and close a file (filled with zeros or regular) in the filesystem, there are much fewer NIC interrupts generated. For instance, just 2-7k interrupts for 10k packets. It has a detrimental effect on the system performance, but after rebooting everything with NIC interrupts is fine again.

START SYSTEM

std::fstream f;
f.open("/mnt/system/myfile");
f.close(); 

WAIT FOR SOME TIME

SEND 10k PACKETS

/mnt/system # cat /proc/interrupts
          CPU0       
24:       2045         MIPS  NIC
29:       7192         MIPS  timer
30:          0         MIPS  UART1
31:       3092         MIPS  serial

ERR:          0

The filesystem is jffs2 and the flash drive is 32M NOR serial device. Why reading a file kills NIC interrupts until reboot?

1) This is not C, maybe C++! 2) YOur question is not clear. 3) Is there a problem at all? — too honest for this site, Oct 08 '16 at 20:34
1) In fact, I tried various ways of reading a file, for example, Linux's tail produces the same problem 2) updated 3) yes, because there is correlation between interrupt count and packet processing performance of the device — Konstantin, Oct 08 '16 at 20:44
So you mean less interrupts means less performace? Did it come to mind that file-access also loads the CPU and possibly uses interrupts? — too honest for this site, Oct 08 '16 at 20:53
I open a file then close it then wait 5 min then send 10k packets. Only 2k NIC interrupts are generated. I know, it sounds ridiculous, but maybe someone had a similar problem, — Konstantin, Oct 08 '16 at 20:56
Not sure when it was introduced but you may also be seeing the effects of NAPI. When packet interrupt load becomes a problem the kernel disables the interrupt and dedicates a kernel thread to polling the NIC which continues until the polling thread is not getting new packets anymore. That's more efficient than interrupts all the time but if the system is also using 100% CPU for other tasks it will lose packets. But at that point you'd need to be assigning real-time thread priorities and accepting that your CPU is just not fast enough for the work. — Zan Lynx, Oct 08 '16 at 23:34
@ZanLynx I just checked the git log and there is an NAPI checkin from at least May 2005, so it could easily be part of OP's kernel. It would be nice to know the specific rev OP is using (e.g. 2.6.x.y) so it could be pseudo-bisected to look for NAPI fixes or other driver fixes. — Craig Estey, Oct 09 '16 at 00:04

Craig Estey · Accepted Answer · 2016-10-08T23:26:31.367

Caveat: This may not be a total solution, but I have some ideas. More information/testing may be necessary.

When you do [the system does] nothing else, the NIC driver is fast enough to respond to an interrupt and the ISR will complete processing on the single packet before the next packet arrives. (i.e.) There is a one-to-one relationship between incoming packets and NIC interrupts.

If you have other system activity, this may delay entry to the NIC driver's ISR. Also, this other activity may slow down processing in the NIC driver due to resource competition (e.g. locks, kmalloc, etc.)

Meanwhile, more NIC packets arrive. When the NIC ISR is finally/ultimately entered, it sees that there are multiple packets pending. It will process all of them, without leaving the ISR. So, it might process (e.g.) 5 packets on each interrupt, so the interrupt count is reduced by 5x.

This is what most "smart" NIC cards and drivers do. It actually improves throughput with heavy NIC traffic. Ordinarily, fewer NIC interrupts is a good thing because it reduces the wasted overhead of entering/exiting the ISR repeatedly.

This really depends upon the packet arrival rate and [average] interval.

So, the real question is: "When the interrupt count is reduced, do you lose packets/data or just see performance loss?"

And, how are you measuring the system performance difference?

Is the filesystem in question used otherwise (e.g. it is a root fs)? Or, is your program opening the file the only access to the fs [other than mounting it]

I'm not familiar with jffs2 or your flash driver, so I'd ask if you suspect either of doing something that would disable interrupts for an extended period of time?

UPDATE:

I just noticed that you're using linux version 2.6 That kernel is approximately 10 years old. It's well past end-of-life, so it probably isn't getting bug fixes for drivers unless the platform vendor is providing them.

So, another thing to consider is that any of the drivers may have a bug that causes the performance problem. There's a fair to good chance that the drivers have been fixed in more modern kernels.

You might want to switch to a newer kernel if you can. If not, you may be left with the [unenviable] task of backporting the newer drivers to the older kernel [or at least cherry pick some of the bug fixes].

Reading a file kills NIC interrupts

1 Answers1