4

I am trying to test the NAPI functionalities on embedded linux environment. I used 'pktgen' to generate the large number of packets and tried to verify the interrupt count of my network interface at /proc/interrupts.

I found out that interrupt count is comparatively less than the packets generated. Also I am trying to tune the 'netdev_budget' value from 1 to 1000(default is 300) so that I can observe the reduction in interrupt count when netdev_budget is increased.

However increasing the netdev_budget doesn't seems to help. The interrupt is similar to that of interrupt count observed with netdev_budget set to 300.

So here are my queries:

  1. What is the effect of 'netdev_budget' on NAPI?

  2. What other parameters I can/should tune to observe the changes in interrupt count?

  3. Is there any other way I can use to test the NAPI functionality on Linux?(Apart from directly looking at the network driver code)

Any help is much appreaciated.

Thanks in advance.

sunisiha
  • 57
  • 6
  • Test for a "rotting packet". The arrival of a frame exactly during the transition from napi polling back to interrupt mode could go undetected. That frame is not handled until another one arrives to trigger a receive interrupt. – sawdust Jul 07 '16 at 00:12
  • Hi @sawdust thanks for the hint. But could you please elaborate your answer? Because I don't see any dropped packets unless I increase the packet/frame size. So what exactly I should do for checking whether there are 'rotting packets' ? – sunisiha Jul 07 '16 at 05:30
  • A rotting packet is not a dropped packet. It's a delayed packet. To easily notice such a bug you have to have timed packets, e.g. expecting a response within X seconds, and the link varies between a burst of activity then a period of inactivity. The circumstances to create the problem involve a time window of maybe 10 instructions. But Murphy's Law prevails. It can be hard to create the circumstances, which partly explains why drivers get released with this bug. – sawdust Jul 07 '16 at 19:45

1 Answers1

5

I wrote a comprehensive blog post about Linux network tuning which explains everything about monitoring, tuning, and optimizing the Linux network stack (including the NAPI weight). Take a look.

Keep in mind: some drivers do not disable IRQs from the NIC when NAPI starts. They are supposed to, but some simply do not. You can verify this by examining the hard IRQ handler in the driver to see if hard IRQs are being disabled.

Note that hard IRQs are re-enabled in some cases as mentioned in the blog post and below.

As far as your questions:

  1. Increasing netdev_budget increases the number of packets that the NET_RX softirq can process. The number of packets that can be processed is also limited by a time limit, which is not tunable. This is to prevent the NET_RX softirq from eating 100% of CPU usage. If the device does not receive enough packets to process during its time allocation, hardirqs are reneabled and NAPI is disabled.

  2. You can also try modifying your IRQ coalescing settings for the NIC, if it is supported. See the blog post above for more information on how to do this and what this means, exactly.

  3. You should add monitoring to your /proc/net/softnet_stat file. The fields in this file can help you figure out how many packets are being processed, whether you are running out of time, etc.

A question for you to consider, if I may:

Why does your hardirq rate matter? It probably doesn't matter, directly. The hardirq handler in your NIC driver should do as little work as possible, so it executing a lot is probably not a problem for your system. If it is, you should carefully measure that as it seems very unlikely. Nevertheless, you can adjust IRQ coalescing settings and IRQ CPU affinity to distribute processing to alter the number of hardirqs generated by the NIC and processed by a particular CPU, respectively.

You should consider whether you probably are more interested in packet processing throughput or packet processing latency. Depending on which is the concern, you can tune your network stack appropriately.

Remember: to completely tune and optimize your Linux networking stack, you have to monitor and tune each component. They are all intertwined and it is difficult (and often insufficient) to monitor and tune just a single aspect of the stack.

Joe Damato
  • 1,558
  • 14
  • 15