I have two NICs on a RHEL 6 server. They are both set at 1GB/sec full duplex. The first NIC has lots of small (< 100 byte) TCP socket writes. The second NIC is used to receive a 50 MB/sec TCP input stream where latency is critical.
In this setup, will the first NIC adversely impact the latency on the second NIC?
It depends on the type of NIC. If they're both "Soft NICs" that rely on the CPU to do processing for them, then yeah there could be an impact because you're wasting CPU time. If they're on a card with 2 ports and the card only has one processor for both of them, similarly that can impact them. Really nice dual interface NICs have a processor per port to avoid that sort of impact, but without knowing more I can't say for sure.
It is theoretically possible but with such low data rates unlikely to be measurable. The NIC sends interrupts to one or more processors, the OS can be configured to limit the routing of these IRQs to specific cores. Thus if one core is processing all IRQs then a significant number of interrupts from one NIC will delay the processing of interrupts from another.
At a practical level quad-Ethernet cards can implement IRQ throttling, sharing and coalescing for the entire card thus reducing the impact of large data rates. However do realise these are problems for very high data rates, i.e. 10GigE networks and millions of packets per second.
There is an article on how IRQ routing aided performance for one company here:
http://www.kernel.org/doc/Documentation/IRQ-affinity.txt
http://lserinol.blogspot.com/2009/02/irq-affinity-in-linux.html
