Packet flow in bottom half

Question

I was reading about packet flow in the receive path from NIC interrupt handler to user space.

I wanted to know till which point does the newly allocated skbuff remain in bottom half context.

Taking the snull_rx() code from LDD:

void snull_rx(struct net_device *dev, struct snull_packet *pkt)
{
    struct sk_buff *skb;
    struct snull_priv *priv = netdev_priv(dev);

    /*
     * The packet has been retrieved from the transmission
     * medium. Build an skb around it, so upper layers can handle it
     */
    skb = dev_alloc_skb(pkt->datalen + 2);
    if (!skb) {
        if (printk_ratelimit(  ))
            printk(KERN_NOTICE "snull rx: low on mem - packet dropped\n");
        priv->stats.rx_dropped++;
        goto out;
    }
    memcpy(skb_put(skb, pkt->datalen), pkt->data, pkt->datalen);

    /* Write metadata, and then pass to the receive level */
    skb->dev = dev;
    skb->protocol = eth_type_trans(skb, dev);
    skb->ip_summed = CHECKSUM_UNNECESSARY; /* don't check it */
    priv->stats.rx_packets++;
    priv->stats.rx_bytes += pkt->datalen;
    netif_rx(skb);
  out:
    return;
}

So after the netif_rx(skb) till what point will the skb remain in bottom half ?.

Thanks.

Answer 1

EDIT: I've written a blog post outlining the entire linux network stack (the receive path) that provides lots of detailed information, take a look.

The answer is complicated, but yes the netfilter code runs in the softirq context.

The way the flow works is like this:

1. a packet arrives, which enables NAPI.
2. a NAPI kernel thread running in the soft irq context (NET_RX_SOFTIRQ in /proc/softirqs) harvests packets from memory (where the NIC DMA'd the data).
3. the softirq can only consume up to its budget of packets OR a time limit on packet processing. you can find this code here.
4. This prevents the softirq from consuming the entire CPU.
5. Eventually, the function __netif_receive_skb_core is called. The exact path to this function depends on the driver, but for e1000e the path is:
   1. NAPI softirq calls e1000e_poll
   2. e1000e_poll calls e1000_clean_rx_irq
   3. e1000_clean_rx_irq calls e1000_receive_skb
   4. e1000_receive_skb calls napi_gro_receive
   5. napi_gro_receive calls napi_skb_finish
   6. napi_skb_finish calls netif_receive_skb
   7. netif_receive_skb calls netif_receive_skb
   8. netif_receive_skb calls __netif_receive_skb_core
6. Depending on whether or not you are using receive packet steering, the code paths here diverge a little bit.
7. In either case, eventually packets are delivered to the protocol layer here.
8. If we are looking at IP as our protocol of choice, packets are handed up to ip_rcv which will also check netfilter.
9. The packet continues through each of the protocol stacks until it is queued to a socket's receive buffer with a call to sock_queue_rcv_skb. For example, UDP does this here from a function called __udp_queue_rcv_skb.
10. The function sock_queue_rcv_skb queues the data to the socket receive buffer. You can find this code here.

Some notes:

• You can adjust the budget for NAPI by changing the sysctl net.core.netdev_budget. The higher the budget, the more packets will be queued to the process' receive queue, but the CPU will have less time for running user processes.
• If your NIC supports multiple RX queues, you can distribute the incoming packet processing load between multiple CPUs.
• If your NIC does not support multiple RX queues, you can use Receive Packet Steering to distribute the packet processing load across multiple CPUs.

Let me know if you have any other questions about this process that I can answer.

Answer 2

The code above runs on hardware interrupt context the netif_rx queue the packet and signal the kernel to continue on SOFTIRQ context (NET_RX)