1

I'm writing a multi-process UDP server, which uses SO_REUSEPORT to let multiple worker processes listen on the same port.

Is there a way of telling the kernel that I would like consistent delivery of packets to processes, based on the packets' source address?

(The server needs to hold a little state, per source address. It would be easier to have state local to the processes, rather than for state to have to be shared between processes.)

fadedbee
  • 42,671
  • 44
  • 178
  • 308
  • Presuming you are trying to separate the streams, what if one host launches multiple clients (thus streams)? Wouldn’t you still need to separate them? – mevets Jul 10 '19 at 16:27
  • I really (really) don't recommend a process per client. I don't recommend using a thread per client either... unless, of course, you know your server will never experience high loads of concurrency. Consider the fact that, at 1K clients, you will have 1K processes / thread. That's a lot of overhead - at 2Mb stack per thread, the cheapest memory cost will be 2Gb, and I'm not even considering the memory used for copying the page table (when forking) or the added cost in context switching. – Myst Jul 11 '19 at 07:11
  • @Myst Sorry, I wasn't clear - I'm using one worker process per core. I would like to hold each client's state in only one of eight worker processes. – fadedbee Jul 11 '19 at 16:55

2 Answers2

2

You should take a look and configure RPS: Receive Packet Steering:

The first step in determining the target CPU for RPS is to calculate a flow hash over the packet’s addresses or ports (2-tuple or 4-tuple hash depending on the protocol). This serves as a consistent hash of the associated flow of the packet. The hash is either provided by hardware or will be computed in the stack.

...

Each receive hardware queue has an associated list of CPUs to which RPS may enqueue packets for processing. For each received packet, an index into the list is computed from the flow hash modulo the size of the list. The indexed CPU is the target for processing the packet, and the packet is queued to the tail of that CPU’s backlog queue.

...

RPS scales kernel receive processing across CPUs without introducing reordering. The trade-off to sending all packets from the same flow to the same CPU is CPU load imbalance if flows vary in packet rate.

Another option is Intel Ethernet Flow Director:

Intel Ethernet FD supports advanced filters that direct received packets to different queues, and enables tight control on flow in the platform. It matches flows and CPU cores where the processing application is running for flow affinity, and supports multiple parameters for flexible flow classification and load balancing. When operating in Application Targeting Routing (ATR) mode, Intel Ethernet FD is essentially the hardware offloaded version of Receive Flow Steering available on Linux* systems, and when running in this mode, Receive Packet Steering and Receive Flow Steering are disabled.

Maxim Egorushkin
  • 131,725
  • 17
  • 180
  • 271
1

You might be able to achieve a similar result using a BPF with the SO_ATTACH_REUSEPORT_CBPF or SO_ATTACH_REUSEPORT_EBPF socket options to assign each client to a specific socket index.

mshamory
  • 84
  • 5