0

Scenario: Two threads sending UDP on the same socket, and thread 1 wants to set different diffserv/QoS class than thread 2. We have solved this previously by wrapping sendto() calls in a mutex, and doing setsockopt() of appropriate QoS class before and after the sendto() (and then unlock mutex).

We got a deadlock/hang in rare circumstances (due to interaction with a signal) using this solution, and my question is this - can we remove the mutex entirely if we instead send the wanted QoS class as ancillary data in a sendmsg() call? To clarify, is sendmsg() atomic so the datagrams will be sent using correct QoS class, and not risk having the sendmsg() calls from thread 1 and 2 interfering with each other?

I have found similar questions on SO, so I know normal sendmsg() of one UDP datagram on one socket is 'atomic', but the question is if the entire call, including temporarily changing QoS bits for the socket, is atomic as seen by the user-space thread?

Relevant patch in Linux kernel is this:

commit aa6615814533c634190019ee3a5b10490026d545
Author: Francesco Fusco <ffusco@redhat.com>
Date:   Tue Sep 24 15:43:09 2013 +0200

ipv4: processing ancillary IP_TOS or IP_TTL

If IP_TOS or IP_TTL are specified as ancillary data, then sendmsg() sends out
packets with the specified TTL or TOS overriding the socket values specified
with the traditional setsockopt().

The struct inet_cork stores the values of TOS, TTL and priority that are
passed through the struct ipcm_cookie. If there are user-specified TOS
(tos != -1) or TTL (ttl != 0) in the struct ipcm_cookie, these values are
used to override the per-socket values. In case of TOS also the priority
is changed accordingly.

Two helper functions get_rttos and get_rtconn_flags are defined to take
into account the presence of a user specified TOS value when computing
RT_TOS and RT_CONN_FLAGS.

Signed-off-by: Francesco Fusco <ffusco@redhat.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Erik Alapää
  • 2,585
  • 1
  • 14
  • 25
  • I can't say for sure, but it looks like it is atomic: the `struct msghdr` gets passed to `udp_sendmsg()` in net/ipv4/udp.c, and inside that function the whole packet is constructed based on the information it has at that point, and then sent off. It's not setting some state on a socket, and then creating a packet based on the socket state and the payload you want to send. – G. Sliepen Oct 12 '16 at 12:11
  • Thank you for the analysis. I also thought about what would be a sane implementation of blocking udp send in the kernel - it would be strange if sendmsg() from thread 1 changed socket state (the QoS bits) and then went to sleep, then thread 2 does its sendmsg() on same socket, where the QoS bits are modified, then thread 2:s datagram is sent off and thread 1 wakes up, and thread 1:s datagram is sent off using wrong QoS bits. So I guess a sane implementation in the kernel should do things 'atomically'. – Erik Alapää Oct 12 '16 at 12:43

0 Answers0