1

I realize it's a bad idea to rely on the UDP protocol to provide any sort of ordering guarantees and TCP should be used instead. Please refrain from answers suggesting I use TCP.

I am debugging some legacy networking code and was wondering what are the ordering guarantees provided by the socket interface. The setup I have consists of a linux box running Debian talking to an embedded device over a direct ethernet cable. The embedded device cannot fit an entire TCP stack and even if it did, the legacy stack is too old to refactor.

Specifically, if I have a NIC configured with the default single pfifo_fast, and I am sending packets over a single socket, using a single thread, and they all have the same ToS, under these conditions, am I guaranteed that all my packets will be sent over the wire in the order I send them?

This is the behavior I observe in practice but I haven't found any standard, POSIX or otherwise that guarantees this behavior and I would like to make sure this is in fact supported behavior under the environments and assumptions I listed above.

In contrast, if I send my packets over two separate sockets I observe that they are not sent over the NIC in the same order I sent them from the application code. It's clear that the kernel is reserving the right to re-order packets sent over separate sockets, but refrains from doing so when using a single socket.

As I understand it, calling send() on a socket places the packet into the appropriate queue for the NIC synchronously. The existence of a queue suggests to me a notion of order (otherwise a list would be a more appropriate name for the data structure). Whether or not such ordering guarantees exist, I am interested in some sort of documentation or specification stating so.

red0ct
  • 4,840
  • 3
  • 17
  • 44
gilsho
  • 11
  • 3
  • 1
    They are not guaranteed to *arrive* in order, or at all, which is the important thing. What happens at the sender is therefore irrelevant. – user207421 Jul 22 '19 at 21:48
  • 2
    If it wasn't relevant I wouldn't have asked it! :) For more context, this is an existing protocol implemented to communicate to an embedded device that cannot fit an entire TCP stack. The linux box and the embedded device are connected through a direct cable, so no re-ordering can take place in the network. – gilsho Jul 22 '19 at 22:21
  • 1
    If it is relevant you haven't stated how. Even if there was a cast-iron guarantee at the sender it wouldn't help anything at the receiver. Any element in the network, including both the sender and the receiver, is free to re-order UDP datagrams. – user207421 Jul 22 '19 at 23:06
  • 1
    I think the question may make for an interesting thought experiment when using Unix domain sockets on the local machine, where the datagram may not enter the network stack. If that is the case then you should add the additional information to the question. Otherwise, you get the standard guarantees. But also see [Ensuring packet order in UDP](https://stackoverflow.com/q/3745115/608639) and [Why do I get UDP datagrams out of order even with processes running locally?](https://stackoverflow.com/q/2533873/608639) – jww Jul 23 '19 at 05:27
  • Thanks for the references. I clarified my setup in the question. It is indeed entering the networking stack, but the packet is only traversing a single link. The links you provided are not too informative (but thanks for providing them). In one of the links the windows networking stack is used which I know nothing about, and in the other link a race condition appears to be the culprit. – gilsho Jul 23 '19 at 08:22

1 Answers1

1

There's no guarantee of this behavior because in the general case it would be irrelevant, as user207421 mentioned. POSIX, or even Linux for that matter, wouldn't guarantee that behavior because it necessarily constrains the implementation for an extremely uncommon case. Reordering packets for various reasons is common, and allowing Linux to do that for performance or other reasons (e.g. packet filtering or QoS) improves throughput.

Even if the sender did guarantee this behavior, the receiver could still experience an overfull buffer or temporary network hardware issue that would prevent the packet from being received properly, so any guarantee on the sending side would still be meaningless. You just can't rely on in-order delivery of UDP packets without a higher-level protocol on top, no matter what.

If you need in-order retrieval and retries but can't use TCP, look at QUIC for an example of how to do it. You may (or may not) want to implement the crypto portion, but the protocol layering may be helpful.

bk2204
  • 64,793
  • 6
  • 84
  • 100
  • 1
    As far as I am aware the reordering of packets due to packet filtering and QoS happens at the network queues, which as I mentioned I am using the default pfifo_fast that only does minimal filtering based on the ToS bits, and all of the packets I send have the same ToS bits set. I understand that there are other network issues such as drops and delays that I need to be handled, but I am trying to understand if re-ordering is not one of these issues. Furthermore, I am dealing with an existing system, so redesigning it is out of scope. – gilsho Jul 23 '19 at 08:14
  • Also, many answers here assume the UDP packets will be traversing the internet. It is not. It is only traveling over a dedicated ethernet cable. – gilsho Jul 23 '19 at 08:16
  • My answer stands. There is no guarantee it will work and Linux can do whatever it wants with your packets. Even if it works now, it may not in the future. I understand that may be inconvenient, but that's the way it is. – bk2204 Jul 23 '19 at 22:22
  • Thank you for the answer. I believe this position is very internet-centric, assuming all networking logic is inherently internet driven. I don't think that is correct. Linux powers all sorts of embedded devices and other applications that might require carefully interacting with network drivers. It's already the case that once a packet reaches the network queue configured in the manner I mentioned that there will be ordering imposed. It is hard to therefore imagine the kernel going through all that effort to order packets once they reach the network queues, but then re-order them just before. – gilsho Jul 24 '19 at 15:03