0

I am testing my code on a cheap Digital Ocean droplet (2GB ram, but I also tried adding 4GB of swap and it didn't change anything).

A part of my program has many UDP client DGRAM sockets open to various local ports (from/to 127.0.0.1).

Sometimes, without an insane amount of data being sent, the send() call fails with No buffer space available (error 105).

My program then tries to use all other sockets available (to other local ports) but they all fail at the same time.

I have set /proc/sys/net/core/wmem_max (and /proc/sys/net/core/rmem_max) to 16MB, and each of these sockets has setsockopt SOL_SOCKET, SO_SNDBUF set to 8MB. None of these change anything.

How can send() fail with No buffer space available? Isn't this supposed to only happen when we send() too quickly? But here this is local and I don't send a lot of data.

What puzzles me the most is that when the process retries any another socket it gets the same error. Aren't sockets buffers supposed to be completely independent?

What are the possible causes?

FlorianB
  • 2,188
  • 18
  • 30

1 Answers1

1

The error No buffer space available does not mean the socket buffer is full. It means that linux kernel cannot allocate a memory for the socket buffer. There is no memory for any socket buffer so when this condition occurs it is not possible to send any data over any socket till some memory is freed. Increasing wmem_max and rmem_max may make the problem even worse because they may increase memory consumption per socket. You can check the overall memory consumption and how much memory is allocated for udp socket buffers:

$ cat /proc/net/sockstat
sockets: used 315
TCP: inuse 8 orphan 0 tw 0 alloc 13 mem 1
UDP: inuse 3853 mem 240812
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

UDP sockets in this example use 240812 pages per 4KB and that is ~940MB. There are 3853 opened sockets

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           1.9G        1.8G         75M        1.2M         66M         28M
Swap:            0B          0B          0B

The system i this example has 2 GB physical memory and very low free memory. It is highly probable that a send function fails with the error No buffer space available.

If the memory is actually consumed by socket buffers then adding swap hardly helps because AFAIK the buffers cannot be swapped out.

You can try a VM with more memory. It may help or at least postpone the problem.

You can also audit the application and check how are sockets used. High memory consumption by UDP socket buffers may be caused by an error in the application or by a wrong design. For example if an application listens on UDP socket and there are data but application does not read anything then kernel does not free the memory till the process terminates or till application calls recv. A lot of opened sockets without calling recv may use up memory.

Zaboj Campula
  • 3,155
  • 3
  • 24
  • 42
  • So in other words, individual socket buffers are *not* full (SO_SNDBUF) but all of them together take so much that the kernel cannot find more (non-swap) memory for them? As for increasing wmem_max/rmem_max and SO_SNDBUF someone here mentioned the memory isn't actually used until needed: https://stackoverflow.com/questions/64523733/what-is-the-predictable-behavior-of-changing-sol-socket-so-rcvbuf-on-the-fly-on Do you agree? – FlorianB Dec 13 '20 at 21:17
  • I managed to reproduce while checking both `/proc/net/sockstat` and `free -h` and the 180 sockets on average never take more than 80 pages (because most of them are just waiting to be used) but the non-swap memory however reaches a critically low point at which the issue happens. Thank you! Now I know there is nothing wrong with the sockets themselves, they are simply the first things to fail when the system runs out of memory. – FlorianB Dec 13 '20 at 23:00
  • 1
    Yes, the memory for buffer is allocated when needed - typically when data is received or application calls `send`. If the socket buffer is full (data waiting to send or data waiting to read) then kernel discards some data but does not report an error. It is allowed for UDP. But as you wrote - the problem is in the overall memory consumption. 80 pages for socket buffers are not very much. There is probably other memory eater. – Zaboj Campula Dec 14 '20 at 07:46