ZeroMQ buffer size v/s High Water Mark

Question

In the zeromq socket options, we have flags for both a high water mark and a buffer size.

For sending, it's ZMQ_SNDHWM and ZMQ_SNDBUF.

Can someone explain the difference between these two?

Answer 1

Each one controls some other thing:

ZMQ_SNDBUF: Set kernel transmit buffer size

The ZMQ_SNDBUF option shall set the underlying kernel transmit buffer size for the socket to the specified size in bytes. A _{( default )} value of -1 means leave the OS default unchanged.

where man 7 socket says: ^{( credits go to @Matthew Slatery )}

[...]

SO_SNDBUF
          Sets or gets the maximum socket send buffer in bytes.  The  ker-
          nel doubles this value (to allow space for bookkeeping overhead)
          when it is set using setsockopt(), and  this  doubled  value  is
          returned  by  getsockopt().   The  default  value  is set by the
          wmem_default sysctl and the maximum allowed value is set by  the
          wmem_max sysctl.  The minimum (doubled) value for this option is
          2048.
[...]

NOTES
   Linux assumes that half of the send/receive buffer is used for internal
   kernel structures; thus the sysctls are twice what can be  observed  on
   the wire.
[...]

whereas

ZMQ_SNDHWM: Set high water mark for outbound messages

The ZMQ_SNDHWM option shall set the high water mark for outbound messages on the specified socket. The high water mark is a hard limit on the maximum number of outstanding messages ØMQ shall queue in memory for any single peer that the specified socket is communicating with. A _{( non-default )} value of zero means no limit.

If this limit has been reached the socket shall enter an exceptional state and depending on the socket type, ØMQ shall take appropriate action such as blocking or dropping sent messages. Refer to the individual socket descriptions in zmq_socket(3) for details on the exact action taken for each socket type.

ØMQ does not guarantee that the socket will accept as many as ZMQ_SNDHWM messages, and the actual limit may be as much as 60-70% lower depending on the flow of messages on the socket.

---

Verba docent, Exempla trahunt:

Having an infrastructure setup, where zmq.PUB side leaves all the settings in their default values, and such sender will have about 20, 200, 2000 zmq.SUB abonents listening to the sender, there would be soon depleted the default, unmodified O/S kernel buffer space, as each .bind()/.connect() relation ( each abonent ) will try to "stuff" as much as about the total cummulative sum of ~ 1000 * aVarMessageSIZE [Bytes] of data, once having been broadcast in a .send( ..., ZMQ_DONTWAIT ) manner & if the O/S would not have provided a sufficient buffer space -- there we go --

"Houston, we have a problem..."

If the message cannot be queued on the socket, the zmq_send() function shall fail with errno set to EAGAIN.

Q.E.D.