Yet another confustion about sending/recieving large amount of data over (unix-) socket

Question

I have a C++ program which reads frames from a high speed camera and write each frame to a socket (unix socket). Each write is of 4096 bytes. Each frame is roughly 5MB. ( There is no guarantee that frame size would be constant but it is always a multiple of 4096 bytes. )

There is a python script which reads from the socket : 10 * 4096 bytes at each call of recv. Often I get unexpected behavior which I think boils down to understand the following about the sockets. I believe both of my programs are write/recving in blocking mode.

• Can I write whole frame in one go (write call with 5MB of data)? Is it recommended? Speed is major concern here.
• If python client fails to read or read slowly than write, does it mean that after some time write operation on socket would not add to buffer? Or, would they overwrite the buffer? If no-one is reading the socket, I'd not mind overwriting the buffer.

Ideally, I'd like my application to write to socket as fast as possibly. If no one is reading the data, then overwriting is fine. If someone is reading the data from socket but not reading fast enough, I'd like to store all data in buffer. Then how can I force my socket to increase the buffer size when reading is slow?

Answer 1

It sounds like a design flaw that you need to send this much data over the socket to begin-with and that there is this risk of the reader not keeping up with the writer. As an alternative, you may want to consider using a delta-encoding, where you alternate between "key frame"s (whole frames) and multiple frames encoded as deltas from the the prior frame. You may also want to consider writing the data to a local buffer and then, on your UNIX domain socket, implementing a custom protocol that allows reading a sequence of frames starting at a given timestamp or a single frame given a timestamp. If all reads go through such buffer rather than directly from the source, I imagine you could also add additional encoding / compression options in that protocol. Also, if the server application that exports the data to a UNIX socket is a separate application from the one that is reading in the data and writing it to a buffer, you won't need to worry about your data ingestion being blocked by slow readers.

Answer 2

Can I write whole frame in one go (write call with 5MB of data)? Is it recommended? Speed is major concern here.

Well, you can certainly try, but don't be too surprised if the call to socket.send() only sends a portion of the bytes you've asked it to send. In particular, you should always check the return value of socket.send() to see how many bytes it actually accepted from you, because that value may be greater than zero but less than the number of bytes you passed in to the call. (If it is less, then you'll probably want to call socket.send() again to send the remaining bytes from your buffer that weren't handled by the first call... and repeat as necessary; or alternatively you can call socket.sendall() instead of socket.send(), and that will do the necessary looping and re-calling of the socket.send() command for you, so you don't have to worry about it... the tradeoff is that socket.sendall() might not return for a long time, depending on the speed of your network connection and how much data you told socket.sendall() to send)

Note that when sending datagrams it is common for maximum packet sizes to be enforced; packets larger than that will either be fragmented into smaller packets for transmission (and hopefully re-assembled on the receiving side) or they might simply be discarded. For example, when sending UDP packets over Ethernet, it is common to have an MTU of 1500 bytes. When sending over a Unix socket the MTU will likely be larger than that, but likely there will still be a limit.

If python client fails to read or read slowly than write, does it mean that after some time write operation on socket would not add to buffer? Or, would they overwrite the buffer? If no-one is reading the socket, I'd not mind overwriting the buffer.

If you are sending on a stream-style socket (SOCK_STREAM), then a slow client will cause your server's send() calls to block if/when the buffer fills up. If you are sending on a datagram-style socket (SOCK_DGRAM) and the buffer fills up, the "overflow" datagrams will simply be discarded.

Then how can I force my socket to increase the buffer size when reading is slow?

You can set the socket's send-buffer size via socket.setsockopt(SOL_SOCKET, SO_SNDBUF, xxx). Note that this is typically done in advance (e.g. right after the socket is created) rather than trying to do it "on the fly" in response to a slow reader.