1

First, of, I've read around a fair amount of time including many threads on this site, however I still need some clarification on Sockets, TCP and Networking in Python, as I feel like I don't fully understand what's happening in my program.

I'm sending data from a server to a client via an Unix Domain Socket (AF_UNIX) using TCP (SOCK_STREAM).

On the server side, a process is continuously putting items on a Queue.Queue and another process is sending items to the client by running

while True:
    conn.sendall(queue.get())

On the client side, data is read by running

while True:
    conn.recv(1024)
    # time.sleep(10)

Now, I emulate a slow client by sending the client process to sleep after every call on recv(). What I expect is that the queue on the server side is filled up, since send() should block because the client can't read off data fast enough.

I monitor the number of items send to the client as well as the queue size. What I notice is that several dozen messages (roughly depending on the size of the messages, but slightly different message sizes might behave the same) are sent to the client (which are received by the client with delay, due to time.seep()) before the queue starts to fill up.

What is happening here? Why is send() not blocking immediately?

I suspect that some sort of network or file buffer is involved, which queues the send items and fills up before my implemented queue.

Scholar
  • 463
  • 5
  • 19
  • socket.send/socket.sendall is not blocking – Iain Shelvington Aug 04 '19 at 00:15
  • @IainShelvington But at some point it has to block, otherwise my queue wouldn't will up at all, wouldn't it? – Scholar Aug 04 '19 at 00:23
  • The buffer at the other end of the socket will eventually "get full" and stop accepting new data. At that point send will "block" yes – Iain Shelvington Aug 04 '19 at 00:27
  • If you have 2 processes that need to send data to each other but the producing process produced messages faster than the consumer can handle I would look at message queues as a solution – Iain Shelvington Aug 04 '19 at 00:28
  • @IainShelvington That's exactly what I implemented, yet I wonder why exactly my queue doesn't fill up immediately, even though the client doesn't read the data off fast enough. – Scholar Aug 04 '19 at 00:30

1 Answers1

3

There are a number of buffers in various places in the system, on both the sender and the receiver. Your call to a sending function won't block until all those buffers are filled up. When the receiver drains some of the buffers, data will flow again and eventually it will unblock the send call.

Typically there's a buffer in the sender holding data waiting to be put on the wire, a buffer "in flight" allowing a certain number of bytes to be send before having to wait for the receiver to acknowledge, and lastly receive buffers holding data that has been acknowledged but not yet delivered to the receiving application.

Were this not so, forward progress would be extremely limited. The sender would be stuck waiting to send until the receiver called receive. Then, whichever one finishes first would have to wait for the other one. Even if the sender was finished first, it couldn't make any forward progress at all until the receiver finished processing the previous chunk of data. That would be quite sub-optimal for most applications.

David Schwartz
  • 179,497
  • 17
  • 214
  • 278
  • Interesting, thanks. Are these buffers bound to a socket or shared? When handling multiple clients, the last thing I want is to run into situations where one slow client fills up these buffers, causing send() to block for all other clients. What would be a good place to read up on those things (while staying reasonably high-level)? – Scholar Aug 04 '19 at 01:44
  • 1
    If you want to avoid the slow-client-blocking-send problem, I suggest setting your sockets to non-blocking mode and using select() to block until it’s time to call send() (or recv()) again on a socket. Note that you’ll need to handle short-sends and EWOULDBLOCK errors gracefully in that scenario. – Jeremy Friesner Aug 04 '19 at 01:56
  • @JeremyFriesner Currently, I'm handling multiple clients using threading, but if the threads (or sockets) share the systems buffer(s), a 'rogue' client could bring the whole server to a stand-still, if the server doesn't disconnect it fast enough. – Scholar Aug 04 '19 at 02:03
  • @bi_scholar it's hard to see how it could bring the whole server to a stand-still -- AFAICT it might at worst block that thread, and then you'd have a thread that might be difficult to get rid of, but (unless that thread is holding a mutex or something while it is blocked) it shouldn't block other threads. – Jeremy Friesner Aug 04 '19 at 02:08
  • 1
    Note that the buffers are per-TCP-socket, and (presumably) each of your threads has its own separate TCP socket. In particular, if socket/connection A's buffers fill up, that won't affect socket/connection B's behavior. – Jeremy Friesner Aug 04 '19 at 02:09