which `sock_fd` to use in `setsockopt` function

Question

My client app uses socket as http connections, meaing one-time send,receive, and close, i considered using a single persistent connection, instead of separate connection, each time, but problem was, sometimes the server received 2 packets of 32 kiB, while a single packet of 64kiB, the other times, and somtimes even smaller size, so to compensate for the latency introduced by 3 way handshake, i wanted to enable TCP_FASTOPEN, and TCP_NODELAY,

I use setsockopt with both options to the client-side(connecting) socket, but im confused that:

• At the server side, which socket to setsockopt, the one created with socket function, or the accepted one, or both, for both options,
• Also, is there any solution, to the problem, of un-intentional partitioning of data packets, like 2x32k, or 4x16k, instead of 1x64k

thanking you

I read the documentation available in linux man pages, but found it confusing

Answer 1

On the client to enable TCP_FASTOPEN, we can combine both connection establishment and data transmission simultaneously using sendto() or sendmsg(), as an example,

sfd = socket(AF_INET, SOCK_STREAM, 0); 
sendto(sfd, data, data_len, MSG_FASTOPEN, 
            (struct sockaddr *) &server_addr, addr_len);
    // Replaces connect() + send()/write()
// read and write further data on connected socket sfd
close(sfd);

However on the server side to enable TCP fastopen you need to set the option on server's listening socket as shown below,

sfd = socket(AF_INET, SOCK_STREAM, 0);   // Create socket
bind(sfd, ...);                          // Bind to well known address
int qlen = 5;                            // Value to be chosen by application
setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen));
listen(sfd, ...);                        // Mark socket to receive connections
cfd = accept(sfd, NULL, 0);              // Accept connection on new socket
// read and write data on connected socket cfd
close(cfd);

For more details worth reading https://lwn.net/Articles/508865/

To set TCP_NODELAY nagle buffering off, set the option as below on both sockets client/server,

/* Disable Nagle */
int opt = 1; // 0/1: disable/enable TCP_NODELAY
setsockopt(sfd, IPPROTO_TCP, TCP_NODELAY, &opt, sizeof(opt));

Answer 2

You're approaching this the wrong way.

TCP is a streaming protocol, so there's no guarantee that a single call to send on one side will correspond to a single call to recv on the other side. You could for example call send once and have to call recv twice to get everything, or you could call send twice and read it all in a single recv.

Your protocol needs some way of knowing exactly how much data to read. If you're actually using HTTP this is built into the protocol, i.e. read header lines separated by carriage-return/newline i.e. \r\n followed by a blank line followed by a body whose size is given by the Content-Length header.

If you're not using HTTP you could do something simple such as first sending 4 bytes for the data length followed by that many bytes of data.

In both cases, you recv as many bytes as you can and if the protocol indicates that there's more data then you recv again. You'll also need to watch for cases where you've read a full message but there's more data in the buffer you've read in, indicating that there's another message you need to start processing.

You'll need to do all this whether or not you make a new connection each time, so you might as well just leave the connection open and not worry about the socket options.