1

I have a performance sensitive client and server application with stream POSIX TCP sockets. All communications on the socket are done with non-blocking flag. I am using sendmsg(2) system call to send a message with multiple buffer and use recv(2) to pick the buffers individually on the other side of the wire.

sendmsg(2) can send the message partially and I have no idea how can I tell which portion of the message is not sent so I can send it again. I don't want to send entire message again.

I did a little research and found for example Linux CIFS in this case terminates the connection session and send the entire message again. Reference. That is not really what I want.

For example:

void sys_sendmsg(int fd, struct iovec *iov, size_t length, int flags) 
{
    struct msghdr msg = (struct msghdr) {
        .msg_iov    = iov,
        .msg_iovlen = length,
    };  

    ssize_t iov_byte_len;
    for (i = 0; i < msg.msg_iovlen; ++i) {
        iov_byte_len += msg.msg_iov[i].iov_len;
    }

    ssize_t sent_len;
    sent_len = sendmsg(fd, &msg, flags);

    if( sent_len > 0 && sent_len < iov_byte_len){
        // What should I do here ?
        // I don't want to send entire message again
        // Which portion of this message is sent ?
        // Does it send the iovs sequentially ? 
    }

}
dbush
  • 205,898
  • 23
  • 218
  • 273
ARH
  • 1,355
  • 3
  • 18
  • 32
  • 1
    Are you using TCP or UDP? – Jesper Juhl Sep 04 '19 at 16:52
  • @JesperJuhl This is all TCP sockets – ARH Sep 04 '19 at 16:58
  • 4
    *"I have no idea how can I tell which portion of the message is not sent"* - sendmsg returns the total size it has send. It sends the buffers in the order you've given. It sends each buffer from the beginning. Since you know the size of the buffers you can compute where it stopped sending and thus where you need to continue. Of course, this might be inside one of the buffers. *"Does it send the iovs sequentially ?"* - sure it does, what would you expect it to do otherwise. TCP is a byte stream and not a message protocol, so there is no real choice apart from sending all in the given order – Steffen Ullrich Sep 04 '19 at 17:01
  • 1
    If you do non-blocking io then you may think about using one of the select, poll, epoll functions. This is the standard approach. You can also implement a Proactor. You may find some related stuff here: https://github.com/Armin-Montigny/SmlParser – A M Sep 04 '19 at 17:07
  • @SteffenUllrich I accept your answer. I wish if you provide a reference for the fact that iovs are sent sequentially and there is no way that they can sent out of order in the current kernel and would not be the case in the future kernel releases as well. – ARH Sep 04 '19 at 17:18
  • @ARH: It is not clearly defined in POSIX since I guess nobody even had the idea that data to an ordered byte stream could be written differently then in the most logical order in `iov`, i.e. from the start of the array to the end of the array. Similar it is not explicitly written down that the buffer in `send` gets written from start to end, since everything else would just be strange. – Steffen Ullrich Sep 04 '19 at 19:05
  • @SteffenUllrich isn't this from [here](https://pubs.opengroup.org/onlinepubs/9699919799/functions/sendmsg.html) enough?: "The data from each storage area indicated by msg_iov is sent in turn" –  Sep 04 '19 at 21:36
  • @mosvy: I'm not fully familiar with the fine details of English language. But for me "in turn" just means that they are sent in a specific and fixed order and not what this specific order actually is. This is more clearly defined in the Linux man page for [writev](https://linux.die.net/man/2/writev) which has a similar interface: *"Buffers are processed in __array order__. This means ... __iov[0] before ... iov[1]__, and so on. ... writev() __writes out the entire contents of iov[0] before proceeding to iov[1]__, and so on."* – Steffen Ullrich Sep 05 '19 at 04:10
  • See also: https://stackoverflow.com/questions/14241235/what-happens-when-i-write-data-to-a-blocking-socket-faster-than-the-other-side/14244450#14244450 which describes the TCP operation and how buffering works. Though that was written in the context of a _blocking_ send, the behavior is identical except that a _non-blacking_ `sendmsg()` returns immediately telling you how many bytes from the beginning of the buffer were added to the local send window. – Brian White Sep 05 '19 at 12:36

0 Answers0