How sendmsg works?

Question

As you know sendmsg has this declaration:

int sendmsg(int s, const struct msghdr *msg, int flags);

and msghdr structure has this form:

struct msghdr {
    void         * msg_name;     /* optional address */
    socklen_t    msg_namelen;    /* size of address */
    struct iovec * msg_iov;      /* scatter/gather array */
    size_t       msg_iovlen;     /* # elements in msg_iov */
    void         * msg_control;  /* ancillary data, see below */
    socklen_t    msg_controllen; /* ancillary data buffer len */
    int          msg_flags;      /* flags on received message */
};

As you see msghdr has an array of buffer, iovec and has buffer count msg_iovlen. What I wonder is that how sendmsg sends these buffers. Does it concatenate all buffers and send or does it send in a for loop?

Just a note: if this is for curiosity, then cool. If you are trying to write something that depends on this knowledge, then you're almost certainly doing it wrong and asking for trouble. — San Jacinto, Nov 23 '10 at 17:37
@SanJacinto it would be useful for you to elaborate on **why** writing something that depends on this knowledge is asking for trouble. Are you able to elaborate please? — Steve Lorimer, Jul 09 '12 at 01:55
@lori Because the documentation provides you with a set of code interfaces and tells you what to expect from them. The interfaces are very slow to change. The underlying code has no such guarantee. If you're gleaning knowledge of the internals and writing your code around that knowledge, then you should not be surprised if you update your kernel or some driver on the network stack and your code that calls it doesn't work anymore. You made a bad choice if you did this. — San Jacinto, Jul 09 '12 at 11:07

score 31 · Accepted Answer · edited Apr 30 '14 at 17:52

31

The manpage speaks of a message (singular) and multiple elements (plural):

For send() and sendto(), the message is found in buf and has length len. For sendmsg(), the message is pointed to by the elements of the array msg.msg_iov. The sendmsg() call also allows sending ancillary data (also known as control information).

For a stream socket, it wouldn't matter either way. Any data you send will just end up as one long stream of data on the other side.

For datagram or message sockets, I can see why a bit more clarity would be helpful. But it appears that you send just one datagram or message with a single sndmsg call; not one per buffer element.

I actually went digging in the Linux source code out of curiosity and to get a better feeling about this answer. It looks like send, and sendto are just wrappers for sendmsg in Linux, that build the struct msghdr for you. And in fact, the UDP sendmsg implementation makes room for one UDP header per sendmsg call.

If performance is what you're worried about, it doesn't look like you'll benefit from sendmsg if you pass in just a single iovec. If you're concatenating buffers in user-space, though, this could potentially win you some.

It's a bit similar to writev, with the added benefit that you can specify a destination address for use with connectionless sockets like UDP. You can also add ancillary data, if you're into that sort of thing. (Commonly used to send file descriptors across UNIX domain sockets.)

edited Apr 30 '14 at 17:52

Austin Adams

6,535
3
23
27

answered Nov 23 '10 at 19:20

Stéphan Kochen

19,513
9
61
50

If I've read the SCTP code right (SCTP supports SOCK_SEQPACKET sockets), it's also 1 SCTP message per sendmsg call. It's also the same for AF_UNIX SEQPACKET sockets. – ninjalj Nov 23 '10 at 19:52
"If the message is too long to pass atomically through the underlying protocol, the error EMSGSIZE is returned, and the message is not transmitted." (http://linux.die.net/man/2/sendmsg) - isn't this the exact specification for what linux does in the UDP/packet sockets ("atomically"!)? Or the other way round, if linux did something different, wouldn't it fail the specification? – Aconcagua Jan 28 '16 at 10:32
@Aconcagua I think you're looking at something different. The `iovec` stuff is about concatenating a bunch of buffers and sending them as one message. `EMSGSIZE` is something that could happen after that processing, if the total message exceeds some protocol limit. – Stéphan Kochen Feb 02 '16 at 10:35
@Shtééf Do you think so? You found out by code 'digging' that linux reserves one UDP header per sendmsg call - and I delivered the reason why it does so - because required by the standard... – Aconcagua Feb 05 '16 at 13:23
On the other hand - if the question is really for the OS internals, ninjalj's answer possibly is more appropriate for this specific question... – Aconcagua Feb 05 '16 at 13:33

Kamal · Answer 2 · 2010-11-24T01:11:20.217

2

According to http://opengroup.org/onlinepubs/007908799/xns/sendmsg.html ...

The data from each storage area indicated by msg_iov is sent in turn.

~~My interpretation is that sendmsg() will not concatenate the message data stored in the iovec's; each will be sent as a separate message.~~

[Edit: My interpretation was not correct; see the other answers for a better explanation.]

edited Nov 24 '10 at 01:11

answered Nov 23 '10 at 19:13

Kamal

7,160
2
21
12

2

Sorry, but this is incorrect in the general case. For instance, on Linux, iovec's get concatenated in the kernel in `proto_ops->sendmsg()`. – ninjalj Nov 23 '10 at 22:33
..and in particular, if you use `sendmsg()` on a `SOCK_DGRAM` socket, you'll get *one* datagram with the data from all iovecs. – caf Nov 23 '10 at 23:46

score 2 · Answer 3 · answered Nov 23 '10 at 19:59

It depends on your TCP/IP stack. Embedded TCP/IP stacks could potentially send the different iovecs directly to the NIC. But on usual TCP/IP stacks there must already be a copy from userspace memory to kernelspace memory, so there is no gain there, and iovecs get conceptually copied to a single big chunk of memory (it can be separate pages of memory, if the driver supports scather/gather I/O, but the important part here is that iovec boundaries don't get preserved).

How sendmsg works?

3 Answers3

Linked