Passing multiple buffers with iovec in C Linux sockets

Question

I'm writing a linux C client-server programs that communicates with each other with unix domain sockets and passes couple of buffers each time. I'm using ioverctors but for some reasons the server program only receives the first io vector. Any idea ? I attached the relevant code snippets.

Client code:

    struct iovec iov[2];
    struct msghdr mh;
    int rc;
    char str1[] = "abc";
    char str2[] = "1234";
    iov[0].iov_base = (caddr_t)str1;
    iov[0].iov_len = sizeof(str1);
    iov[1].iov_base = (caddr_t)str2;
    iov[1].iov_len = sizeof(str2);
 
    memset(&mh, 0, sizeof(mh));
    mh.msg_iov = iov;
    mh.msg_iovlen = 2;
 
    n = sendmsg(sockfd, &mh, 0);            /* no flags used*/
    if (n > 0) {
        printf("Sendmsg successfully executed\n");
    } 
}

Server code:
{
    struct sockaddr_un *client_sockaddr = (sockaddr_un *)opq;
    struct  msghdr msg;
    struct  iovec io[2];
    char    buf[16];
    char    buf2[16];

    io[0].iov_base      = buf;
    io[0].iov_len       = sizeof(buf);
    io[1].iov_base      = buf2;
    io[1].iov_len       = sizeof(buf2);

    msg.msg_iov     = io;
    msg.msg_iovlen  = 2;
 
    int len = recvmsg(sock, &msg, 0); 

    if (len > 0) {
        printf("recv: %s %d %s %d\n", msg.msg_iov[0].iov_base, msg.msg_iov[0].iov_len, msg.msg_iov[1].iov_base, msg.msg_iov[1].iov_len);
    }
    return 0;
}

The output i'm getting from the server: recv: abc 16 16

He he now i see the problem, replace `sizeof(str1)` and `sizeof(str2)` with `strlen(str1)` and `strlen(str2)` and see what happens. If you wonder why, understand what scatter/gather io means. In simple words, if the first buffer is filled up, the second buffer starts to fill. — Erdal Küçük, Jul 20 '21 at 15:19
Well, i'm confused. I thought that iov1 = "abc" and the iov2="123" and I could distinguish between the two in the server. The buffer I get is a concatenation of both. — Linuxuser0221, Jul 20 '21 at 15:32
You have to do that one protocol level up (basically your own). As far as i understood scatter/gather, is to use it to avoid multiple calls of the io primitives (aka read/write). One simply builds a vector of buffers (io vec) and passes that to the kernel. The kernel sends then that byte stream byte for byte to the underlying channel. But the opposite side has no idea whats comming out of the channel, for it it's simply a stream of bytes, which it will store either in one buffer or in an io vector (a large buffer cut into pieces). — Erdal Küçük, Jul 20 '21 at 15:37
Addendum: instead of calling `write` twice (to write two strings), you call `writev` once, gathering the two buffers in one call. See [man writev](https://linux.die.net/man/2/writev) — Erdal Küçük, Jul 20 '21 at 15:45

score 2 · Accepted Answer · answered Jul 21 '21 at 12:26

sendmsg(), writev(), pwritev(), and pwritev2() do not operate on multiple buffers, but one discontiguous buffer. They operate exactly as if you'd allocate a large enough temporary buffer, gather the data there, and then do the corresponding syscall on the single temporary buffer.

Their counterparts recvmsg(), readv(), preadv(), and preadv2() similarly do not operate on multiple buffers, only on one discontiguous buffer. They operate exactly as if you'd allocate a large enough temporary buffer, receive data into that buffer, then scatter the data from that buffer to the discontiguous buffer parts.

Unix domain datagram (SOCK_DGRAM) and seqpacket (SOCK_SEQPACKET) sockets preserve message boundaries, but stream sockets (SOCK_STREAM) do not. That is, using a datagram or seqpacket socket you receive each message as it was sent. With a stream socket, message boundaries are lost: two consecutively sent messages can be received as a single message, and you can (at least in theory) receive a partial message now and the rest later.

You can use the Linux-specific sendmmsg() function to send several messages in one call (using the same socket). If you use an Unix domain datagram or seqpacket socket, these will then retain their message boundaries.

Each message is described using a struct mmsghdr. It contains struct msghdr msg_hdr; and unsigned int msg_len;. msg_hdr is the same as you use when sending a single message using e.g. sendmsg(); you can use more than one iovec for each message, but the recipient will receive them concatenated into a single buffer (but can scatter that buffer using e.g. recvmsg()). msg_len will be filled in by the sendmmsg() call: the number of bytes sent for that particular message, similar to the return value of e.g. sendmsg() call when no errors occur.

The return value from the sendmmsg() call is the number of messages sent successfully (which may be fewer than requested!), or -1 if an error occurs (with errno indicating the error as usual). Thus, you'll want to write a helper function or a loop around sendmmsg() to make sure you send all the messages. For portability, I recommend a helper function, because you can then provide another based on a loop around sendmsg() for use when sendmmsg() is not available.

The only real benefit of sendmmsg() is that you need fewer syscalls to send a large number of messages: it boosts efficiency in certain situations, that's all.

Passing multiple buffers with iovec in C Linux sockets

1 Answers1