0

I'm trying to find the most efficient way to stream large amounts of data out of a UDP socket. Using a double (or triple) buffering approach in which one thread fills a large buffer with data and then another thread takes ownership of the buffer and streams it out via a socket seems more efficient than sending the sender thread small packet sized chunks.

However, the protocol I'm using requires that protocol-specific headers be added to each chunk of data before sending. The header must be at the beginning of each packet.

The best way I've thought to accomplish this so far would require memcopying every byte of data out of the large buffer before sending, similar to the code below:

void UdpSender::udp_send()
{
    /**
     * A large buffer, simulating a ping-pong buffer filled in a separate thread
     */
    const size_t BUFFER_SIZE = 1024*1024*2;
    uint8_t send_buffer[BUFFER_SIZE];
    std::memset(send_buffer, 'a', BUFFER_SIZE);


    const uint16_t MAX_PACKET_PAYLOAD_SIZE = 1400;

    struct Packet
    {
        uint32_t header_value1;
        uint32_t header_value2;
        uint8_t payload[MAX_PACKET_PAYLOAD_SIZE];
    };

    //Offset into the large buffer
    uint8_t send_buffer_offset = 0;

    Packet packet;
    packet.header_value1 = 1;
    packet.header_value2 = 2;
    //Requires copying everything out of the buffer before sending
    std::memcpy(&packet.payload, &send_buffer+send_buffer_offset, MAX_PACKET_PAYLOAD_SIZE);

    sendto(active_socket_fd, &packet, sizeof(packet), 0, p_dest_addr, dest_addr_size);
}

The data I need to send is already in a contiguous buffer that would be efficient to send (requiring no additional memcopys) if I didn't have to add a header to each chunk of data being sent out. Is there any better way I could approach this problem that would avoid memcopying the entire buffer?

Thanks!

tyler124
  • 613
  • 1
  • 8
  • 20
  • UDP is a message protocol, where TCP is a stream protocol. To use UDP that way, you want to create an application-layer protocol that can use UDP as its transport. That means making small messages to pass to UDP. That is how other streaming protocols use UDP, e.g. VoIP with RTP. Remember that you _will_ lose UDP datagrams, so your protocol either needs to be able to ignore the missing data, or you need to provide for it a way to request missing data be resent. Because UDP loses datagrams, it is better to send small datagrams so that you lose less when a datagram is lost. – Ron Maupin Mar 02 '20 at 00:27
  • Thanks for your response Ron! The protocol I'm using does handle lost packets and the other gotchas involved when using UDP outside of the code snippet provided for this post. Apologies for not explaining it well enough, but this post was asking for the most efficient way to break a large continuous buffer into small datagrams with headers (with no or as few memcopies as possible). – tyler124 Mar 02 '20 at 02:07
  • Linux has some clever ways to do scatter/gather I/O (e.g. to send a UDP packet constructed from multiple source-buffers, using a single system call, and thereby avoid the additional memory-copy that would be required to manually assemble a header-chunk and a data-chunk together before calling `send()` or `sendto()`): https://stackoverflow.com/questions/20355565/combining-sento-write-writev – Jeremy Friesner Mar 02 '20 at 04:29
  • @JeremyFriesner Thanks, that seems to be exactly what I'm looking for! – tyler124 Mar 02 '20 at 05:23
  • Though unfortunately it appears scatter/gather system calls aren't available on some embedded OSs (VxWorks/FreeRTOS). – tyler124 Mar 02 '20 at 05:43

1 Answers1

0

Most implementations of UDP (at least the standard ones) unfortunately do not support scatter/gather. But .. I've done some tests and copying a K or so of ram memory to ram memory is not as expensive as I thought, especially if you use built in functions instead of for loop.

Perry St-Germain
  • 1
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jun 03 '22 at 21:23