-1

I have a few questions (all related), so I'll first explain the context and then ask. My primary question is the title, but beyond that I think I just need clarification and confirmation on whether my understanding is correct.

We have some developers who are exchanging data between servers via multicast. They are sending an ~8k text file that is being transmitted as a single, large packet and other servers are listening to the multicast address to receive the file as an update. Naturally, this is getting broken up into fragments and, given it's multicast, if a fragment is lost then the whole packet is lost (which is what is happening, frequently).

When a 25MB file (powerpoint for example) is sent via email -- how is it broken down? My understanding was always that the application layer (FTP for example) would calculate the file size, break it into individual packets, and send them (while keeping track of the packets getting sent)... now if it sent it and somewhere along the path there was a smaller MTU, the router (at layer 3/IP) would fragment this. Once the fragments arrived, it would be reassembled at layer 3, then further up at layer 7, all the packets would be reassembled (for example by an FTP server). Do I understand that right?

If my understanding is right, I know that with TCP there is PMTU (Path MTU) which can check the entire path (RFC 1191) for the lowest MTU size and use that before transmission. Given that multicast uses UDP for the transport layer, I wasn't sure if there was a way for the sending host to check for this. Not only is there a problem with it being UDP, but the fact the destination would be a multicast address... so even if it could check the path, it would only know the half of the path between the sender and the multicast address, not the half of the path between the multicast address and the listener. Or is there a way to check this and still reduce the size of the transmitted packet?

Ultimately, I believe they need to reduce the size of the packets being sent and I assume this would be done at a higher layer (something in the software), so as opposed to an 8k packet, it would be 6 packets that the application tracks... though even with this 'solution' I'm not sure if it's feasible with multicast. Is it?

BenH
  • 1
  • 1
  • 1
  • 1
    In addition to breaking up your data into multiple smaller packets (which is preferable to just sending one large packet and allowing it to be fragmented, if only because you can be more intelligent about handling partial failures than the built-in fragmentation algorithm is), there is one other easy way to reduce transmission size -- zlib-compress your data before sending it (and have the receivers zlib-decompress it after receiving it). That will work particularly well on text data which is usually highly compressible. – Jeremy Friesner Oct 16 '18 at 15:15

1 Answers1

1

PMTU works at the IP layer, not the TCP layer. However since you're using multicast you're out of luck, as PMTU doesn't work with IPv4 multicast.

Your best bet is to stick with the standard 1500 byte Ethernet frame size (which includes the IP and TCP/UDP headers) and break up the data at the application layer.

With TCP this tends to be easier because it's a stream based protocol, which means the application layer doesn't need to worry about packets. For example, the sender can first send 8 bytes specifying the file size N, then send the N data bytes. Then the receiver reads 8 bytes to get the size N then reads N bytes from the socket. Breaking up the data into packets and handling the transmission and retransmission of those packets is taken care of for you by the TCP layer.

But because you're using multicast you have to use UDP, which means you need to handle breaking the data into packets, the speed those packets are send, and any necessary retransmissions yourself. The details of this aren't exactly trivial, but it is very much doable.

I wrote a multicast file transfer application called UFTP that should work for what you're doing. Give it a try and let me know how it works for you.

dbush
  • 205,898
  • 23
  • 218
  • 273