How to receive UDP packets in Python without dropping

Question

I am writing simple software to parse MPEG-TS stream to check CC (cointinuity counter) to see if any packets were dropped. When I run my script against file it works flawlessly. But when using it on UDP stream it shows losses (which are not confirmed by another software):

It is quite simple:

while True:
    received = sock.recv(7 * 188)
    parsepacket(received)

I left out parsepacket function for clarity. It is just using bitstring to analyzee packet bit by bit.

In my understanding while I run parsepacket function other UDP packets are just ignored, because I am not doing sock.recv quick enough (on 5 Mbps stream it should parse about 500 packets pers second).

I tried using sock.makefile but no luck. I get same results.

My idea is to have receiving thread running in background and another thread continously parsing what it receives. But honestly I have no idea how (besides putting it all in memory which would run out very quickly).

Why do you use UDP protocol for streaming. It seems quite counter intuitive. Now you need to implement missing functionality, like buffering and ordering, by hand. — luk32, Oct 24 '14 at 19:46
I am analysing MPEG-TS streams from our IPTV provider and they are delivered as UDP multicasts. My another idea was to use threading in python, but unfortunately it did not help. I still get dropped packets. — Michał Margula, Oct 24 '14 at 20:22

score 1 · Accepted Answer · answered Oct 25 '14 at 00:46

Without seeing your script, or how you're actually doing any of this, we won't be able to help you. However, this is trivial if you use Python's twisted library. Here's a simple example, taken from their examples page:

from twisted.internet.protocol import DatagramProtocol
from twisted.internet import reactor


class MulticastPingPong(DatagramProtocol):

    def startProtocol(self):
        """
        Called after protocol has started listening.
        """
        # Set the TTL>1 so multicast will cross router hops:
        self.transport.setTTL(5)
        # Join a specific multicast group:
        self.transport.joinGroup("228.0.0.5")

    def datagramReceived(self, datagram, address):
        print "Datagram %s received from %s" % (repr(datagram), repr(address))
        if datagram == "Client: Ping":
            # Rather than replying to the group multicast address, we send the
            # reply directly (unicast) to the originating port:
            self.transport.write("Server: Pong", address)


# We use listenMultiple=True so that we can run MulticastServer.py and
# MulticastClient.py on same machine:
reactor.listenMulticast(8005, MulticastPingPong(),
                        listenMultiple=True)
reactor.run()

You can parse each packet in the def datagramReceived function.

Jeremy Friesner · Answer 2 · 2014-10-25T01:42:28.910

My idea is to have receiving thread running in background and another thread continously parsing what it receives. But honestly I have no idea how (besides putting it all in memory which would run out very quickly).

That's probably the best way to do it, because that way your socket's incoming-UDP-packets buffer (located in the TCP stack) won't fill up while your python code is busy parsing an earlier packet.

You can implement that pattern quite easily by spawning a second thread and using a synchronized FIFO Queue to send data from the network-handling thread to the packet-parsing thread. As soon as your network thread reads a packet, it just put()'s the packet into the Queue and then goes back to read the next packet. Meanwhile, your processing thread runs in a loop calling get() on the same Queue, and wakes up only when get() returns with a packet for it to process.

One thing to watch out for is what happens if your network thread is continuously receiving packets faster than the processing thread can parse them. In that case, either your Queue will grow larger without bound (eventually using up all of your machine's RAM), or (if you've set a maximum size for the Queue), the Queue will reach its maximum size and put() will raise a Full exception.

How to receive UDP packets in Python without dropping

2 Answers2