Python logging datagram handler

Question

I am having a problem getting the example code shown in the python docs for the logging DatagramHandler, the code shown below gives EOFError exceptions on every datagram received.

import socket
import logging
import cPickle
import struct
import sys

sock = socket.socket (socket.AF_INET, socket.SOCK_DGRAM)
sock.bind (('localhost', 9000))

handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("UDP LogViewer %(asctime)s %(message)s"))
logger = logging.getLogger("Test")
logger.addHandler(handler)

try:
    while True:
        dgram_size = sock.recv(4)
        if len(dgram_size) < 4:
            break
        slen = struct.unpack(">L", dgram_size)[0]
        data = sock.recv(slen)

        while len(data) < slen:
            data = data + sock.recv(slen - len(data))

        try:
            obj = cPickle.loads(data)
            record = logging.makeLogRecord(obj)
            logger.handle(record)
        except:
            print "exception", sys.exc_info()[0]



finally:
    sock.close()

However this code works, any ideas

data, address = sock.recvfrom(8192)
rec = logging.makeLogRecord(cPickle.loads(data[4:]))
logger.handle(rec)

Regards

score 3 · Accepted Answer · answered Dec 09 '10 at 11:52

I expect your first recv(4) call copies the first four bytes out of your datagram and then throws the rest of the packet on the floor; your second call to recv then finds nothing to read and returns EOFError. From my system's udp(7) manpage:

   All receive operations return only one packet.  When the packet
   is smaller than the passed buffer, only that much data is
   returned; when it is bigger, the packet is truncated and the
   MSG_TRUNC flag is set.  MSG_WAITALL is not supported.

Try reading in the entire datagram, grabbing the length out of the first four bytes, and then working on the subset of the array that stores the entire datagram.

Of course, if your pickles don't fit entirely within the MTU of the connection, it'll probably never work as you intend.

Thanks sarnold, I'll give that a whirl – mikip Dec 09 '10 at 12:24 — mikip, Dec 09 '10 at 12:24
Cheers sarnold, much appreciated – mikip Dec 09 '10 at 13:07 — mikip, Dec 09 '10 at 13:07

score 0 · Answer 2 · answered Sep 16 '20 at 11:14

A complete MWA looks something like this.

Notice we just discard the first four bytes (the size of the pickle-data), these exist because DatagramHandler inherits SocketHandler, which operates in TCP mode. Since we're on UDP we can just read to the end of the packet however. As Sarnold points out, if the data doesn't fit into a packet, we'd need a better solution and, since UDP can miss packets, that would require server side customisations. Perhaps the simplest server-side solution would just to be to split up any messages so we don't go over the limit in the first place!

import socketserver
import socket
import pickle
import logging

class MyDatagramRequestHandler( socketserver.DatagramRequestHandler ):
    def handle( self ):
        try:
            while True:
                chunk = self.socket.recv( 2048 )
                chunk = chunk[4:]
                obj = self.unPickle( chunk )
                record = logging.makeLogRecord( obj )
                self.on_handle( record )
        except socket.timeout:
            pass

    def unPickle( self, data ):
        return pickle.loads( data )
        
    def on_handle( self, record: logging.LogRecord ):
        # do something, e.g.
        logger = logging.getLogger( record.name )
        logger.handle( record )

Adapted from the TCP example at https://docs.python.org/3/howto/logging-cookbook.html#logging-cookbook.

Python logging datagram handler

2 Answers2