Python logging - batch write

Question

Is it possible within the python logging module to batch write?

For instance, collect 20 log lines (of any type, info, warning, etc) and then write them out using the handler?

Answer 1

I believe Vinay Sajip's answer may be the key to your problem.

However, from my understanding of the MemoryHandler, the messages are just stacked in the buffer and then one by one handled. If your intention is to, for example, safe expensive writes to file, this would increase the overall performance.

Alternative would be to rewrite the Handler's emit method to flush every n messages. As example code for StreamHandler would be:

class BatchStreamHandler(logging.StreamHandler):
    _BATCH_SIZE = 20
    _msg_counter = 0

    def emit(self, record):
        try:
            msg = self.format(record)
            stream = self.stream
            stream.write(msg + self.terminator)
            self._msg_counter += 1
            if self._msg_counter == self._BATCH_SIZE:
                self._msg_counter = 0
                self.flush()
        except RecursionError:  # See issue 36272
            raise
        except Exception:
            self.handleError(record)

Neverthless, I would definitely recommend you to stick to standard (tested and optimized) tools, if possible. Provided code is intended to give you an insight and also an alternative.