Getting last log value with Structlog and passing it as a variable to a function

Question

Currently I am trying to implement a function call that sends failed messages from a converter to a DLQ topic with Kafka. As part of the DLQ message I want to include the exception error that we also log.

the code:

except json.decoder.JSONDecodeError:
          log.error('failed to parse json',
                    topic=msg.topic(),
                    partition=msg.partition(),
                    offset=msg.offset()
                    )                                 
          produce_dlq_message(converters.get("DLQ"), msg, error_message)

I need to get the value of that latest log.error() call and assign it to variable: error_message

I call this exact same function at another exception block but it has a different value in log.error() call so I need something that gets the last/latest error message.

Answer 1

While there might be a way to achieve that with multistructlog, processors and whatnot, I would recommend to take the low-tech route:

try:
   ...
except json.decoder.JSONDecodeError:
    em = {
        "event": "failed to parse json",
        "topic": msg.topic(),
        "partition": msg.partition(),
        "offset": msg.offset(),
    }

    log.error(**em)
    produce_dlq_message(converters.get("DLQ"), msg, json.dumps(em))

It means em gets serialized twice, but I think overall the simplicity makes it worth it.

Answer 2

I used this solution in the end.

try:
   ...
except json.decoder.JSONDecodeError as e:
   log.error("failed to parse json",
              topic=msg.topic(),
              partition=msg.partition(),
              offset=msg.offset(),
              )

    error_message = str(e)
    produce_dlq_message(converters.get("DLQ"), msg, error_message)