How to catch faust python json serialization error?

Question

I have simple faust agent. It consumes jsons from kafka topic, and parses them to dicts by default faust serializator:

@app.agent(source_topic, sink=[destination_topic])
async def fetch(records):
    async for record in records:
        result = do_some_stuff(record)
        yield result

The deserialization itself happens somewhere outside my code, it managed by faust framework, not by me. How I can catch and process deserialization exceptions, e.g in case of invalid json?

There are two questions you have asked, one this and another https://stackoverflow.com/questions/67326259/how-to-correctly-define-faust-agents-on-error-callback. Maybe keep it simple with one question to find the error? Unless both are different — Nagaraj Tantri, Apr 30 '21 at 13:01
I don't have the setup to test it, but there is an [Open PR](https://github.com/robinhood/faust/pull/574) which states that the `on_error` is not being invoked for the argument being passed, but also has a workaround as shown [here](https://github.com/robinhood/faust/issues/572#issuecomment-683319534), maybe you can try that — Nagaraj Tantri, Apr 30 '21 at 14:20

score 0 · Answer 1 · answered May 03 '21 at 13:11

You could manually serialize your data and catch your error:

topic = app.topic('custom', value_type=bytes)

@app.agent
async def processor(stream):
    async for payload in stream:
        data = json.loads(payload)

source: https://faust.readthedocs.io/en/latest/userguide/models.html

How to catch faust python json serialization error?

1 Answers1