How to send none values (tombstones) to a kafka topic using Faust

Question

I'm trying to send data from one topic in Kafka to another using Faust. If the value in the original topic is None (the message is a tombstone), I'm sending the current key with a None value to the target topic.

async def order_delete(key, target_topic):
    await target_topic.send(key=key, value=None, headers={'__op': b'd'})
                                                       

@app.agent(topic)
async def order_info(orders):
    async for key, order in orders.items():

        if order is None:
            for target_topic in target_topics.values():
                await order_delete(key, target_topic)
            continue

I expect it to be a tombstone in the target topic, but it's not. It has a delete header and the value of:

"ERROR":{ "message":"src property must be a valid json object" }

I'm new to Faust so I'm probably missing something... Is there a way to send tombstones with it?

score 0 · Answer 1 · answered Feb 18 '22 at 15:03

0

Looks like you are having a similar issue as the one flagged here https://github.com/robinhood/faust/issues/143

answered Feb 18 '22 at 15:03

Ftisiot

1,808
1
7
13

already tried it, but it doesn't send tombstones – Dmitriy Fedotov Mar 27 '22 at 18:33

score 0 · Answer 2 · edited Mar 03 '22 at 15:28

0

Simplest solution for me looks like sending empty dictionary {}. It is still valid json.

edited Mar 03 '22 at 15:28

aaossa

3,763
2
21
34

answered Mar 03 '22 at 14:16

Иван Деркач

51
2

In the end I did exactly that But I was just wondering if you could send a tombstone in faust – Dmitriy Fedotov Mar 27 '22 at 18:36

score 0 · Answer 3 · answered Mar 27 '22 at 18:34

0

I asked the guys in faust slack channel and it seems there is no way to send tombstones in faust. In the end I decided to go with sending an empty json

answered Mar 27 '22 at 18:34

Dmitriy Fedotov

1

How to send none values (tombstones) to a kafka topic using Faust

3 Answers3