0

Lets suppose there is Kafka topic orders. Data is stored in JSON format:

{
   "order_id": 1,
   "status": 1
}

Status defines status of order (pending - 1, completed - 2).

How to change it on completed when it is finished?

As I know Kafka topic immutable and I can not change message JSON, just create a new message with chnaged value, right?

1 Answers1

1

If your order changes state, a process that is changing the state should generate a new message with the new state in the topic. The kafka streams application can react on new messages, do transformations aggregations or similar and output the modified/aggregated messages in new topics... So you need a kafka producer that when the order state changes, produces a message to the order topic.

aballaci
  • 1,043
  • 8
  • 19
  • Thank you, so stream reads data from topic? And can transform data and put it to permanent table? –  Jul 05 '20 at 08:09
  • 1
    A Ktable in kafka is an abstraction over a topic. Meaning its a structure where the values are grouped by key and each key contains the last value in the series. The output of a stream is materialised as another topic... – aballaci Jul 05 '20 at 08:15
  • So, one sub-question please, if I have topic with available users. Each user must confirm yourself. Should I create another topic confirmedUsers? –  Jul 05 '20 at 08:17
  • Is concept right - one entity changes - is separate topic? –  Jul 05 '20 at 08:20
  • 1
    That would make sense. So you would have a topic user_registration_requests and when the users confirm then you put them to a new topic registered_users. This makes it easer to build functionality on top of your data structure... So for example if you want that registration requests expire in two days, you do a retention policy of two day in the topic, then if the user does not register, the pending registrations are automatically deleted. The user token should have already expired by then. This adds flexibility. – aballaci Jul 05 '20 at 08:21
  • If status order was changed, producer sends this message to the same topic orders, right? Not creating a ne topic finished_orders? –  Jul 05 '20 at 08:26
  • How to set expired regisrations? Is it timestamp data? –  Jul 05 '20 at 08:28
  • Nothing prevents you from doing that... Consider thatn you would have to overload consumers, so that they react on different state changes. – aballaci Jul 05 '20 at 08:33