0

enter image description here

Data is available in chat.* topics in Kafka. We can just spin up multiple consumers and make the all hook into this single event for consumption.

But, not all products want to work on the complete events in the main topic. We want to branch the topic into multiple topics based on the content of the record in topic. How to perform content based routing in Kafka?

Kafka Streams are good for Java applications, but we need a multi stack solution that works across the companies for many languages?

We are looking for a event router kind of thing for Kafka. AWS event bridge does just that! But it is not with Kafka. How do you do such content based routing in Kafka?

Any suggestion?

Alok Kumar Singh
  • 2,331
  • 3
  • 18
  • 37
  • do you need to route when producing to topic? or consume from topic and then filter and process separately? – nipuna Jul 27 '21 at 15:38
  • Each product who needs the events should only get the subset of events in separate topics. Do i need to write a router, where in other consumers would subscribe or is there a better way to do this in Kafka? – Alok Kumar Singh Jul 27 '21 at 16:24
  • Assuming you parameterize and package/containerize a Kafka Streams (or MirrorMaker) application that will route/filter topics, does it really matter what language its in? Ultimately, this "one topic per service model" is going to stop scaling cleanly, by the way, for example, if a subset of services are hosted on the same broker, and those go down – OneCricketeer Jul 27 '21 at 17:04
  • @Cricketeer he didn't talk about his broker deployment architecture, Didn't quite understand why wouldn't it scale? For your example having 3 replica , ha deployment, should work, can you please elaborate little more? – Ran Lupovich Jul 27 '21 at 22:44
  • Yes I have a HA setups for Kafka. the replication would happen to multiple nodes. – Alok Kumar Singh Jul 28 '21 at 05:24
  • Basically, need a subscription mechanism in between so that we can branch out mutliple streams. Based on the subscription the filtering and routing happens. – Alok Kumar Singh Jul 28 '21 at 16:10

1 Answers1

0

ksqldb is the open source solution by Confluent that solves this problem.

It is built on top of Kafka Streams. It has a SQL like DSL to make following stream joins possible. Also it has an HTTP API, which allows to query it directly.

  • Support filtering and stream joins.
    • Stream-Stream Join (Window Join)
    • Stream-Table Join (Stream Enrichment)
    • Table-Table Join (Materialized View)
  • Declarative query language to specify how to create derived streams from existing streams and tables. (KSQL) API to specify the query to create derived streams. (KSQL DB API)

KSQL DB now supports adding source connectors as well. This looks brilliant! Going to try it.

Here is how KSQL DB can be used: enter image description here

Alok Kumar Singh
  • 2,331
  • 3
  • 18
  • 37