0

I'm using Kafka Stream to create a ktable only with data specific to client_id, which is not the topic key. I'm new to Kafka Streams that seems pretty straightforward but I got a bit confused into the multiples examples available in the community which are really good.

I'm trying to get the inputTopic data which has client_id=0123456. In KSQL below would be similar to command:

CREATE STREAM TOPIC1_CLIENT1 AS
SELECT * FROM TOPIC1
WHERE client_id= '0123456'
EMIT CHANGES;

Below I'm trying to reproduce same behavior. Can someone please tell what is i'm doing wrong on below ? It's not filtering as I expect.

        final KStream<String, String> stream = builder.stream(inputTopic, Consumed.with(stringSerde, stringSerde));
        final KTable<String, String> convertedTable = stream.filter((client_id,v) -> v.equals("0123456")).toTable(Materialized.as("stream-converted-to-table"));
        stream.to(streamsOutputTopic, Produced.with(stringSerde, stringSerde));
        convertedTable.toStream().to(tableOutputTopic, Produced.with(stringSerde, stringSerde));
mvitor
  • 37
  • 8

1 Answers1

0

v is the entire value of the message. To have named fields in KSQL, there's an associated schema on the stream, whether the data is JSON or Avro, for example, which means that the clientid is only part of the value

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • So I'm only missing to create and associate the schema in my Stream ? Thanks for looking at this. – mvitor Aug 30 '20 at 11:34
  • If so can you pls refer me to some doc. about it ? – mvitor Aug 30 '20 at 11:46
  • Kafka Streams doesn't have schemas. The problem is that you are using a string serde. I don't know what your source data format is, but assuming that you have a record like `{"client_id": 123}`, then of course that whole string will not equal 123... You can refer the Apache or Confluent websites on streams – OneCricketeer Aug 30 '20 at 14:26