1

I'm writing this Kafka streams application that takes the sensor readings that are being registered in a Kafka topic (as messages in JSON), and performs some aggregations on the value of those readings in a per-minute, per-hour and per-day basis. Then I materialize the KTables derived from those aggregates and store them using the default state store. I was wondering if it might be possible to query these tables using KSQL.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
LeandroOrdonez
  • 195
  • 2
  • 13

1 Answers1

2

While Kafka Streams is the runtime for KSQL, KTables that you crate via a Kafka Streams application are not available in KSQL. If you want to have a TABLE in KSQL, you need to write a KSQL query that creates that TABLE.

However note, that KSQL queries are continuous queries, and not "lookup" queries as in a relational database.

In contrast, Kafka Streams support a feature called "interactive queries" (https://docs.confluent.io/current/streams/developer-guide/interactive-queries.html) that allows you to do key-based lookups into the state of a KTable.

Last, there is current work in progress in KSQL, to expose "interactive queries", too.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
  • Thanks @matthias-j-sax. Regarding Kafka Streams, do you know if it is possible to query the state of a `KTable` using a key prefix? – LeandroOrdonez Oct 20 '19 at 16:31
  • 1
    It would be possible but it's hard to get right. You would need to so a `range()` query with "correct" start/end keys and filter the result manually. For example, if you look for a String "foo*" you could use "foo" as start and "fooz" as end (or similar). Figuring out the right start-end keys is important to not miss any result. Note, that the lookups are done on serialized bytes, hence, it important to understand the serialization format and byte layout to set tight boundaries. -- In doubt, maybe set the boundaries bigger to make sure you don't miss anything. – Matthias J. Sax Oct 21 '19 at 15:13