Stack Overflow
Stack Overflow

Questions tagged [ksqldb]

ksqlDB: an event streaming database for Apache Kafka.

ksqlDB is a source-available event streaming database for Apache Kafka® developed by Confluent. It is distributed, scalable, reliable, and real-time. ksqlDB combines the power of real-time stream processing with the approachable feel of a relational database through a familiar, lightweight SQL syntax.

You can download ksqlDB at https://github.com/confluentinc/ksql

There is a quickstart at: https://ksqldb.io/quickstart.html

926 questions
3
votes
0 answers

Inconsistent results when counting Kafka records using ksqldb and confluent-kafka

I wish to count the number of records in a given Kafka topic to set up a monitoring endpoint as a kind of 'sanity check' for data-in vs. data-out. I noticed that doing this with confluent-kafka 1.9.2 produces results that differ from those produced…
filpa
  • 3,651
  • 8
  • 52
  • 91
3
votes
0 answers

Has anyone figured out how to use KSQLDB with AWS GLUE Schema Registry?

We're trying to use AWS MSK (managed Kafka), and we want to use AWS GLUE Schema registry with AVRO rather than Confluent Schema Registry. We have brought up KSQLDB, and connected that to MSK, but it needs ksql.schema.registry.url and all that is…
ChrisAddams
  • 190
  • 1
  • 8
3
votes
0 answers

How to create an output stream (changelog) based on a table in KSQL correctly?

Step 1: Create table I currently have a table in KSQL which created by CREATE TABLE cdc_window_table WITH (KAFKA_TOPIC='cdc_stream', VALUE_FORMAT='JSON') AS SELECT after->application_id AS application_id, COUNT(*) AS…
Hongbo Miao
  • 45,290
  • 60
  • 174
  • 267
3
votes
1 answer

KSQL: Invalid join condition: foreign-key table-table joins are not supported as part of n-way when joining more than 2 Table

I have 3 topics with KAFKA_INT key customer,orders,orderdetails When i do joining 2 tables only select * from orders o join customer c on o.custid = c.custid emit changes; -- OK select od.id, od.orderid from orderdetails od join orders o on…
taymedee
  • 484
  • 2
  • 5
  • 11
3
votes
2 answers

KSQLDB asp.net core

I don't see any article/blog written around KSQL DB for asp.net core and there are less/no documentation available. It would be appreciated if you could help me with the code/article/blog on kSQLDB with asp.net core
Anup Hosur
  • 31
  • 1
3
votes
1 answer

How to delete a value from ksqldb table or insert a tombstone value?

How is it possible to mark a row in a ksql table for deletion via Rest api or at least as a statement in ksqldb-cli? CREATE TABLE movies ( title VARCHAR PRIMARY KEY, id INT, release_year INT ) WITH ( …
Kubus
  • 677
  • 6
  • 18
3
votes
3 answers

How do I install just the ksqlDB CLI tool on Linux?

This seems like something that should be so simple! (Every other CLI tool I use has an easy install process.) We're running ksqlDB in a Kubernetes cluster, and I would like to connect to it with the CLI from my local machine (either Windows 10 or…
Robert
  • 348
  • 3
  • 11
3
votes
1 answer

Kafka topic with Variable nested JSON object as KSQL DB stream

I'm trying to join two existing Kafka topics in KSQL. Some data samples from Kafka (actual values redacted due to corporate environment): device topic: { "persistTime" : "2020-10-06T13:30:25.373Z", "previous" : { "device" : "REDACTED", …
UltraAlkaline
  • 61
  • 1
  • 8
3
votes
2 answers

Why KSQL query returns null values from the stream created by stream-stream join?

I'm facing an unexpected behaviour on stream-stream join query result. Situation KSQL Version: 5.1.3 There are 2 streams created from each kafka topic #1. CREATE STREAM streamA (id VARCHAR) WITH (KAFKA_TOPIC='topicA',…
yuto
  • 31
  • 3
3
votes
1 answer

Kafka KSQLDB server logs constantly "found no committed offset for partition"

I run Kafka and a KSQLDB server on headless mode. On the KSQLDB Server, I have only deployed a couple of queries to experiment with: CREATE STREAM pageviews_original (viewtime bigint, userid varchar, pageid varchar) WITH…
GaryW
  • 57
  • 1
  • 1
  • 5
3
votes
3 answers

Kafka streams - KSQL - Split messages and publish to another topic

Is there a way to split a message into multiple messages using KSQL and publish to a new topic. Just to be clear, I am not looking for a Java based listener and iterate/stream it to a new topic; instead, I am looking for a KSQL that does that for…
so-random-dude
  • 15,277
  • 10
  • 68
  • 113
3
votes
1 answer

KSQL select one row per group that corresponds to having the least timestamp

In KSQL is there a row_number kind of function that can be used in combination with TUMBLING WINDOW in order to group by and only get the event that corresponds to having the least timestamp within the group by thats used.
Vikas J
  • 358
  • 1
  • 5
  • 17
3
votes
1 answer

Kafka KSQL Re partition and rekey problem

I have defined a stream CREATE STREAM QUOTE (quoteId VARCHAR, counterPartyId VARCHAR) WITH (KAFKA_TOPIC='quotes', VALUE_FORMAT='JSON', KEY='quoteId'); I want to aggregate how many quotes I…
Russell You
  • 31
  • 2
3
votes
1 answer

What does KSQL WARN Warning: window end time was truncated to Long.MAX mean?

I get ksql WARN Warning: window end time was truncated to Long.MAX in ksql-server.stdout. I would like to know in which occasion this warning is fired and the degree of importance, as the query that generate this warning also provides the expected…
Sebastien
  • 31
  • 1
3
votes
1 answer

KSQL Multiple Column Join - LogicalBinaryExpression cannot be cast to ComparisonExpression

I am investigating some things for work where we have to prove the value of Kafka and KSQL as part of a POC. We are working on simple game events where a player can make a play and the game engine either assigns them points (considered a win) or…
KrylixZA
  • 31
  • 3
1 2
3
…
61 62