Stack Overflow
Stack Overflow

Questions tagged [ksqldb]

ksqlDB: an event streaming database for Apache Kafka.

ksqlDB is a source-available event streaming database for Apache Kafka® developed by Confluent. It is distributed, scalable, reliable, and real-time. ksqlDB combines the power of real-time stream processing with the approachable feel of a relational database through a familiar, lightweight SQL syntax.

You can download ksqlDB at https://github.com/confluentinc/ksql

There is a quickstart at: https://ksqldb.io/quickstart.html

926 questions
3
votes
1 answer

how to compare (1 billion records) data between two kafka streams or Database tables

we are sending data from DB2 (table-1) via CDC to Kafka topics (topic-1). we need to do reconciliation between DB2 data and Kafka topics. we have two options - a) bring down all kafka topic data into DB2 (as table-1-copy) and then do left outer…
Robin Bajaj
  • 2,002
  • 4
  • 29
  • 47
3
votes
1 answer

How to select the Key of a Record in a Select statement

I created this record: new ProducerRecord(topic = "petstore-msg-topic", key = msg.username, value = s"${msg.route},${msg.time}") I want now to do something like this: CREATE STREAM petstorePages (KEY, route VARCHAR, time VARCHAR) \ …
pme
  • 14,156
  • 3
  • 52
  • 95
3
votes
2 answers

Is it possible to use multiple left join in Confluent KSQL query? tried to join stream with more than 1 tables , if not then whats the solution?

Stream : describe ammas; Field | Type ------------------------------------- ROWTIME | BIGINT (system) ROWKEY | VARCHAR(STRING) (system) ID | INTEGER …
Karthikeyan Rasipalay Durairaj
  • 1,920
  • 13
  • 35
3
votes
2 answers

KSQL EXTRACTJSONFIELD space in key name

I'm having a problem with spaces in names when using EXTRACTJSONFIELD in KSQL For example: SELECT EXTRACTJSONFIELD(payload, '$.Successful Login') FROM MYSTREAM; --this always returns NULL How should I handle spaces in key names? I have…
Jaco O
  • 35
  • 4
3
votes
1 answer

How does Kafka Streams windowing work?

I am having a hard time comprehending how Windowing works in Kafka Streams. The results don't seem to align with what I have read and understood so far. I have created a KSQL Stream with a backing topic. one of the 'columns' in the KSQL SELECT…
rams
  • 6,381
  • 8
  • 46
  • 65
3
votes
1 answer

KSQL Non-Interactive (Headless) mode unable to find the stream

I am trying to test ksql in headless mode. I added the Create Table query in query file and started the server, $path-to-ksql/bin/ksql-server-start \ $path-to-ksql/etc/ksql/ksql-server.properties \ --queries-file /tmp/ksql-queries/queries.sql…
Rahul Bharti
  • 53
  • 8
3
votes
1 answer

Kafka ksql simple join does not work

I have rekeyed data in stream and table i use Confluent 4.1 1) Create Stream CREATE STREAM session_details_stream (Media varchar ,SessionIdTime varchar,SessionIdSeq long) with (kafka_topic = 'sessionDetails', value_format = 'json'); 2) create…
Mikhail
  • 2,690
  • 4
  • 28
  • 43
3
votes
2 answers

How to join two Kafka streams and produce the result in a topic with Avro values

I've got two Kafka Streams with keys in String and values in Avro format which I have created using KSQL. Here's the first one: DESCRIBE EXTENDED STREAM_1; Type : STREAM Key field : IDUSER Timestamp field : Not set…
Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
3
votes
1 answer

Get Latest value from Kafka

I have a Kafka topic called A. format of data in topic A is : { id : 1, name:stackoverflow, created_at:2017-09-28 22:30:00.000} { id : 2, name:confluent, created_at:2017-09-28 22:00:00.000} { id : 3, name:kafka, created_at:2017-09-28 24:42:00.000} {…
shakeel
  • 801
  • 1
  • 8
  • 24
2
votes
1 answer

How to use string array field in Kafka ksql

After creating users stream with its interests field as a string array ARRAY data type: CREATE STREAM users (userid VARCHAR, interests ARRAY) WITH (KAFKA_TOPIC = 'users', VALUE_FORMAT='JSON'); I go ahead and insert…
alphanumeric
  • 17,967
  • 64
  • 244
  • 392
2
votes
1 answer

Create a stream with a struct key in KSQL

How can I create a KSQL stream with a struct as key?
jelmew
  • 543
  • 1
  • 7
  • 15
2
votes
1 answer

How to query unique values with pull query in Kafka ksqldb

Running the command below: SELECT * FROM MY_STREAM WHERE speed != 0 GROUP BY name LIMIT 10; results to an error: Pull queries don't support GROUP BY clauses. Is there a way to query 10 records with the name value being different and unique across…
alphanumeric
  • 17,967
  • 64
  • 244
  • 392
2
votes
0 answers

KSQL: re-build entity from events

I created a topic where I store change events of an customer entity. To process the content of that topic, I created a stream with CREATE STREAM customerstream(event VARCHAR,content MAP) WITH…
user3579222
  • 1,103
  • 11
  • 28
2
votes
2 answers

Kafka Connect 'ExtractField$Key' SMT results in 'Unknown Field' error

I have a setup of Debezium connector (running on ksqlDB-server) that's streaming values from SQL Server CDC Tables to Kafka Topics. I'm trying to transform the key of my message from JSON to Integer value. The example key I'm receiving looks like…
msz
  • 21
  • 1
2
votes
0 answers

KSQL JOIN to return only one result with duplicated Keys

I have some KSQL queries running on top of Kafka (currenlty in version 5.52, planning to upgrade to lastest soon). Imagine you have 2 streams stream_A, stream_B, but you may have some duplicated id's in stream_B.id (for example some retries the…
epsilonmajorquezero
  • 559
  • 4
  • 19
1 2 3
…
61 62