Stack Overflow
Stack Overflow

Questions tagged [ksqldb]

ksqlDB: an event streaming database for Apache Kafka.

ksqlDB is a source-available event streaming database for Apache Kafka® developed by Confluent. It is distributed, scalable, reliable, and real-time. ksqlDB combines the power of real-time stream processing with the approachable feel of a relational database through a familiar, lightweight SQL syntax.

You can download ksqlDB at https://github.com/confluentinc/ksql

There is a quickstart at: https://ksqldb.io/quickstart.html

926 questions
1
vote
0 answers

Ksql tumbling window count sink to postgres

I have created few tumbling window tables to count number of messages per hour/per day.I are trying to sink that in postgres database table. But when I am trying to sink it gives primary key violation instead of it updating the existing records. Is…
jsc_1999841
  • 51
  • 1
  • 4
1
vote
1 answer

How to cross join KsqlDB tables or streams?

I want to cross join 2 KsqlDB Tables. For example: Country Table data_id country 101 Malaysia 102 Singapore Product Table data_id product 301 pen 302 book The result should be: Country_Product…
user3727340
  • 31
  • 2
1
vote
1 answer

How to deal with tombstones and Struct type keys when using Debezium and KSQLDB in Confluent Cloud

I have created a Debezium Kafka connector using KSQLDB. Every time a row is removed in a table, Debezium send a tombstone like this (f.ex): KEY: Struct(cliente_cod=0000) | BODY: null When I materialize a row in a table (with KSQLDB), I have the…
José Vte. Calderón
  • 1,348
  • 12
  • 17
1
vote
0 answers

KsqlDB Materialization Exception on the pull query

I've created a pull table from two push tables based on some aggregations. As soon as I try to query the pull table I'm getting Materialization exception. TABLE1 - Push table TABLE2 -Push table TABLE3- Materialized Table formed from TABLE1 &…
dark ninja
  • 33
  • 5
1
vote
1 answer

How to create a nested json schema with Stream

The INPUT_STREAM in Kafka was created with ksql statement below: CREATE STREAM INPUT_STREAM (year STRUCT>>) WITH (KAFKA_TOPIC = 'INPUT_TOPIC', VALUE_FORMAT = 'JSON'); It defines four levels…
alphanumeric
  • 17,967
  • 64
  • 244
  • 392
1
vote
0 answers

emit final with tumbling window

Use Case : Get the message from the KSQL stream (visitor_topic1) to push in to new kafka topic (final_visitor) every 1 minutes. > CREATE TABLE final_visitors_per_min WITH (KAFKA_TOPIC='final_visitor', KEY_FORMAT='JSON', PARTITIONS = 3, REPLICAS =…
vetti
  • 11
  • 2
1
vote
0 answers

Azure powershell Graph explorer query Cannot process argument because the value of argument "name" is not valid

We have more than 1000 Azure subscriptions and some subscriptions have 1000+ resources. We are running powershell script from automation account to collect using graph explorer module to collect information about all resource in each subscription.…
A A
  • 21
  • 4
1
vote
0 answers

KSQLDB Cluster http call fails in K8s with more than one container in Deployement

So I have a problem with Clustering KSQLDB (0.27.1, 0.26.0, 0.11.0) with K8s (v1.23.8, v1.24.) Using a k8s Deployment I start 2 or 3 (replicas:) KSQLDB containers (image: confluentinc/ksqldb-server:latest). So all instances have the same…
user3008410
  • 644
  • 7
  • 15
1
vote
1 answer

N-Way table joins in ksqlDB

I have three ksqldb tables, whose relation is illustrated in the picture below. I must join them. This query will result in an error: CREATE TABLE `reviewer-email-user` AS SELECT * FROM USER INNER JOIN REVIEWER ON USER.USERID =…
Ciro di Marzo
  • 391
  • 2
  • 15
1
vote
1 answer

ksqlDB for finding average last hour, and store results back to a kafka topic?

We have a readpanda (kafka compatible) source, with sensor data. Can we do the following: Every hour, find the average sensor data last hour for each sensor Store them back to a topic
NorwegianClassic
  • 935
  • 8
  • 26
1
vote
1 answer

How to combine multiple events in a Kafka Topic to new events

I have a Kafka Topic with Json data like {"storeid": 17, "data": "blablabla"} {"storeid": 17, "data": "blublublu"} {"storeid": 18, "data": "bliblibli"} What I need is to publish new events into another topic/stream containing the combined data for…
Nikolaj
  • 49
  • 3
1
vote
0 answers

ksqlDB Java Client: Is it possible to deserialze the result of a ksqlDB query directly to a list of avro objects?

I'm new to ksqlDB and wondering if the following is possible (java client): My current situation is the following, I've: a working kafka infrastructure including the ksqlDB server and the (avro) schema registry some topics with already existing…
50k0n
  • 11
  • 2
1
vote
0 answers

How to migrate a KSQL application?

I have a complex application flow with multiple stream-stream joins and stream-table joins. To simplify things let's say I am only using stream A and B. I tried migrating the "naive" way by dropping every stream and table and recreating them.…
David Prifti
  • 621
  • 2
  • 8
  • 17
1
vote
0 answers

Working with unexpected dynamic JSON in ksqlDB

I have a case in which the client sends us completely dynamic, unexpected data. We want to do some transformations to the data, but index the dynamic data as it is. Each entry in that JSON can be of a different type, so MAP cannot…
SharonReytan
  • 73
  • 1
  • 7
1
vote
1 answer

How come KSQLDB push query outputs chunked json?

I'm new to KsqlDB so I might be missing something obvious. My question is related to the chunked JSON output of a never-ending push-query not being valid JSON. Let me elaborate. In short, my setup is as follows. From a typescript/node process I've…
Filidor Wiese
  • 664
  • 8
  • 16
1 2 3
…
61 62