Stack Overflow
Stack Overflow

Questions tagged [ksqldb]

ksqlDB: an event streaming database for Apache Kafka.

ksqlDB is a source-available event streaming database for Apache Kafka® developed by Confluent. It is distributed, scalable, reliable, and real-time. ksqlDB combines the power of real-time stream processing with the approachable feel of a relational database through a familiar, lightweight SQL syntax.

You can download ksqlDB at https://github.com/confluentinc/ksql

There is a quickstart at: https://ksqldb.io/quickstart.html

926 questions
2
votes
1 answer

Attaching JSON schema to KSQL stream records

I've been using KSQL and so far it's been working great. But now I'd like to sink the output to BigQuery via Kafka Connect, and need to attach a JSON schema. I'm having trouble figuring out how to do this. Here's my query: CREATE STREAM…
foxygen
  • 1,368
  • 1
  • 11
  • 22
2
votes
1 answer

How to list/show most recent data in a ksql stream?

I have created a ksql stream using CREATE STREAM basic_streams (data VARCHAR) \ WITH (KAFKA_TOPIC='main-topic',VALUE_FORMAT='JSON'); I have a producer running, pumping data into the stream. Is it possible to list/show the most recent items sent to…
Suicide Bunny
  • 894
  • 1
  • 10
  • 23
2
votes
1 answer

What is topic.registered in ksql when running list topics

I'm using kafka-connect to stream the rows from a mysql table to a kafka topic. When I run this on my ksql-client: LIST TOPICS; I get a list of topics looking like this: Kafka Topic | Registered | Partitions | Partition Replicas…
lloiacono
  • 4,714
  • 2
  • 30
  • 46
2
votes
2 answers

How to get window start or end time from kafka topic corresponding to KSQL windowed table?

I want to get window start timestamp (here is 1530008520000) from kafka-console-consumer command. It works with KSQL: ksql> select * from DEV_MONITOR_RULE_2557_104782_233_2_TABLE; 1530008581051 | 2557 : Window{start=1530008520000 end=-} | 2557 | 2…
Casel Chen
  • 497
  • 2
  • 8
  • 19
2
votes
1 answer

How can i query a map field in ksql?

I have created a stream from a topic i ksql. The stream has the fields as below. I can query diferent field for example: select category from fake-data-119. I would like to know how can i get a single item from the map field, for example :…
jenjen
  • 163
  • 1
  • 1
  • 10
2
votes
1 answer

KSQL : How can I change separator (comma) of DELIMITED FORMAT?

I try to put a big number of messages (350M) to customer topic (source topic) with value format like this 10957402000||10965746672||2|2756561822|452048703649890|8984048701003649890 and then I make some streams and table on that topic, but the…
PiPo Sweet Baby
  • 35
  • 3
2
votes
1 answer

Not getting result from ksql queries

I have a kafka cluster running locally and a topic named "my-topic" in which im pushing data. I also have the ksql server running and the query: SELECT*FROM "my-topic" gives me "my-topic does not exist". I understand that this query is not right and…
jenjen
  • 163
  • 1
  • 1
  • 10
2
votes
1 answer

Kafka SQL (KSQL) stream is not working for JSON data with nested fields

I'm trying to create a Kafka stream in KSQL on top of a Kafka topic. I have JSON records like below stored in a Kafka topic. { "venue": { "venue_name": "HATCH", "lon": -71.18291, "lat": 42.36667, "venue_id": 22491322 }, …
Srinivas Varala
  • 21
  • 3
2
votes
1 answer

Ksql: Left Join Displays columns from stream but not tables

I have one steam and a table in KSQL as mentioned below: Stream name: DEAL_STREAM Table name: EXPENSE_TABLE When I run the below queries it displays only columns from the stream but no table columns are being displays. Is this the expected output.…
Zamir Arif
  • 341
  • 2
  • 13
2
votes
1 answer

KSQL Kafka formatting of keys

One of my topics has a string-json as a key - {"city":"X","id":22}. In my ksql statement I want to extract it to 2 different fields not one so I can later filter and join. In the docs it seems to allow me to stick only the whole string into the key…
andrew shved
  • 137
  • 1
  • 8
2
votes
3 answers

KSQL - drop topic

Is there a way to drop the topic from KSQL? According to github it is possible, and I tried DROP TOPIC my-topic DROP TOPIC "my-topic" DROP TOPIC 'my-topic' DROP TOPIC `my-topic` But neither of this commands works. I get message …
Lanayx
  • 2,731
  • 1
  • 23
  • 35
1
vote
0 answers

KSQL: The server has encountered an incompatible entry in its log and cannot process further DDL statements

Working on a bunch of containers with docker-compose: 5 kafka broker and 3 ksqlDB Stream. Unfortunalely all the KsqlDB server status are DEGRADED and I have the following message after every command: WARNING: The server has encountered an…
CHAFARIOU Abdoussalam
  • 11
  • 1
1
vote
1 answer

How does ksqlDB recreate a table created from a Topic?

Probably some confusion from my part on how ksqlDB works. I am trying to understand how ksqlDB recreates a table in the event of a failure or repartition with the underlying streams tasks. The documentation suggests that this is from a changelog…
user2294382
  • 871
  • 3
  • 11
  • 25
1
vote
0 answers

kSQLDB: Key format: does not match any supported format with a structured key (AVRO FORMAT)

ERROR: ksql> print 'locator.key' FROM BEGINNING LIMIT 1; Key format: does not match any supported format. It may be a STRING with encoding other than UTF8, or some other format. ... key:…
Johanes
  • 11
  • 2
1
vote
0 answers

ksqlDb Unable to verify if the value schema for topic is compatible with ksqlDB

I'm trying to create a new stream and topic with ksqlDb CREATE STREAM valid_deals ( dealId BIGINT KEY ) WITH ( KAFKA_TOPIC='valid_deals', PARTITIONS=4, REPLICAS=3, VALUE_FORMAT='JSON_SR', VALUE_SCHEMA_ID=434 ); I'm using a…
andresmonc
  • 398
  • 7
  • 21
1 2 3
…
61 62