Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-streams]

Related to Apache Kafka's built-in stream processing engine called Kafka Streams, which is a Java library for building distributed stream processing apps using Apache Kafka.

Kafka Streams is a Java library for building fault-tolerant distributed stream processing applications using streams of data records from topics in Apache Kafka.

Kafka Streams is a library for building streaming applications, specifically applications that transform input Kafka topics into output Kafka topics (or calls to external services, or updates to databases, or whatever). It lets you do this with concise code in a way that is distributed and fault-tolerant.

Documentation: https://kafka.apache.org/documentation/streams/

3924 questions
11
votes
2 answers

Can Kafka Streams be configured to wait for KTable to load?

I'm using materialized KTable to use for left join with my KStream(while the stream is the left side). However, it seem to process immediately, without waiting for the current version of the KTable to load.. I have a lot of values in my source…
Ben Yaakobi
  • 1,620
  • 8
  • 22
11
votes
4 answers

Alternative to openjdk:8-alpine for Kafka Streams

I am using openjdk:8-alpine for deploying Kafka Streams application. I am using Windowing and it crashes with below error: Exception in thread "app-4a382bdc55ae-StreamThread-1" java.lang.UnsatisfiedLinkError: /tmp/librocksdbjni94709417646402513.so:…
mukesh210
  • 2,792
  • 2
  • 19
  • 41
11
votes
1 answer

java.lang.NoSuchMethodError: org.apache.kafka.common.metrics.Sensor.add error in kafka streams

I am getting the below exception when I try to create a kafka streams instance. Exception in thread "main" java.lang.NoSuchMethodError:…
Ashwin Sastry
  • 113
  • 1
  • 7
11
votes
1 answer

Kafka Streams error - Offset commit failed on partition, request timed out

We use Kafka Streams for consuming, processing and producing messages, and on PROD env we faced with errors on multiple topics: ERROR org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer…
Vasyl Sarzhynskyi
  • 3,689
  • 2
  • 22
  • 55
11
votes
1 answer

Kafka Streams: Punctuate vs Process

In a single task within the stream app, does the following two methods run independently (meaning while the method "process" is handling an incoming message from the upstream source, the method "punctuate" can also run in parallel based on the…
Raman
  • 665
  • 1
  • 15
  • 38
11
votes
4 answers

Kafka Streams - The state store may have migrated to another instance

I'm writing a basic application to test the Interactive Queries feature of Kafka Streams. Here is the code: public static void main(String[] args) { StreamsBuilder builder = new StreamsBuilder(); KeyValueBytesStoreSupplier…
Anouer Hermassi
  • 141
  • 1
  • 2
  • 9
11
votes
1 answer

Streaming messages to multiple topics

I have a single master topic and multiple predicates each of which has an output topic associated with it. I want to send each record to ALL topics that whose predicate resolves to true. I am using Luwak to test which predicates a record satisfies…
bm1729
  • 2,315
  • 3
  • 21
  • 30
10
votes
2 answers

How to achieve high availability in a Kafka Streams app during deployment?

Main question: we run Kafka Streams (Java) apps on Kubernetes to consume, process and produce real time data in our Kafka Cluster (running Confluent Community Edition v7.0/Kafka v3.0). How can we do a deployment of our apps in a way that limits…
Ismar Slomic
  • 5,315
  • 6
  • 44
  • 63
10
votes
1 answer

Kafka stream: "TopicAuthorizationException: Not authorized to access topics" for an internal state store

Java: OpenJdk 11 Kafka: 2.2.0 Kafka streams lib: 2.3.0 I am trying to deploy my Kafka streams application in a docker container and it fails while trying to create an internal state store with a TopicAuthorizationException. It works well locally.…
Arnaud Villevieille
  • 1,039
  • 2
  • 10
  • 24
10
votes
1 answer

Kafka Streams processors - state store and input topic partitioning

I would like to fully understand the rules that kafka-streams processors must obey with respect to partitioning of a processor's input and its state(s). Specifically I would like to understand: Whether or not it is possible and what are the…
Aldo Stracquadanio
  • 6,167
  • 1
  • 23
  • 34
10
votes
3 answers

how to process data in chunks/batches with kafka streams?

For many situations in Big Data it is preferable to work with a small buffer of records at a go, rather than one record at a time. The natural example is calling some external API that supports batching for efficiency. How can we do this in Kafka…
samthebest
  • 30,803
  • 25
  • 102
  • 142
10
votes
2 answers

How to debug kafkastreams code?

Has anyone managed to debug kafkastreams code written in Java 8 using IntelliJ IDEA?. I am running a simple linesplit.java code where it takes stream from one topic and splits it and sends it to another topic, but I have no idea where to keep the…
Vijaykumar Gundavarapu
  • 139
  • 1
  • 7
10
votes
2 answers

Tombstone messages not removing record from KTable state store?

I am creating KTable processing data from KStream. But when I trigger a tombstone messages with key and null payload, it is not removing message from KTable. sample - public KStream processRecord(@Input(Channel.TEST)…
R K
  • 382
  • 5
  • 25
10
votes
4 answers

Can we disable log4j logs only for kafka

I'm using following properties for Log4j: //log4j.properties log4j.rootLogger=DEBUG,…
babravahan
  • 113
  • 1
  • 1
  • 7
10
votes
5 answers

kafka streams - how to set a new key for KTable

I am new to Kafka Streams, I am using version 1.0.0. I would like to set a new key for a KTable from one of the values. When using KStream, it cane be done by using method selectKey() like this. kstream.selectKey ((k,v) -> v.newKey) However such…
Stefan Repcek
  • 2,553
  • 4
  • 21
  • 29
1 2 3
…
99 100