Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-streams]

Related to Apache Kafka's built-in stream processing engine called Kafka Streams, which is a Java library for building distributed stream processing apps using Apache Kafka.

Kafka Streams is a Java library for building fault-tolerant distributed stream processing applications using streams of data records from topics in Apache Kafka.

Kafka Streams is a library for building streaming applications, specifically applications that transform input Kafka topics into output Kafka topics (or calls to external services, or updates to databases, or whatever). It lets you do this with concise code in a way that is distributed and fault-tolerant.

Documentation: https://kafka.apache.org/documentation/streams/

3924 questions
10
votes
3 answers

How to deploy Kafka Stream applications on Kubernetes?

My application has some aggregation/window operation, so it has some state store which stores in the state.dir. AFAIK, it also writes the changelog of state store to the broker, so is that OK to consider the Kafka Stream application as a stateless…
a.l.
  • 1,085
  • 12
  • 29
10
votes
1 answer

Kafka Streams: Proper way to exit on error

I've been successful in getting a streams app to consume, transform and produce data, but I've noticed that periodically, the streams processor will transition to a state of ERROR and the process will sit there without exiting. Showing me logs…
rwaweber
  • 113
  • 1
  • 8
10
votes
2 answers

Kafka Streams thread number

I am new to Kafka Streams, I am currently confused with the maximum parallelism of Kafka Streams application. I went through following link and did not get the answer what I am trying to…
Arvin.Z
  • 113
  • 1
  • 1
  • 7
10
votes
1 answer

Kafka leader election causes Kafka Streams crash

I have a Kafka Streams application consuming from and producing to a Kafka cluster with 3 brokers and a replication factor of 3. Other than the consumer offset topics (50 partitions), all other topics have only one partition each. When the brokers…
Bogdan
  • 312
  • 7
  • 16
10
votes
1 answer

Kafka Streams - updating aggregations on KTable

I have a KTable with data that looks like this (key => value), where keys are customer IDs, and values are small JSON objects containing some customer data: 1 => { "name" : "John", "age_group": "25-30"} 2 => { "name" : "Alice", "age_group":…
foxygen
  • 1,368
  • 1
  • 11
  • 22
10
votes
1 answer

Failed to rebalance error in Kafka Streams with more than one topic partition

Works fine when source topic partition count = 1. If I bump up the partitions to any value > 1, I see the below error. Applicable to both Low level as well as the DSL API. Any pointers ? What could be missing…
Abhishek
  • 1,175
  • 1
  • 11
  • 21
10
votes
1 answer

How to handle error and don't commit when use Kafka Streams DSL

For Kafka Streams, if we use lower-level processor API, we can control to commit or not. So if problems happens in our code, and we don't want to commit this message. In this case, Kafka will redeliver this message multiple times until the problem…
jeffery.yuan
  • 1,177
  • 1
  • 17
  • 27
10
votes
3 answers

KStream batch process windows

I want to batch messages with KStream interface. I have a Stream with Keys/values I tried to collect them in a tumbling window and then I wanted to process the complete window at once. builder.stream(longSerde, updateEventSerde, CONSUME_TOPIC) …
samst
  • 536
  • 7
  • 19
9
votes
2 answers

How to deal with Kafka warning "Error while loading kafka-streams-version.properties java.lang.NullPointerException: inStream parameter is null"

We have a KStreams app in kotlin and switched from org.apache.kafka:kafka-streams:3.2.3 to org.apache.kafka:kafka-streams:3.3.1 and are now getting the following warning during runtime: WARN org.apache.kafka.streams.internals.metrics.ClientMetrics…
Napkuchen
  • 105
  • 1
  • 6
9
votes
1 answer

Intermittent WARN ConsumerCoordinator We received an assignment that doesn't match our current subscription

Why am I hitting the below WARN message intermittently when starting my Kafka Streams application? Its more than a warning because it floods the application logs & the Kafka Streams app doesn't start. Typically when I do a re-deploy it then…
DarVar
  • 16,882
  • 29
  • 97
  • 146
9
votes
1 answer

Kafka Streams Testing : java.util.NoSuchElementException: Uninitialized topic: "output_topic_name"

I've written a test class for kafka stream application as per https://kafka.apache.org/24/documentation/streams/developer-guide/testing.html , the code for which is import com.EventSerde; import org.apache.kafka.common.serialization.Serde; import…
Arbaz Sheikh
  • 217
  • 2
  • 10
9
votes
2 answers

Kafka Streams - Processor context commit

should we ever invoke processorContext.commit() in Processor implementation by ourselves? I mean invoking commit method inside scheduled Punctuator implementation or inside process method. in which use cases should we do that, and do we need that at…
Vasyl Sarzhynskyi
  • 3,689
  • 2
  • 22
  • 55
9
votes
4 answers

Does Kafka support java 10 or java 11?

I am planning to use kafka for stream and messaging purpose. Our infrastructure is moving to java 11. So wanted to know does kafka support java 10 or java 11 ?. Thanks
Manoj Jain
  • 111
  • 1
  • 1
  • 5
9
votes
1 answer

Aggregration and state store retention in kafka streams

I have a use case like the following. For each incoming event, I want to look at a certain field to see if it's status changed from A to B and if so, send that to an output topic. The flow is like this: An event with key "xyz" comes in with status…
sobychacko
  • 5,099
  • 15
  • 26
9
votes
0 answers

Kafka Streams: Kafka Streams application stuck rebalancing

After all Kafka brokers restart to upgrade offset.retention.minutes setting (to increase it to 60 days), the Kafka Streams application consuming there were stuck, and the consumer group shows rebalancing: bin/kafka-consumer-groups.sh…
xmar
  • 1,729
  • 20
  • 48
1 2 3
…
99 100