Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-streams]

Related to Apache Kafka's built-in stream processing engine called Kafka Streams, which is a Java library for building distributed stream processing apps using Apache Kafka.

Kafka Streams is a Java library for building fault-tolerant distributed stream processing applications using streams of data records from topics in Apache Kafka.

Kafka Streams is a library for building streaming applications, specifically applications that transform input Kafka topics into output Kafka topics (or calls to external services, or updates to databases, or whatever). It lets you do this with concise code in a way that is distributed and fault-tolerant.

Documentation: https://kafka.apache.org/documentation/streams/

3924 questions
1
vote
2 answers

number of partitions for global state store in kafka streams

Is global state store in kafka always one partition or we can have multiple partitions for global state store changelog topic? I am not able to find any clear documentation around this.
Ashish Vyas
  • 617
  • 1
  • 5
  • 19
1
vote
1 answer

Best topology for joining data from multiple sensors

I have n sensors generating measurements every t minutes to their own topic as follows: Topic_1: {timestamp: 1, measurement: 1}, {timestamp: 2, measurement: 4}, ... Topic_2: {timestamp: 1, measurement: 5}, {timestamp: 2, measurement: 3}, ... …
utxeee
  • 953
  • 1
  • 12
  • 24
1
vote
3 answers

Kafka internal topic : Where are the internal topics created - source or target broker?

We are doing a stateful operation. Our cluster is managed. Everytime for internal topic creation , we have to ask admin guys to unlock so that internal topics can be created by the kafka stream app. We have control over target cluster not source…
Shweta Gulati
  • 566
  • 1
  • 7
  • 17
1
vote
1 answer

KStream join fires join function instantly, how to delay it at the end of window?

As explained in the comprehensive article Crossing the Streams. The Outer KStream-KStream Join emits element as soon as it arrives, even before waiting for its match in another K-Stream. Downside of this is that it duplicates not-joined event along…
Nilesh
  • 2,089
  • 3
  • 29
  • 53
1
vote
1 answer

Kafka - Joining Data from two different Streams when Data comes at different times

I have a scenario where we have two different streams and we get data on them at two different times and i need to join them on the basis of the timestamp that is there in the value. I will try to explain through below example. inputStream1 -> key…
Shashank
  • 11
  • 3
1
vote
1 answer

How to log offset in KStreams Bean using spring-kafka and kafka-streams

I have referred almost all the questions regarding logging offset on KStreams via Processor API's transform() or process() method like mentioned in many questions here - How can I get the offset value in KStream But Im not able to get the solution…
user2340345
  • 793
  • 4
  • 16
  • 38
1
vote
0 answers

KafkaStreams stop consuming partitions after partition leader rebalance

We have experimented an issue that could be caused by the parameter auto.leader.rebalance.enable, which is set to true by default on brokers. In detail, when the automatic rebalance occurs, for example after a broker restart, some partition leaders…
Giorgio Tavecchia
  • 11
  • 1
1
vote
1 answer

Kafka state store stand by task takes long time when becomes active

We are facing an issue with kafka state store which has a standby replica. We have two topologies(streams). One without any internal state store and one with an internal state store. We have problem with the topology which has the state store. So…
Mayil
  • 37
  • 5
1
vote
0 answers

In KStreams How can I dynamically control when Ktable/Ktable joins yield results?

I have a Ktable to KTable join. I create the Ktables using .aggregate() Those yield results to the next stream processor when either side receives a new message. I have a use case where I can receive another message on the left KTable, but the…
0x SLC
  • 143
  • 2
  • 10
1
vote
1 answer

How to handle errors occurring during the processing of data in Kafka Streams

I am writing a Java application using Spring Cloud Stream Kafka Streams. Here is the functional method snippet I'm using: @Bean public Function, KStream> process() { return input -> …
Shubham
  • 109
  • 14
1
vote
1 answer

KStreams - org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before successfully committing offsets

On kafka Streams(version: 2.3.1), we are facing issues with committing offsets: org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before successfully committing offsets This is not happening regularly. What could be the…
optimus
  • 33
  • 5
1
vote
1 answer

What's a difference between a KTable obtained by KStream.toTable() and KTable materialized directly from Topic?

Is any difference between the the two following approaches of obtaining a KTable? Option 1: var kstream = streamsBuilder.stream(topicName, Consumed.with(...)); var mappedKTable = kstream.toTable(...); Option 2: var nativeKTable =…
jayachsi
  • 85
  • 1
  • 1
  • 11
1
vote
1 answer

parallel writes different topics from single stream topic

I have a stream which gives messages map to two different map() call and further is filtered and written to two different topics. KStream[] stream = builder.stream("source-topic"); stream.map(logic1OnData).filter( …
Sudarshan sridhar
  • 201
  • 5
  • 18
1
vote
1 answer

Adding data to state store for stateful processing and fault tolerance

I have a microservice that perform some stateful processing. The application construct a KStream from an input topic, do some stateful processing then write data into the output topic. I will be running 3 of this applications in the same group.…
bchu1392
  • 59
  • 2
  • 8
1
vote
1 answer

How to use multiple transformers using the same topic for kafka streams?

I need to parse complex messages on kafka using multiple transformers. Each transformer parses a part of the message and edits the message by filling some attributes on the message. In the end the fully parsed message is stored in the database using…
Niels
  • 482
  • 1
  • 5
  • 18
1 2 3
…
99 100