Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-streams]

Related to Apache Kafka's built-in stream processing engine called Kafka Streams, which is a Java library for building distributed stream processing apps using Apache Kafka.

Kafka Streams is a Java library for building fault-tolerant distributed stream processing applications using streams of data records from topics in Apache Kafka.

Kafka Streams is a library for building streaming applications, specifically applications that transform input Kafka topics into output Kafka topics (or calls to external services, or updates to databases, or whatever). It lets you do this with concise code in a way that is distributed and fault-tolerant.

Documentation: https://kafka.apache.org/documentation/streams/

3924 questions
20
votes
2 answers

Kafka streams use cases for add global store

When defining a topology in kafka streams, a global state store can be added. It will need a source topic as well as a ProcessorSupplier. The processor receive records and could theorically transform them before adding them to the store. But in case…
zoom
  • 1,686
  • 2
  • 16
  • 27
19
votes
4 answers

A default binder has been requested, but there are no binders available for 'org.springframework.cloud.stream.messaging.DirectWithAttributesChannel'

I am trying to create the simplest as possible hello world with Spring Cloud + Kafka Streams + Spring Boot 2. I realize I miss basic concepts. Basically, I understand that: 1 - I need to define an outbound stream to write messages to a Kafka topic,…
Jim C
  • 3,957
  • 25
  • 85
  • 162
19
votes
5 answers

Kafka INVALID_FETCH_SESSION_EPOCH

We are using a kafka broker setup with a kafka streams application that runs using Spring cloud stream kafka. Although it seems to run fine, we do get the following error statements in our log: 2019-02-21 22:37:20,253 INFO…
mmelsen
  • 636
  • 1
  • 8
  • 24
19
votes
3 answers

Kafka Streams with Spring Boot

I want to work with Kafka Streams real time processing in my spring boot project. So I need Kafka Streams configuration or I want to use KStreams or KTable, but I could not find example on the internet. I did producer and consumer now I want to…
Alpcan Yıldız
  • 671
  • 3
  • 13
  • 33
18
votes
2 answers

Implement Kafka Streams Processor in .Net?

Is that possible? The official .Net client confluent-kafka-dotnet only seems to provide consumer and producer functionality. And (from what I remember looking into Kafka streams quite a while back) I believe Kafka Streams processors always run on…
Evgeniy Berezovsky
  • 18,571
  • 13
  • 82
  • 156
17
votes
1 answer

Can I set Kafka Stream consumer group.id?

I'm using Kafka Stream library for streaming application. I wanted to set kafka consumer group id. Then, I put Kafka stream configuration like below. streamsCopnfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "JoinTestApp"); …
Jeahyun Kim
  • 283
  • 1
  • 3
  • 8
17
votes
2 answers

Kafka Streams and RPC: is calling REST service in map() operator considered an anti-pattern?

The naive approach for implementing the use case of enriching an incoming stream of events stored in Kafka with reference data - is by calling in map() operator an external service REST API that provides this reference data, for each incoming…
Eugene Khyst
  • 9,236
  • 7
  • 38
  • 65
16
votes
1 answer

Kafka Streams - SerializationException: Unknown magic byte

I am trying to create a Kafka Streams Application which processes Avro records, but I am getting the following error: Exception in thread "streams-application-c8031218-8de9-4d55-a5d0-81c30051a829-StreamThread-1"…
R. B
  • 165
  • 1
  • 2
  • 8
16
votes
2 answers

KafkaStreams shuts down with no exceptions

I have four instances of a Kafka stream application running with the same application id. All the input topics are of a single partition. To achieve scalability I have passed it through an intermediate dummy topic with multiple partitions. I have…
Viswapriya
  • 191
  • 2
  • 4
16
votes
2 answers

Multiple streams from a single master topic

How can I make multiple streams from a single master topic? When I do something like this: KStreamBuilder builder = new KStreamBuilder(); builder.stream(Serdes.String(), Serdes.String(), "master") /* Filtering logic */ …
bm1729
  • 2,315
  • 3
  • 21
  • 30
15
votes
1 answer

Kafka Streams: use the same `application.id` to consume from multiple topics

I have an application that needs to listen to multiple different topics; each topic has separate logic for how the messages are handled. I had thought to use the same kafka properties for each KafkaStreams instance, but I get an error like the one…
Rylander
  • 19,449
  • 25
  • 93
  • 144
15
votes
2 answers

Kafka - This server is not the leader for that topic-partition

I have two broker kafka 0.10.2.0 cluster.Replication factor is 2. I am running 1.0.0 kafka stream application against this Kafka. In my kafka stream application, producer config has retries = 10 and retry.backoff.ms = 100 After running few minutes,…
Rednam Nagendra
  • 353
  • 2
  • 8
  • 20
14
votes
2 answers

Kafka Stream Suppress session-windowed-aggregation

I have written this code in a Kafka stream application: KGroupedStream groupedStream = stream.groupByKey(); groupedStream.windowedBy( SessionWindows.with(Duration.ofSeconds(3)).grace(Duration.ofSeconds(3))) .aggregate(() -> {...}) …
dborn
  • 143
  • 1
  • 5
14
votes
1 answer

What are the differences between KTable vs GlobalKTable and leftJoin() vs outerJoin()?

In Kafka Stream library, I want to know difference between KTable and GlobalKTable. Also in KStream class, there are two methods leftJoin() and outerJoin(). What is the difference between these two methods also? I read KStream.leftJoin, but did not…
Ajit Dongre
  • 871
  • 2
  • 8
  • 11
14
votes
1 answer

Dynamically connecting a Kafka input stream to multiple output streams

Is there functionality built into Kafka Streams that allows for dynamically connecting a single input stream into multiple output streams? KStream.branch allows branching based on true/false predicates, but this isn't quite what I want. I'd like…
kellanburket
  • 12,250
  • 3
  • 46
  • 73
1
2
3
…
99 100