Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-streams]

Related to Apache Kafka's built-in stream processing engine called Kafka Streams, which is a Java library for building distributed stream processing apps using Apache Kafka.

Kafka Streams is a Java library for building fault-tolerant distributed stream processing applications using streams of data records from topics in Apache Kafka.

Kafka Streams is a library for building streaming applications, specifically applications that transform input Kafka topics into output Kafka topics (or calls to external services, or updates to databases, or whatever). It lets you do this with concise code in a way that is distributed and fault-tolerant.

Documentation: https://kafka.apache.org/documentation/streams/

3924 questions
1
vote
1 answer

Need suggestion regarding Spring Boot Kafka Stream Binder Application error

I was trying to run a Spring boot Kafka Stream example from https://cloud.spring.io/spring-cloud-stream-binder-kafka/spring-cloud-stream-binder-kafka.html#_usage_2 site. Am able to successfully build it. But while running it getting error as shown…
ITIB
  • 353
  • 6
  • 16
1
vote
2 answers

How can a key/value stored in app instance A's state store be deleted using app instance B

It is my understanding that state stores are local per app instance (re: partition).... from the docs. Because Kafka Streams partitions the data for processing it, an application’s entire state is spread across the local state stores of the…
0x SLC
  • 143
  • 2
  • 10
1
vote
0 answers

Kafka Stream using same predicate to send the message to multiple branches

I needed to have same message to be given to two different branch but reading from single stream. Both branches should get all messages (i.e. my predicate for both branches is same. I tried below code but I see only first branch is…
Sudarshan sridhar
  • 201
  • 5
  • 18
1
vote
1 answer

Cannot compile maven project

I started a simple project using an artefact from confluent. But when I try to compile the project using mvn clean install or mvn clean package I am getting message like [ERROR]…
edi
  • 917
  • 7
  • 17
1
vote
1 answer

How configure the application.server dynamically for Kafka Streams remote interactive queries on a spring boot app running on Kubernetes

We have a Kubernetes cluster with three pod running, i want to know what are the RPC endpoint we need to provide in application.server to make interactive query work. So we have a use case where we need to query state-store with gRPC server. while…
Mohit Singh
  • 401
  • 1
  • 10
  • 30
1
vote
1 answer

How do you write data into a Redis custom state store using Kafka Streams

I've recently been learning about how to use the Kafka Streams client and one thing that I've been struggling with is how to switch from the default state store (RocksDB) to a custom state store using something like Redis. The Confluent…
bchen
  • 13
  • 4
1
vote
1 answer

Could Drools Fusion be integrated with kafka streams?

I am looking for Kafka Streams complex event processing (CEP). I did not find Kafka streams native CEP. However, I found drools fusion. I have a few questions? Could Drools Fusion be integrated with Kafka streams? What's the advantage of such…
Paul Yang
  • 11
  • 1
1
vote
2 answers

How to close Kafka streams application when topic not available?

I am using Kstreams with SpringBoot Application. I have added given below code to handle shutdown for the streams. KafkaStreams streams = new KafkaStreams(builder.build(), props); final CountDownLatch latch = new CountDownLatch(1); …
dev Joshi
  • 305
  • 2
  • 21
1
vote
1 answer

Kafka join storage

I use Kafka to join two streams with 3 days join window: ... private final long retentionHours = Duration.ofDays(3); ... var joinWindow = JoinWindows.of(Duration.ofMinutes(retentionHours)) …
Viktor Kurchenko
  • 35
  • 8
1
vote
1 answer

Kafka stream append to JSON as event enrichment

I have a producer that writes a json file to the topic to be read by a kafka consumer stream. Its simple key-value pair. I want to stream the topic and enrich the event by adding/concatenating more JSON key-value rows and publish to another…
Billie AK
  • 65
  • 10
1
vote
1 answer

Waiting for N Events in Kafka: How does the Group-Count-Filter Approach work?

Background In the book Designing Event-Driven Systems, the author highlighted a common use case in business systems where there is a need to wait or N events to occur. The example given is of an orders service that needs to wait for three separate…
yongjieyongjie
  • 813
  • 10
  • 18
1
vote
2 answers

KTable with billions unique keys

My requirement is to build real-time aggregation pipeline using kafka stream with large volume of data. Based on the estimate, the possible unique keys will be ~ 3 to 4 billion and total message size ~5TB. The high level architecture is, read from a…
Ravikumar
  • 1,121
  • 1
  • 12
  • 23
1
vote
1 answer

Spring Boot application using the Spring Cloud Stream Kafka Binder + Kafka Streams Binder not working - Producer doesn't send messages

My Spring Boot 2.3.1 app with SCS Hoshram.SR6 was using the Kafka Streams Binder. I needed to add a Kafka Producer that would be used in another part of the application so I added the kafka binder. The problem is the producer is not working,…
codependent
  • 23,193
  • 31
  • 166
  • 308
1
vote
1 answer

How to remove old records from a state store using a punctuator? (Kafka)

I've created a Ktable for a topic using streamsBuilder.table("myTopic"), which I materialize to a state store so that I can use interactive queries. Every hour, I want to remove records from this state store (and associated changelog topic) whose…
user6429576
1
vote
1 answer

Quarkus Kafka Streams/Reactive Messaging Deserializing Exception

Hey so I was experimenting with both Kafka Streams and MP Reactive Messaging to read from a Kafka Topic and then to produce back to it. The Kafka Streams error - org.apache.kafka.streams.errors.StreamsException: Deserialization exception handler is…
animusdx
  • 380
  • 3
  • 16
1 2 3
…
99 100