Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-streams]

Related to Apache Kafka's built-in stream processing engine called Kafka Streams, which is a Java library for building distributed stream processing apps using Apache Kafka.

Kafka Streams is a Java library for building fault-tolerant distributed stream processing applications using streams of data records from topics in Apache Kafka.

Kafka Streams is a library for building streaming applications, specifically applications that transform input Kafka topics into output Kafka topics (or calls to external services, or updates to databases, or whatever). It lets you do this with concise code in a way that is distributed and fault-tolerant.

Documentation: https://kafka.apache.org/documentation/streams/

3924 questions
1
vote
1 answer

Kafka POM Dependency Issue - ClassNotFoundException: org.apache.kafka.test.TestCondition

I'm seeing - java.lang.NoClassDefFoundError: org/apache/kafka/test/TestCondition in my integration tests. I think this is something related to way the repository import the kafka related packages. I'm adding pom dependencies with scope and…
unknown_boundaries
  • 1,482
  • 3
  • 25
  • 47
1
vote
1 answer

ProcessorContext#header() is Empty

We have kafka stream application. Producer is adding header in kafka message before sending it to Kafka Streaming application. In Kafka streaming app we are using AbstractProcessor and context.forward(null, Optional.of(event)); to forward message to…
amitwdh
  • 661
  • 2
  • 9
  • 19
1
vote
0 answers

Difference between kafka rest api and confulent rest proxy?

is there any difference between kafka rest api and confulent rest proxy. Any advantage and disadvantage for both? As i can see kafka provides rest API. so, why we use confulent rest api instead of Kafka rest API?
John
  • 11
  • 1
1
vote
1 answer

Cannot resolve symbol 'StreamsBuilder'

I'm trying to create a Kafka Streams application with Java and having a hard time managing the dependencies. Please let me say that I am completely noob to Java and it's ecosystem of tools. The project is compiled with Maven and I'm using IntelliJ…
Élodie Petit
  • 5,774
  • 6
  • 50
  • 88
1
vote
2 answers

Unit testing Kafka-streams with TopologyTestDriver and Micronaut

When I try to run a basic unit test I get the following error: java.lang.IllegalArgumentException: Unknown topic: streams-wordcount-output at org.apache.kafka.streams.TopologyTestDriver.getRecordsQueue(TopologyTestDriver.java:705) at…
Andras Hatvani
  • 4,346
  • 4
  • 29
  • 45
1
vote
1 answer

How to perform a flatTransform in a Spring Cloud Kafka Streams app?

I'm trying to perform a flatTransform in a Spring Cloud Kafka Streams app. But I'm not sure where exactly to put the KafkaStreamsStateStore annotation. At the moment I'm getting the error: Invalid topology: StateStore activeInstruments is not added…
user6429576
1
vote
1 answer

How to delete old keys from a KTable

I have a producer which publishes records with a key of either A or B to a Kafka topic. In a streams app, I'm flat-mapping each record with key A to records with key U, V or W, and each record with key B to records with key X, Y or Z. The number of…
user6429576
1
vote
1 answer

How to make Spring cloud stream Kafka streams binder retry processing a message if a failure occurs during the processing step?

I am working on Kafka Streams using Spring Cloud Stream. In the message processing application, there may be a chance that it will produce an error. So the message should not be commited and retried again. My application method - @Bean public…
shreyas.k
  • 181
  • 1
  • 2
  • 15
1
vote
1 answer

Kafka Streams: does NUM_STREAM_THREADS_CONFIG > 1 break partition's total ordering?

Here we go: I got quite complicated topology of various joins, aggregations, filters, maps, etc. By defaul the NUM_STREAM_THREADS_CONFIG parameter equals to 1 and that's completely determenistic by definition - thus, partition's total ordering (that…
Zazaeil
  • 3,900
  • 2
  • 14
  • 31
1
vote
0 answers

Quarkus native with Kafka Streams and Schema Registry

Quarkus (1.5.0.Final) as native executable works fine with Kafka Streams and Avro Schema Registry. But in case of a Kafka streams consuming a topic with Avro Serdes, if a new event is added, there is an exception: The kafka-streams-avro-serde…
Daveva
  • 21
  • 2
1
vote
1 answer

Kafka streams Exception: org.apache.kafka.streams.errors.StreamsException - Deserialization exception handler

I am trying to implement a simple counter with kafka streams. It takes a key value and adds new values if the same key comes. This is the code I wrote so far package exercises; import org.apache.kafka.clients.consumer.ConsumerConfig; import…
Bharat
  • 1,044
  • 15
  • 34
1
vote
2 answers

GlobalKTable Refresh Logic

When there are updates done to an underlying topic of a GlobalKTable, what is the logic for all instances of KStream apps to get the latest data? Below are my follow up questions: Would the GlobalKTable be locked at record level or table level when…
guru
  • 409
  • 4
  • 21
1
vote
1 answer

How to create a KStream with a collection object as value?

I am working on Spring cloud stream kafka streams binder. In my consumer bean method, I want to return KStream with List of String as value - @Bean public Function, KStream>> method() { return…
shreyas.k
  • 181
  • 1
  • 2
  • 15
1
vote
1 answer

Error while serializing aggregate state store with custom serde on Spring Cloud Stream

I'm trying to create a simple functional bean with Spring Cloud Stream that processes messages from a KStream and a GlobalKTable, joins them, aggregates them, and outputs the result to a new stream but I'm having difficulties in configuring properly…
Roberto Francescangeli
  • 25
  • 3
1
vote
1 answer

Dynamic window aggregation for each key

I have a specific use case in which I am consuming data from a single topic. That topic receives messages that contain a specific type. My service has a mapping between those types and a time window (for example type X is 1 hour, type Y is 2 hours,…
Emil Gelman
  • 125
  • 1
  • 10
1 2 3
…
99 100