Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-streams]

Related to Apache Kafka's built-in stream processing engine called Kafka Streams, which is a Java library for building distributed stream processing apps using Apache Kafka.

Kafka Streams is a Java library for building fault-tolerant distributed stream processing applications using streams of data records from topics in Apache Kafka.

Kafka Streams is a library for building streaming applications, specifically applications that transform input Kafka topics into output Kafka topics (or calls to external services, or updates to databases, or whatever). It lets you do this with concise code in a way that is distributed and fault-tolerant.

Documentation: https://kafka.apache.org/documentation/streams/

3924 questions
1
vote
2 answers

Overriding KStreams default serializer (ByteArraySerializer)

I can't seem to override the serializer of a topic to Serdes.String(). I'm trying a simple use case of reading from a topic (stream), and writing to a KTable. What I have so far: @Component class Processor { @Autowired public void…
Tiberiu
  • 990
  • 2
  • 18
  • 36
1
vote
1 answer

Kafka Stream output rate is not as per the window

I am using Kafka stream to perform aggregation to find windowed average. ( 3 seconds window size) I tried both tumbling window and hoping window , but the output topic have results only in every 30 seconds, not in every 3 seconds. I changes the…
Jobs
  • 1,257
  • 2
  • 14
  • 27
1
vote
2 answers

Kafka stream: Monthly time windows

Based on this example (https://github.com/confluentinc/kafka-streams-examples/blob/5.5.0-post/src/test/java/io/confluent/examples/streams/window/DailyTimeWindows.java), I would like to create a Monthly time windows. The problem is the size method…
Miloune
  • 15
  • 6
1
vote
1 answer

how to write to many output types from one input type?

I am a newbie to kafka streams. I am able to achieve the following. I have an input topic. The message object (say Flower). I have a transformer that converts Flower into Fruit. I then store the Fruit to output topic. What I want to achieve is like…
user1940163
  • 191
  • 2
  • 7
1
vote
1 answer

Reading already partitioning topic in Kafka Streams DSL

Repartitioning a high-volume topic in Kafka Streams could be very expensive. One solution is to partition the topic by a key on the producer’s side and ingest an already partitioned topic in Streams app. Is there a way to tell Kafka Streams DSL that…
Boris Sukhinin
  • 110
  • 1
  • 7
1
vote
0 answers

How to sync Kafka stream commit interval with tumbling window in kafka stream?

My requirement is to get aggregated data from kafka stream with 1 min tumbling window but as commit interval is not in sync with window size it is getting committed even before the proper aggregation. kafka stream properties: …
Shivam Watts
  • 41
  • 1
1
vote
0 answers

Clean up application local store when using persistent store in a KafkaStream created with quarkus-kafka-stream plugin

I created a Kafka Streams, using quarkus-kafka-stream extension, that uses a persistent store to keep an aggregate and I would like to kwon how to reset the application properly. Cause of the persisted aggregated, it's not possible to do a clean…
Mauro Brembilla
  • 11
  • 1
  • 3
1
vote
1 answer

To close or to not close RocksDB Cache and WriteBufferManager in kafka streams app

I am currently playing around with a custom RocksDB configuration in my streams app by extending RocksDBConfigSetter interface. I see conflicting documentation around closing cache & writeBufferManager instances. Right now, I see that the javadoc &…
Wizard
  • 1,154
  • 2
  • 14
  • 41
1
vote
1 answer

Is it OK to call Kafka Streams StreamBuilder.build() multiple times

We're using micronaut/kafka-streams. Using this framework in order to create a streams application you build something like this: @Factory public class FooTopologyConfig { @Singleton @Named public KStream configureTopology { …
Sparky
  • 2,694
  • 3
  • 21
  • 31
1
vote
1 answer

Mix of State Stores and Partitions on kafka stream instances

I built a kafka streaming application with a state store. Now I am trying to scale this application. When running the application on three different servers Kafka splits up partitions and state stores randomly. For example: Instance1 gets:…
ViviZa
  • 31
  • 4
1
vote
2 answers

How to do integration testing of KStream topology using spring-boot EmbeddedKafka?

I have a simple spring-boot KStream topology that transforms a string from lowercase to uppercase. I want my integration test to launch an embedded kafka, and then test the topology. I would like to know if it possible to write integration tests…
tintin
  • 5,676
  • 15
  • 68
  • 97
1
vote
1 answer

Kafka Cluster showing continuous logs "INFO [SocketServer] Failed authentication (SSL handshake failed) (org.apache.kafka.common.network.Selector)"

I have 3 nodes of Kafka cluster in the Windows environment. I have recently added security to this existing cluster with the SASL_SSL mechanism. Here is my server.properties security configurations on each…
Namrata Khobragade
  • 67
  • 6
1
vote
1 answer

Kafka Streams JsonSerde ObjectNode cannot be cast to class

I'm working on an example by Stepane Mareek regarding KafkaStreams. Basically, the code takes transactions and calculates the AccountBalance. Unfortunately I get a ClassCastException and i do not understand why or whats happening. The…
Zimmer
  • 41
  • 6
1
vote
1 answer

Does data in a GlobalKTable expire?

I am using a GlobalKTable as a key-value store. The data in the store changes rarely, but it does change. Once a week I receive a file, and the contents of that file is put onto a stream, in a key-value pair format. This is then stored as a…
simonalexander2005
  • 4,338
  • 4
  • 48
  • 92
1
vote
1 answer

Only auto-create producer topics using spring cloud stream kafka streams binder

Is it possible to enable spring.cloud.stream.kafka.streams.binder.auto-create-topics in the application.yml only for topics that the application produces and not auto-create topics that the application consumes? I'd like for my producers to only be…
monker
  • 110
  • 1
  • 7
1 2 3
…
99 100