Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-streams]

Related to Apache Kafka's built-in stream processing engine called Kafka Streams, which is a Java library for building distributed stream processing apps using Apache Kafka.

Kafka Streams is a Java library for building fault-tolerant distributed stream processing applications using streams of data records from topics in Apache Kafka.

Kafka Streams is a library for building streaming applications, specifically applications that transform input Kafka topics into output Kafka topics (or calls to external services, or updates to databases, or whatever). It lets you do this with concise code in a way that is distributed and fault-tolerant.

Documentation: https://kafka.apache.org/documentation/streams/

3924 questions
1
vote
1 answer

How to read Kafka message header from a kstream apllication

Is it possible to read Kafka message header which I set in my kafkaProducer application from kstream application? My KafkaProducer looks like this; I have set header in my message public class Producer { private final org.slf4j.Logger log =…
Bunny
  • 111
  • 2
  • 11
1
vote
0 answers

Kafka Stream State Store distribution

I have a query on sharing of state-stores. Do the applications, should have it's local state-store to exchange meta-data? I mean, let's assume there are 2 appliations, one application process data published to a topic to it's local state-store and…
Venkata Madhu
  • 93
  • 1
  • 14
1
vote
1 answer

Sending message from Kafka Streams to a WebSocket with Quarkus

So I'm not sure how to do this. I've done this using Quarkus and the MicroProfile Reactive Messaging framework and the javax.websocket library stuff but I'm not sure how I can port this to using Kafka Streams. With MP Reactive Messaging I can just…
animusdx
  • 380
  • 3
  • 16
1
vote
1 answer

kafka-streams 2.5.0 does not support Named on reduce?

Hi, I am needing to give a custom name to the internal/intermediate topics created in kafka-streams. However, only the "*repartitioned" topics get renamed, while the "aggregate" ones are not renamed. It looks as if kafka-streams 2.5.0 does not…
Nicolae Marasoiu
  • 308
  • 2
  • 6
1
vote
0 answers

Kafka Stream Co-Partition on Stream / Table Join

I read this doc where it said stream / table join must have same partitions number on both side. But I have this topics and partitions, and the code still works, no exception thrown. kafka-topics.sh --bootstrap-server localhost:9092 --create…
Timothy
  • 855
  • 1
  • 13
  • 29
1
vote
1 answer

Query KTable in the same Application where it is created

I have an Kafka streams application in which I read from a topic, do aggregation and materialize in a KTable. I then create a Stream and run some logic on the stream. Now in the stream processing, I want to use some data from the aforementioned…
ab m
  • 422
  • 3
  • 17
1
vote
1 answer

Can I rely on a in-memory Java collection in Kafka stream for buffering events by fine tuning punctuate and commit interval?

A custom processor which buffers events in a simple java.util.List in process() - this buffer is not a state store. Every 30 seconds WALL_CLOCK_TIME, punctuate() sorts this list and flushes to the sink. Assume only single partition source and sink.…
Vinodhini Chockalingam
  • 306
  • 2
  • 17
1
vote
1 answer

Kafka streams processor api single vs batch records processing

Question 1: Does kafka streams processor api allows both single and batch record processing ( or it is always batch record processing internally ) ? Question 2: What config or setting is needed to set single or batch processing and how to implement…
Maazen
  • 107
  • 1
  • 14
1
vote
1 answer

KStream - KTable Join not triggering

I’ve 2 topics(actually more but keeping it simple here) which I am joining using Streams DSL and once joined, publishing data to downstream.  I am creating a KTable on top of Topic 1 and storing it into a named state store. Key for Topic1 looks…
user123
  • 281
  • 1
  • 3
  • 16
1
vote
0 answers

Kafka Streams internal topic naming for KTable-Ktable foreign key join

I am using Kafka 2.4.1 (Confluent 5.4.1) and am trying to implement a KTable to KTable join using the new foreign key join functionality. It works great locally, however due to company policy I have to set the topic name: a.join(b, …
nsanglar
  • 1,632
  • 1
  • 14
  • 24
1
vote
1 answer

Is it possible to use @KafkaStreamsStateStore annotation on Spring Cloud Stream Kafka Streams 3.0 Binder style API methods?

Is it possible to use @KafkaStreamsStateStore annotation on Spring Cloud Stream Kafka Streams 3.0 Binder style API methods? It looks like if I'm placing @KafkaStreamsStateStore over my 3.0 style methods like @KafkaStreamsStateStore(name="my-store",…
Sergey Shcherbakov
  • 4,534
  • 4
  • 40
  • 65
1
vote
1 answer

Consuming from multiple topics using single kafka stream

which one is recommended to use : 1. Single kafka stream consuming from multiple topics 2. Different kafka streams consuming from different topics (I've used this one already with no issues encountered) Is it possible to achieve #1 ? and if yes,…
kuti
  • 161
  • 1
  • 3
  • 13
1
vote
1 answer

Kafka Streams processor api allows multi cluster topology?

Using kafka processor API( not DSL) to read from a source topic and write to target topic it works fine for a single kafka cluster setup (that is if both source and target topics reside on the same cluster) , but when source and target topics…
Maazen
  • 107
  • 1
  • 14
1
vote
1 answer

How can I pause (turn on/off) stream processing w/ Spring Cloud Stream & Kafka Streams Binder?

I'm using Spring Cloud Stream (3.0.4.RELEASE) with the Kafka-Streams binder (3.0.0.RELEASE). I'm also using the 'Functional Programming Model' (so no @StreamListener etc). What a lovely piece of tech! I need to be able to pause stream processing /…
Keir Bailey
  • 11
  • 2
1
vote
1 answer

Kafka Streams 2.3 to 2.5 upgrade breaks Scala compilation

When upgrading from KafkaStreams library from 2.3 to 2.5, keeping the same Scala version, it breaks when running with the following error: [info] com.mypackage.dp.streams.applications.EmbeddedKafkaVisitorTest *** ABORTED *** [info] …
xmar
  • 1,729
  • 20
  • 48
1 2 3
…
99 100