Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-streams]

Related to Apache Kafka's built-in stream processing engine called Kafka Streams, which is a Java library for building distributed stream processing apps using Apache Kafka.

Kafka Streams is a Java library for building fault-tolerant distributed stream processing applications using streams of data records from topics in Apache Kafka.

Kafka Streams is a library for building streaming applications, specifically applications that transform input Kafka topics into output Kafka topics (or calls to external services, or updates to databases, or whatever). It lets you do this with concise code in a way that is distributed and fault-tolerant.

Documentation: https://kafka.apache.org/documentation/streams/

3924 questions
1
vote
1 answer

KStreams: implementing session window with pocessor API

I need to implement a logic similar to session windows using processor API in order to have a full control over state store. Since processor API doesn't provide windowing abstraction, this needs to be done manually. However, I fail to find the…
Majid Azimi
  • 5,575
  • 13
  • 64
  • 113
1
vote
2 answers

Stop processing kafka messages if something goes wrong during process

In my processor API I store the messages in a key value store and every 100 messages I make a POST request. If something fails while trying to send the messages (api is not responding etc.) I want to stop processing messages. Until there is evidence…
Alex P.
  • 3,073
  • 3
  • 22
  • 33
1
vote
1 answer

Purge KTable entries after sending values to output topic

I have a DB that stores pageviews per webpage. It does that by consuming a Kafka topic named pageviews, where each message have the page name as the key and the value as the number of views since the previous message. This is a sample of the…
geexee
  • 339
  • 2
  • 13
1
vote
1 answer

Kafka streams: join on ingestion time

I have two topics of fairly varied volumetry (could be something like 1000 events emitted in the left topic for every event in the right topic). I'm trying to leftJoin those two topics together and I'm having the impression that the join window is…
Florentin Hennecker
  • 1,974
  • 23
  • 37
1
vote
1 answer

kafka streams runs into timeout when commiting offsets

Occasionally, my kafka streams application dies with the following error: [-StreamThread-4] o.a.k.s.p.i.AssignedStreamsTasks : Failed to commit stream task 0_9 due to the following error: org.apache.kafka.common.errors.TimeoutException: Timeout of…
D-rk
  • 5,513
  • 1
  • 37
  • 55
1
vote
1 answer

Suddenly Kafka Streams (scala) app falls into the rejoining process (with no obvious reason) and never completes it

I am messing aroung with Kafka Streams handled by the K8s. It goes more or less fine so far, yet weird behaviour is observed on the test environment: [Consumer…
Zazaeil
  • 3,900
  • 2
  • 14
  • 31
1
vote
0 answers

Kafka Streams is not detecting renewed kerberos ticket after initial tickets expiry

I've found some similar questions, but they're not quite the same situation as this. I have a Kafka Streams application which authenticates with brokers using Kerberos ticket details found within a Credential Cache. The application works great until…
Scott Smith
  • 23
  • 2
1
vote
1 answer

Processor API: bulk POST request for events stored in KeyValueStore

As suggested here https://stackoverflow.com/a/60942154/1690657 I've used the Processor API to store the incoming requests in a KeyValueStore. Every 100 events I want to send a POST Request. So I did this: public class BulkProcessor implements…
Alex P.
  • 3,073
  • 3
  • 22
  • 33
1
vote
1 answer

Can multiple consumers of kafka application work on the same thread?

On the below diagram from the Confluent site we can see that multiple consumer tasks can work on the same thread. My question is that if an application consists of multiple consumers (that can read from different topics) can it happen that some of…
Atom
  • 616
  • 1
  • 5
  • 14
1
vote
0 answers

How to instruct KStream consumer not to commit the offset when a condition is not met

We have a use-case where the Cache may not be populated occasionally. I need a way to instruct the Kstream consumer not to commit the offset when (cachePoplulated == false). The idea is to trigger a reload of the cache and try the reprocessing of…
Jack
  • 111
  • 1
  • 6
1
vote
1 answer

What does a full Kafka stream cache cause as internal operation?

When setting the cache we set the size and the commit interval ? I understand that when the commit interval is passed, a commit is called, but what operation is triggered when the cache is full. Does it also trigger a commit, causing the kafka…
MaatDeamon
  • 9,532
  • 9
  • 60
  • 127
1
vote
1 answer

is Kafka stream KTable commit interval based on Wall-clock-time?

I could not find any indication about the time used for the commit interval on KTable. Is it Walk-Clock-Time or Stream Time , Producer Time ?
MaatDeamon
  • 9,532
  • 9
  • 60
  • 127
1
vote
2 answers

Kafka Streams Flow Of Control

I have a basic question regarding the flow of control in a kafka stream application. If there are two source topics A & B. Lets suppose that A has records with timestamps that are earlier than B. Is there a guarantee of the order in which the…
Sumit Baurai
  • 233
  • 1
  • 12
1
vote
1 answer

Kafka Streams - Using An Existing State Store After Adding a New Source Stream

I have an existing stream which uses two topics as its source: val streamsBuilder = new StreamsBuilder val stream1 = streamsBuilder.stream[K, V]("topic1") val stream2 = streamsBuilder.stream[K, V]("topic2") stream1 .merge(stream2) .groupByKey …
Feyyaz
  • 3,147
  • 4
  • 35
  • 50
1
vote
1 answer

Will changing replication factor of Kafka Streams internal topics affect kafka streams? Will streaming be in error state?

I have two Kafka streams applications running on two different nodes. Brokers are 5. Replication factor of topics and changelog topics is 2. Now I want to increase or decrease replication factor for internal topics. Would this affect my running…
Sagar
  • 1,115
  • 2
  • 11
  • 23
1 2 3
…
99
100