Questions tagged [apache-kafka-streams]

Related to Apache Kafka's built-in stream processing engine called Kafka Streams, which is a Java library for building distributed stream processing apps using Apache Kafka.

Kafka Streams is a Java library for building fault-tolerant distributed stream processing applications using streams of data records from topics in Apache Kafka.

Kafka Streams is a library for building streaming applications, specifically applications that transform input Kafka topics into output Kafka topics (or calls to external services, or updates to databases, or whatever). It lets you do this with concise code in a way that is distributed and fault-tolerant.

Documentation: https://kafka.apache.org/documentation/streams/

3924 questions
1
vote
0 answers

Is it possible to have a 'fast' queue and a 'slow' queue using the same state stores in Kakfa Streams?

We have a series of Kafka Streams applications that perform operations on messages. The apps have state stores. Client records are fed through the apps and are enriched by using data in the state stores. The state stores are updated from other…
Jed Arndt
  • 21
  • 4
1
vote
1 answer

How to use Kafka Streams with Cassandra and AWS Redshift

I have a use case where I receive the click-streams and I need to do certain computes using data from Cassandra and finally push the computed value to Redshift. For this use case for streaming and compute I'm exploring the required Tech Stack. Is it…
Bankelaal
  • 408
  • 1
  • 9
  • 24
1
vote
1 answer

spring-kafka: Use Interactive queries in Spring Boot

What's an «idiomatic» way of configuring (providing host and port info) and accessing Interactive Queries in Kafka Streams + Spring Boot application? What's the correct way to access KafkaStreams instance to access the state stores? I'm aware of…
Vik Gamov
  • 5,446
  • 1
  • 26
  • 46
1
vote
0 answers

Reactor Kafka consumer continues to consume messages after the partitions are revoked

When the node where I am running io.projectreactor.kafka:reactor-kafka:1.1.0.RELEASE gets overloaded sometimes it happens that the consumer loses partitions. I am guessing that it can't manage to send heartbeats. This is not the problem I need help…
1
vote
1 answer

Kafka Streams replication factor not applied to state store changelog topics

We are using Kafka Streams via Spring Cloud Stream integration. I configured the replication factor to be used across all internal Kafka Streams topics by…
msilb
  • 505
  • 6
  • 18
1
vote
1 answer

Retention on Kafka Streams > 2.10

I am joining to KStreams thus I need to set JoinWindows. The joined data has computed Timestamps from CustomTimestampExtractors. The Data could be out of order. I can set the retention with the deprecated JoinWindows.until(long)-method but since its…
PSchoe
  • 66
  • 9
1
vote
1 answer

i'm trying a windowed word count application streams, in consumer console i have some unreadable characters alongwith count

The application (.java) file is as given below; public class WordCountFinal { public static void main(String[] args) { StringSerializer stringSerializer = new StringSerializer(); StringDeserializer stringDeserializer = new…
1
vote
1 answer

Kafka KTable Materialized-State-Store control

We materialize the KTable into a Internal-State-Store. a.) How and where can I specify that, this Internal-State-Store should be Persistent and be automatically backed-up to another kafka topic ? b.) How can we specify that, this…
Aditya Goel
  • 201
  • 1
  • 15
1
vote
0 answers

When you change number of partitions for user kafka topic, will the Kafka stream adjust number of partitions for internal topic?

We have a Kafka Stream which is created from 3 topics: topic1 4 partitions topic2 1 partition topic3 1 partiton. Kafka stream created internal topic with value of max(4,1,1) which would be 4 partitions. Recently we had to change number partitions…
imaru
  • 37
  • 4
1
vote
0 answers

Join one-to-many relation with spring cloud kafka stream

I'm trying to join data from two topics person and address where one person can have multiple addresses. The data published into the topics look like the following: //person with id as key {"id": "123", "name": "Tom Tester"} //addresses with id as…
1
vote
0 answers

Kafka streams send to topic in transactional manner

Post reading messages one by one into the KStreams, we apply some filtering and transformation and then produce it to the SINK topic. Now, our scenario transformation is little complex task. Is there a way to make this portion of work (i.e.…
Aditya Goel
  • 201
  • 1
  • 15
1
vote
1 answer

Is it necessary to use transactions explicitly in Kafka Streams to get "effectively once" behaviour?

A Confluence article states Stream processing applications written in the Kafka Streams library can turn on exactly-once semantics by simply making a single config change, to set the config named “processing.guarantee” to “exactly_once” (default…
Evgeniy Berezovsky
  • 18,571
  • 13
  • 82
  • 156
1
vote
1 answer

Kafka Streams. Obtaining a time ordered stream of events per key after window grace period closes

My use case involves my application receiving events, some of which we can expect to arrive out of time order (up to 2 days after the 'event' time) which I need to group by key. I don't want to aggregate the records but simply get an ordered list of…
1
vote
0 answers

Confluent Kafka JSON Deserialization Error

I have a.NET application which is writing a JSON message to Kafka TOPIC using the Confluent JSON schema aware serializer. I have another spring boot JAVA application which is reading from that topic and creating a KTABLE and grouping by Keys and…
1
vote
2 answers

how to handle one to many relationship using kafka streams join operations

can you please help me how to achieve this using Kafka streams? Scenario: Grouping all the invoices for an order data. In real-time streaming, there might be a delay in receiving invoices. so we want to wait 20 mins to group all the invoices before…