Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-streams]

Related to Apache Kafka's built-in stream processing engine called Kafka Streams, which is a Java library for building distributed stream processing apps using Apache Kafka.

Kafka Streams is a Java library for building fault-tolerant distributed stream processing applications using streams of data records from topics in Apache Kafka.

Kafka Streams is a library for building streaming applications, specifically applications that transform input Kafka topics into output Kafka topics (or calls to external services, or updates to databases, or whatever). It lets you do this with concise code in a way that is distributed and fault-tolerant.

Documentation: https://kafka.apache.org/documentation/streams/

3924 questions
1
vote
1 answer

Scala - How to filter KStream (Kafka Streams)

I'm new on Scala and I'm trying to filter a KStream[String, JsonNode] based on the second component fields. As a example, the working Java code is this: import org.apache.kafka.streams.StreamsBuilder; import…
Ángel Igualada
  • 891
  • 9
  • 13
1
vote
2 answers

How to modify KStream key and values in Kafka word count program?

I am new to Kafka Streams and kind of stuck in basic word count program. In the below program, I am trying to change the case of value but it's not working (val wordCountInputProcessed = wordCountInput.mapValues(value => value.toLowerCase)). Is…
Goldie
  • 164
  • 12
1
vote
1 answer

Kafka streams suppress does not return any values

Im struggling on a relatively simple Windowed Word Count example. I'm trying to get only the windowed results but does not receive anything at all. KStream sl = s ... .groupBy((key, value) -> value) …
Patrick Schubert
  • 43
  • 5
1
vote
2 answers

How to change the default serdes of @KafkaStreamsStateStore kafka Stream Cloud

How to change the default serdes of @KafkaStreamsStateStore? I know that in the new version 3.0.1 in Kafka stream cloud the way is explained here :…
Koprinka
  • 11
  • 3
1
vote
1 answer

INCOMPLETE_SOURCE_TOPIC_METADATA error in KafkaStreams

I writing KafkaStreams application and set the maximum.num.threads as one. I've three source topics and have 6,8,8 partitions respectively. Currently running this streamtopology with 4 instances so 4 running streams threads. I'm getting…
PrabaharanKathiresan
  • 1,099
  • 2
  • 17
  • 32
1
vote
1 answer

How to distribute Kafka-Streams state store on multiple disks

On a Kafka Broker, it's recommended to use multiple drives for the message logs to improve throughput. That's why they have a log.dirs property that can have multiple directories that will be assigned to partitions in a round-robin fashion. We have…
GeertPt
  • 16,398
  • 2
  • 37
  • 61
1
vote
1 answer

Creating Global State Store in Kafka Streams (Spring)

I am new to Kafka and have tried to create a small Kafka KTable implementation. I have successfully added a KTable and was able to query. I have used local state store and it worked as expected. Below is my Local State Store Config @Bean(name =…
Kumar Pallav
  • 590
  • 1
  • 6
  • 16
1
vote
2 answers

Get all events with specific key from Kafka

I have topics in my system that store events for given entities. Now I would like to perform some analysis on the eventlog. Therefore I need to query all events belonging to some entity within a certain time period. Is there a possibility to…
jossdb
  • 43
  • 4
1
vote
1 answer

How does a global state store differ from a normal state store in kafka?

How does a global State Store differ from a normal state store ? Has a global state store a copy of the data in all the instances running on different machines ? How does it behave in case of a restart ? Because a global state store doesn't use any…
Mohit Singh
  • 401
  • 1
  • 10
  • 30
1
vote
1 answer

Kafka streams Ktable Ktable join emits no result in output topic

I am trying a very simple Ktable to Ktable join in Scala. Both topics have single partition but still I don't see anything coming to the 'output-topic' object SimpleMerge extends App { val properties = KafkaProperties.get() val streamBuilder =…
Vipin
  • 11
  • 2
1
vote
2 answers

Kafka Tombstoning with a DSL KStream to KTable Transformation

I have a KStream which I essentially want to convert to a KTable The only way I could find to achieve this using the DSL is with a map, group then reduce. val stream: KStream = ... val table: KTable =…
Eduardo
  • 6,900
  • 17
  • 77
  • 121
1
vote
1 answer

Permanent Kafka Streams/KSQL retention policy

I'm presently working on an use case in which user interaction with a platform is tracked, thus generating a stream of events that gets stored into kafka and will be subsequently processed in Kafka Streams/KSQL. But I've run into an issue concerning…
gmilan
  • 27
  • 1
  • 6
1
vote
1 answer

Kafka Stream: What happens to processor object when REBALANCE occur

1) We use Low level processor API. When the rebalance occurs does the processor object gets re-initialized and init method invoked again? 2) Is REBALANCE partition specific (single application having multiple threads consuming different…
Raman
  • 665
  • 1
  • 15
  • 38
1
vote
1 answer

Kafka streams dropping messages during windowing and on restart

I am working on Kafka Streams application with following topology: private final Initializer> eventInitializer = () -> new HashSet<>(); final StreamsBuilder streamBuilder = new StreamsBuilder(); final KStream
Rohan Kapoor
  • 11
  • 2
1
vote
1 answer

how to send json data stream to multiple topics in kafka based on input fields

I have to consume json data coming to kafka stream and send to diffrent topics (distinct combination of app id and entity) for further consumption. topic names : app1.entity1 app1.entity2 app2.entity1 app2.entity2 Json Data [ …
amarzeet
  • 63
  • 2
  • 11
1 2 3
…
99
100