Stack Overflow
Stack Overflow

Questions tagged [apache-kafka-streams]

Related to Apache Kafka's built-in stream processing engine called Kafka Streams, which is a Java library for building distributed stream processing apps using Apache Kafka.

Kafka Streams is a Java library for building fault-tolerant distributed stream processing applications using streams of data records from topics in Apache Kafka.

Kafka Streams is a library for building streaming applications, specifically applications that transform input Kafka topics into output Kafka topics (or calls to external services, or updates to databases, or whatever). It lets you do this with concise code in a way that is distributed and fault-tolerant.

Documentation: https://kafka.apache.org/documentation/streams/

3924 questions
1
vote
1 answer

Join/group multiple topics in Kafka based on identical key

I have multiple topics produces by a external application that share the same keys and they share this model: Topic1: Produces messages using the unix ms timestamp as key. Topic2: Reads from Topic1 and produce messages with the same key Topic3:…
Felix
  • 138
  • 2
  • 11
1
vote
1 answer

How to classifier topics/partitions in Kafka?

For example user can subscribe on specific categories of films. When a new films apperas in Kafka I must to send information about that to consumers who subscribes on category of film. How to classifier this? Using partigion or topics? Because…
user13419533
1
vote
0 answers

Java Kafka Stream - Avro Serializer Error

I am sending avro data to sink topic using kafka streams in java using generic avro serde. error stack: org.apache.kafka.common.errors.SerializationException: Error serializing Avro message caused by java.lang.IllegalStateException: Too many schema…
Amey
  • 11
  • 2
1
vote
0 answers

Serialize/Deserialize generic types in Spring Cloud Kafka Streams

The main purpose is to read a stream from a topic, apply some transformations and then send two events to other topics. For that we are using Kstream.branch() function and using functional style programming. The code is: Input…
Nuno Ferreira
  • 23
  • 6
1
vote
0 answers

What is causing 'no committed offset for partition' followed by an IllegalStateException

I have a recently upgraded Kafka Streams 2.4.1 application and we have started seeing the application periodically crash due to the following error: 2020-05-13T11:27:08.553Z INFO <> [chat-98346af0-1be5-41e9-b15f-3cba364efa35-StreamThread-2]…
James Hay
  • 12,580
  • 8
  • 44
  • 67
1
vote
1 answer

Kafka Stream not Working : Group Coordinator Keep getting discovered and unavailable/invalid

Recently our (Kafka 1.1.1) brokers went down and our Kafka-stream application stopped working. So we stopped the application manually to stop the alerts. After the Kafka came live again, We started our stream application but it didn't read any…
Himanshu Tomar
  • 13
  • 5
1
vote
1 answer

Kafka Stream delivery semantic for a simple forwarder

I got a stateless Kafka Stream that consumes from a topic and publishes into a different queue (Cloud PubSub) within a forEach. The topology does not end on producing into a new Kafka topic. How do I know which delivery semantic I can guarantee?…
user2274307
  • 124
  • 3
  • 16
1
vote
1 answer

How do I scale Kafka Streams app running on kubernetes

I have a kafka streams App which runs on a pod in Kube cluster. Trying to find a way for Kube to scale up/down the pods based on kafka topic lag. Has anyone walked the path before and willing to share some details ?.
Bala.vrad
  • 53
  • 9
1
vote
1 answer

Kafka Stream fixed window not grouping by key

I get a single Kafka Stream. How can I accumulate messages for a specific time window irrespective of the key? My use case is to write a file every 10 minutes out of a stream not considering the key.
user2274307
  • 124
  • 3
  • 16
1
vote
1 answer

Standby tasks not writing updates to .checkpoint files

I have a Kafka Streams application that is configured to have 1 standby replica created for each task. I have two instances of the application running. When the application starts the application writes .checkpoint files for each of the partitions…
James Hay
  • 12,580
  • 8
  • 44
  • 67
1
vote
2 answers

Behaviour of kafka stream application when offset commit isn't being done either by application (and auto commit off)

I want to know what behaviour of kafka streams would be when (using low level API) auto commit is turned off and application don't do explicit commit? In case application is restarted (auto commit off, and application itself don't do explicit commit…
CuriousMind
  • 8,301
  • 22
  • 65
  • 134
1
vote
1 answer

Kafka Streams - Extracting Timestamp for List of Objects per record

What I want to achieve is to get a count of each message present in the record based on the timestamp present in the message. Each record consists of List object. I would like to extract the timestamp for each metric and aggregate the metric…
Surya
  • 45
  • 5
1
vote
1 answer

Kafka Streams - override default addSink implementation / custom producer

It is my first post to this here and I am not sure if this was covered here before, but here goes: I have a Kafka Streams application, using Processor API, following the topology below: 1. Consume data from an input topic (processor.addSource()) 2.…
Guilherme Alcântara
  • 135
  • 1
  • 8
1
vote
1 answer

KStream Join with GlobalKTable over non-key values

I am trying to join KStream with GlobalKTable, the join is not completely on keys. GlobalKTable employeesDetails = builder.globalTable("EMPLOYEE_TOPIC",..); KStream empIdOverLoginUserId = builder.stream("LOG_TOPIC",…
Venkata Madhu
  • 93
  • 1
  • 14
1
vote
2 answers

Kafka stream groupBy based on timestamp

I use kafka for voting app, where user can choose candidate and change the selection during 1 hour time-range. Since this is suitable for KTable, I use kafka stream app. However, there is time-range requirement, means I need to groupBy().count()…
Timothy
  • 855
  • 1
  • 13
  • 29
1 2 3
…
99 100