Stack Overflow
Stack Overflow

Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
0
votes
1 answer

Visualise data from kafka consumer

I'm new to kafka. I have written a simple producer script which writes JSON data in the form of latlong coordinates and another consumer app that can read the latlong data as it is being produced. If I wish to plot the latlong data using D3js or…
DhiwaTdG
  • 748
  • 1
  • 10
  • 26
0
votes
2 answers

Can Kafka producer do consumer lag aware partitioning

We a Kafka producer produces messages to a topic that is consumed by only one group of consumers. The speed of consumers is quite different. We always have a certain amount of consumer lags as a buffer. Is it possible for the producer to know the…
zwush
  • 75
  • 2
  • 9
0
votes
1 answer

Kafka compression fail in comsumer side

I have enabled snappy compression on the producer side with the following change: props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG,"snappy"); Now if any change is required on consumer side for this? I checked the Consumer.config but did not find…
Sahil Arora
  • 31
  • 6
0
votes
2 answers

ProducerStream producing only to single partition

I am trying to produce some messages to a single topic having 2 partitions. All the messages are going to partition number 2 only. I would expect that a producer stream would distribute the messages across all partitions. const kafka =…
Dan W
  • 138
  • 10
0
votes
1 answer

Kafka Client compatibility

we are using the following Kafka client on our Linux red-hat servers 0.8.1 Release Released March 12, 2014 Release Notes Source download: kafka-0.8.1-src.tgz (asc, md5) Binary downloads: Scala 2.8.0 - kafka_2.8.0-0.8.1.tgz (asc, md5) Scala 2.8.2 -…
jessica
  • 2,426
  • 24
  • 66
0
votes
1 answer

Can I view the id/version number of the schema that is being sent to Kafka? for debugging purpose

I want to view the Id/version of the schema to be referred in a record while publishing data from producer. If I have 3 versions of schemas and I'm publishing data for the version 1. Can I see the version number before sending the record to kafka?
Subham Saraf
  • 421
  • 2
  • 4
  • 13
0
votes
1 answer

Kafka streaming - TimeoutException: Expiring * record(s) for TOPIC:* ms has passed since batch creation

Streaming application is rolled out in production and right after 10 days observing errors/warnings in the CustomProductionExceptionHandler for expired transactions which belongs to older day window. FLOW : INPUT TOPIC --> STREAMING…
Swapnil
  • 45
  • 1
  • 1
  • 9
0
votes
2 answers

kafka create costumer in pooling with two tomcat balancing

I have two tomcat with a webapp. I run kafka and zookeeper service with docker, and i run tomcats. In kafka console i see that are created 2 consumer at second with this message: kafka_1 | [2019-12-20 16:30:20,725] INFO [GroupCoordinator…
Marco Di Falco
  • 59
  • 8
0
votes
1 answer

Not able to produce message into kafka topic using aws lambda function

So i am trying to write a lambda function on S3 event which will put message into kafka topic . My aws lambda function is triggering and not getting any error also . But i am not able to see those messages into Kafka topic . Here is my lambda…
Atharv Thakur
  • 671
  • 3
  • 21
  • 39
0
votes
0 answers

Persist state of Kafka Producer within Spring Clod/Boot

I want to implement a Kafka Producer with Spring that observes a Cloud Storage and emits meta informations about newly arrived files. Until now we did that with a Kafka Connector but for some reasons we now have to do this with a simple Kafka…
maimArt
  • 389
  • 1
  • 11
0
votes
0 answers

Pushing data to a kafka server with Logstash

i have a logstash conf file that looks like this: input { jdbc { jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver" jdbc_connection_string => "jdbc:oracle:thin:@//the-s2.db.oracle.yn:1521/DPP2.mind.com" …
Haalanam
  • 89
  • 9
0
votes
1 answer

ruby-kafka: is it possible to publish to two kafka instances at the same time

Current flow of the project that I'm working on involves pushing to a local kafka using ruby-kafka gem. Now the need arose to add producer for the remote kafka, and duplicate also messages there. And I'm looking for a better way, than calling…
Elmor
  • 4,775
  • 6
  • 38
  • 70
0
votes
0 answers

Creating Reactive Kafka SenderRecord from Flux

I am doing a prototype integrating Spring Cloud Gateway with Reactive Kafka. My goal is to push the post body to kafka. I have access to Flux which could potentially give me the body in chunks. Need expert advice how to convert this Flux of…
anaray
  • 309
  • 1
  • 8
0
votes
1 answer

How to measure the number of messages produced by kafka producer per second?

I am learning Kafka for my university project, I would like to know the logging technique to monitor number of messages produced per second by Kafka producer. Thanks for helping.
jay patel
  • 23
  • 5
0
votes
0 answers

What is the node-id tag in kafka producer MetricName for and how can I get it from KafkaProducer instance?

I've have kafka producer running same code from 3 different linux servers sending messages to same kafka cluster topic. Below is the sample code - KafkaProducer producer = new KafkaProducer(props); Map
user51
  • 8,843
  • 21
  • 79
  • 158
1 2 3
…
99 100