Questions tagged [kafka-producer-api]

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

Use for questions related to the Apache Kafka producer API. Any question related to producing into Kafka topics. Producer failures and recover, idempotency, and transactional API.

1916 questions
6
votes
2 answers

Connect to Kafka inside VirtualBox

I'm having troubles connecting from my Host (Windows) to Guest (Linux) where I installed Kafka. I have set up a VM (with VirtualBox) where I installed Confluent tools. In this VM, I run the command: confluent start schema-registry It starts…
6
votes
2 answers

Kafka Streams: one record to multiple records

Given: I have two topics in Kafka let's say topic A and topic B. The Kafka Stream reads a record from topic A, processes it and produces multiple records (let's say recordA and recordB) corresponding to the consumed record. Now, the question is how…
6
votes
3 answers

For AvroProducer to Kafka, where are avro schema for "key" and "value"?

From the AvroProducer example in the confluent-kafka-python repo, it appears that the key/value schema are loaded from files. That is, from this code: from confluent_kafka import avro from confluent_kafka.avro import AvroProducer value_schema =…
Kode Charlie
  • 1,297
  • 16
  • 32
6
votes
2 answers

error: package org.apache.kafka.clients.producer does not

I am new to kafka. When I run this command javac -cp "C:\kafka\kafka_2.11-0.10.2.0\libs\kafka-clients-0.10.2.0.jar" *.java I get an error message error: package org.apache.kafka.clients.producer does not exist
6
votes
2 answers

Kafka command line producer/consumer have 1 second latency

I'm doing a test run with Kafka using the command line producer and consumer. I'm running this in one Terminal window bin/kafka-console-producer.sh --broker-list localhost:9092 --topic tag7 and this in another bin/kafka-console-consumer.sh…
6
votes
1 answer

Kafka Producer RecordTooLargeException

I am getting the following error org.apache.kafka.common.errors.RecordTooLargeException: The message is 196773 bytes when serialized which is larger than the total memory buffer you have configured with the buffer.memory configuration" But the…
Sankar
  • 101
  • 1
  • 2
  • 6
6
votes
4 answers

How to guarantee order in Kafka partition

Ok so I understand that you only get order guarantee per partition. Just random thought/question. Assuming that the partition strategy is correct and the messages are grouped correctly to the proper partition (or even say we are using 1 partition) I…
user432024
  • 4,392
  • 8
  • 49
  • 85
6
votes
1 answer

Bidirectional messaging system using kafka

Is there any possible to develop a bi-directional messaging system using apache kafka ? I need to subscribe for a topic from my consumer as well as I need to send message from my consumer.
Prasath
  • 1,233
  • 2
  • 16
  • 38
6
votes
4 answers

kafka java producer stuck in producing message

I am implementing apache kafka producer using java api. Apache Kafka is installed on localhost. Zookeeper is also running but still producer.send() function stuck in sending message and message is not published. I have already created…
6
votes
1 answer

How to put data from MySQL to Kafka producer?

I am a fresher in Big Data. I have database in MySQL and I don't know how to import this to Apache Kafka for streaming process. And then get data from consumer. Please give me advice.
6
votes
4 answers

Difference in kafka 0.8 with kafka 0.10

Can anybody list out basic differentiating features in Kafka 0.10 as compared to kafka 0.8. We are thinking of upgrading our kafka-8 to kafka-10. What challenges can one face in upgrading? Thanks in advance.
Nilotpal
  • 3,237
  • 4
  • 34
  • 56
6
votes
1 answer

Kafka producer is not able to update metadata after some time

I have an kafka environment which has 3 brokers and 1 zookeeper. I had pushed around >20K message in my topic. Apache Storm is computing the data in topic which is added by producer. After few hours passed, While I am trying to produce messages to…
Anirudh Kaki
  • 83
  • 1
  • 7
6
votes
1 answer

Producing and Consuming Avro messages from Kafka without Confluent components

I am trying to find an example where I can produce and subscribe avro messages from kafka. At this point of time, I want to use a "vanilla" kafka deployment without any confluent add-ons. is this possible? All examples I found so far very quickly…
Knows Not Much
  • 30,395
  • 60
  • 197
  • 373
6
votes
2 answers

Explain replication-offset-checkpoint AND recovery-point-offset in Kafka

Can Some explain what these files means, present inside kafka broker logs. root@a2md23297l:/tmp/kafka-logs-1# cat recovery-point-offset-checkpoint 0 5 my-topic 0 0 kafkatopic_R2P1_1 0 0 my-topic 1 0 kafkatopic_R2P1 0 0 test 0 0 …
Nilotpal
  • 3,237
  • 4
  • 34
  • 56
6
votes
7 answers

Kafka producer to read data files

I am trying to load a data file in loop(to check stats) instead of standard input in Kafka. After downloading Kafka, I performed the following steps: Started zookeeper: bin/zookeeper-server-start.sh config/zookeeper.properties Started…
Sumit
  • 61
  • 1
  • 1
  • 3