Stack Overflow
Stack Overflow

Questions tagged [kafka-topic]

202 questions
1
vote
1 answer

Unable to read the data from kafka topics using spark streaming

I am trying to read the data from the kafka topic using the spark streaming. I am able to produce the message into the kafka topic, but whilereading the data from topic using spark streaming i am getting error message as given below: ERROR…
Pramod
  • 113
  • 2
  • 14
1
vote
1 answer

Topic1 to Topic2 using kafka streams

I am new to kafka streams and I would like to read a topic and write a part of it in a new topic using kafka streams api. My key is string and value is Avro Is there a documentation/example that I can use ? Edit : final StreamsBuilder builder =…
KnowledgeSeeker
  • 77
  • 1
  • 12
1
vote
0 answers

how to get partition info and offset for Kafka topic without knowing consumer group info

I am totally a squat in Kafka land If I run the command /cfintools/confluent-4.0.0/bin/kafka-avro-console-consumer --topic $t --bootstrap-server $bt --consumer.config /cfintools/avro_consumer_ssl_cfin_prod.properties --property…
user1874594
  • 2,277
  • 1
  • 25
  • 49
1
vote
1 answer

kafka - ERROR Error when sending message to topic test-topic with key: null, value: 17 bytes with error

I am working on deploying Kafka/Zookeeper in Kubernetes using MINIKUBE. below is my YAML file: ################################## # Setup Zookeeper Deployment ################################## --- apiVersion: extensions/v1beta1 kind:…
Vignesh
  • 814
  • 7
  • 29
1
vote
2 answers

Failed to list kafka topics in kubernetes after adding jmx__javaagent

I have a running kafka pod in my kubernetes cluster. For getting custom metrics in prometheus format, I have configured a jmx_prometheus_javaagent and assagined the port 2255. And I am able to list the metrics from localhost:2255/metrics. The issue…
manu thankachan
  • 433
  • 3
  • 9
  • 19
1
vote
0 answers

How to take Kafka Topic Backup and restore?

I need to take backup all the topics in Kafka to the file named in respective topic names and need to restore the topic as per user requirement. Note: This script needs to be run in the Kerberized environment. kafkabackup.sh monyear=`date | awk…
Krishnaraj V
  • 23
  • 6
1
vote
1 answer

Kafka not getting rid of data when setting retention.ms

So when I look for a way to count the messages in a topic, this one is good kafka-run-class kafka.tools.GetOffsetShell --broker-list broker1:9092,broker2:9092,broker3:9092 --topic rev-dly-upd --time -1 The only thing is, when I change the…
uh_big_mike_boi
  • 3,350
  • 4
  • 33
  • 64
1
vote
1 answer

Kafka ISR Shrinked and doesn't Expand

I have a Kafka Cluster of 3 nodes with 1 ZK and 1 Broker each. Kafka version 0.10.1.1. Each Topic has replication-factor=3 and min.insync.replicas=2. We have checked by issuing --describe thereafter, that the Topic(s) were successfully created with…
Divs
  • 1,578
  • 2
  • 24
  • 51
1
vote
0 answers

Kafka-Manager showing Inactive Consumer Groups and incorrect Lag

I have a Kafka cluster of 3 separate physical nodes, with each hosting 1 zk and 1 broker each. We are using Kafka 0.10.1.1. The Topics are created on ZK ensemble and the brokers connect to the ZK ensemble. My Producer and Consumer is a Spring Boot…
Divs
  • 1,578
  • 2
  • 24
  • 51
1
vote
2 answers

Kafka Consumer distribution not working as expected

I have Three topics each having three partitions on a cluster of kafka. now, there are total 9 partitions. and when i create 9 consumers... the 6 are being idle. only three consumers are being used. the expectation is: each consumer should pickup…
Radhi
  • 6,289
  • 15
  • 47
  • 68
0
votes
0 answers

Extracting values from TopicPartition - confluent kafka

Trying to extract the tuple values from TopicPartition like topic, partition, offset etc. Could not locate a helper method. Added code snippet below from confluent_kafka import DeserializingConsumer conf = { 'bootstrap.servers':…
madmatrix
  • 205
  • 1
  • 4
  • 12
0
votes
1 answer

Kafka consumer behaviour with multiple topics

For some reason, I have to use one consumer for two topics. Now the questions I have are: What is the reasoning behind we don't have a way to specify which topic we want to poll? Will a single pull/consume return messages from different…
dragon66
  • 2,645
  • 2
  • 21
  • 43
0
votes
1 answer

Do each Kafka Producer/Consumer occupy a one partition in topic?

I have a question of Kafka topic. I made system like below. And I made a Kafka topic (1 partition) using below command. ./kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic temp.t And then, I…
JinHyup Kim
  • 11
  • 1
0
votes
1 answer

Dependencies in archetype-post-generation.groovy script cannot be resolved

I have some problem when creating a maven project from a maven custom archetype. I have the expected directory structure for a maven-archetype, including a post-generation groovy script, which I want to create some Kafka topics in a broker…
estarko
  • 11
  • 2
0
votes
1 answer

Kakfa createTopics is getting failed

I am new to Kafka and I am trying to create a topic using the command line and it is giving me the following error: C:\kafka\bin\windows>kafka-topics.bat --create --topic tutorialspedia --bootstrap-server localhost:9092 Error while executing…
Harsh Shah
  • 2,162
  • 2
  • 19
  • 39
1 2 3
…
13 14