Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [kafka]

Kafka is a distributed messaging queue application written in java.

Kafka website: http://kafka.apache.org/

When asking a question about provides relevant information like the kafka version and the version. The design of your cluster should be mentioned with the number of brokers, the topic configuration.

When you have a bug on a particular topic, perform the same test on another topic with different parameters and observe if you can reproduce.

97 questions
2
votes
0 answers

Can connect to kafka, but cannot consume

I have a 3 broker kafka cluster set up in an AWS VPC with kubernetes, I am using the https://github.com/Yolean/kubernetes-kafka repo for this. I'm having the strange issue of being able to connect internally, produce and consume messages fine. When…
Daniel Cull
  • 121
  • 3
2
votes
2 answers

What is ideal directory for kafka messages

The default value of logs.dir=/tmp/kafka-logsin server.properties. Usually /tmp is avoided from keeping any important files and we are storing messages and offsets! Any particular reason why one may not choose /var/log/kafka-logs or…
Divs
  • 121
  • 3
2
votes
2 answers

Determine a Kafka broker's ID from JMX

Given a broker host/ip is there a way to interrogate it to learn the broker ID other than by querying ZK? I'm trying to map the IP to the broker ID for monitoring. ZK is authenticated and I don't have access. I'm preferably looking for a value in…
MrEvil
  • 121
  • 3
2
votes
0 answers

kafka docker - ERROR when sending a message

I would like to implement kafka in a docker container. I use the official zookeper instance for kafka as docker container, too. When I want to write a message about the terminal, on my linux mint 18.1 OS, I get the following error message. foo@bar ~…
Volker Raschek
  • 347
  • 1
  • 5
  • 17
2
votes
2 answers

What do MBean parameters mean in collectd?

I'm using JMX to monitor an Apache Kafka cluster with collectd's GenericJMX plugin. However, I'm not sure what some of the MBean parameters mean. Particularly, I have one metric called kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,…
Matheus Portela
  • 181
  • 1
  • 9
2
votes
1 answer

Wrong measurements being reported by JMX and collectd for Apache Kafka

I'm using JMX to gather metrics from Apache Kafka and send to Librato, a visualization and monitoring service, via collectd. The problem is that some metrics seems to be reporting wrongly. For instance, without anyone using the Kafka cluster, some…
Matheus Portela
  • 181
  • 1
  • 9
2
votes
2 answers

What is the correct way to export environmental values through ansible playbook?

I wrote an ansible playbook which is supposed to install a tool called kafkacat. Most of the tasks in the playbook work, all but one (the most important one). I'm compiling the tool from source and I've successfully installed it manually using the…
Itai Ganot
  • 10,644
  • 29
  • 93
  • 146
2
votes
1 answer

Page cache in Kafka

We are receiving alerts for kafka memory utilization in our kubernetes cluster and analysing it seems that most of the memory is page cache. I am aware that kafka uses loads of page cache for IO performance and reads/writes are cached before…
Putharekulu
  • 31
  • 1
2
votes
1 answer

No connection to Kafka from Faust client

I am having a hard time connection to a machine running Kafka from a client running a Faust script.The script looks like this: import faust import logging from asyncio import sleep class Test(faust.Record): msg: str app = faust.App('myapp',…
ElToro1966
  • 177
  • 2
  • 8
1
vote
0 answers

Getting error INFO [SocketServer brokerId=3] Failed authentication with /10.2.***.* (SSL handshake failed) (org.apache.kafka.common.network.Selector)

I have kafka multibroker deployed on kubernetes, with interbroker SSL communication, I have set the folowing…
namrata
  • 113
  • 5
1
vote
1 answer

When to use Apache Kafka log roll jitter?

There are two log roll jitter related parameters in config doc, i.e., log.roll.jitter.ms/log.roll.jitter.hours. Both of them are marked as high importance, but I have no idea when to use it. What's the scenario to set log roll jitter?
petertc
  • 2,500
  • 1
  • 15
  • 10
1
vote
0 answers

Replace value of key in Apache Kafka

I want kafka to only store the latest value of a given key in a topic. This seems so be possible, as shown here. However, the only think I archived so far, are that ALL Key/Value pairs get deleted or nothing at all. I replace a given Key/Value pair…
SirYea
  • 11
  • 1
1
vote
3 answers

kafka scripts in "bin" directory fail with "Address already in use" (running in kubernetes)

I have a kafka set up running in minikube. It's up and running, and I'm able to produce messages into it and consume them back out. However, when I'm logging in to one the brokers (using kubectl exec if relevant) and try to get some information…
Tom Klino
  • 621
  • 1
  • 8
  • 15
1
vote
0 answers

Kafka broker listing empty nodes list *for a while*

For integration testing purposes I'm creating a very simple single-node Kafka deployment: 1x Zookeeper 1x Kafka 1x Kafka client (e.g. AdminClient, creating topics) (All cleanly deployed in fresh Docker containers.) I'm seeing intermittent failures…
gertvdijk
  • 3,504
  • 4
  • 30
  • 46
1
vote
1 answer

How to do data rebalance on kafka if data is stored persistently

I'm new to kafka and preparing use it for production. What strategies can be used for rebalancing data storage if brokers for a topic's current partitions are running out of disk space, if more brokers can be added to the cluster? By a simple…
Zachary
  • 11
  • 1
1
2
3 4 5 6 7