Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [kafka]

Kafka is a distributed messaging queue application written in java.

Kafka website: http://kafka.apache.org/

When asking a question about provides relevant information like the kafka version and the version. The design of your cluster should be mentioned with the number of brokers, the topic configuration.

When you have a bug on a particular topic, perform the same test on another topic with different parameters and observe if you can reproduce.

97 questions
1
vote
1 answer

Kafka long coordinator load time and small ISRs

I'm using Kafka 0.8.2.1, running a topic with 200 partitions and RF=3, with log retention set to about 1GB. An unknown event caused the cluster to enter the "coordinator load" or "group load" state. A few signals made this apparent: the…
Emmett Butler
  • 111
  • 3
1
vote
1 answer

How to perform bulk topic deletion in kafka

We have 3 node Kafka cluster with replication factor 3 and default log retention period of 4 days. We have enabled the auto topic creation and It will create topics for each of our tasks. Once the task is finished we don't need the topic created for…
Sachin PK
  • 83
  • 1
  • 2
  • 6
1
vote
0 answers

What is the attribute you need to specify for kafka Log start offset metric?

I have an environment where logstash (v6.2.3) is connecting kafka_2.11 cluster (v.1.1.0) using kafka input. What I want to achieve is fetch consumer offset information using kafka-consumer-group.sh . However, I end up with below…
Yu Watanabe
  • 616
  • 2
  • 8
  • 18
1
vote
1 answer

kafka + how to revert topic deletion

just in case we delete the wrong topic as /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper hdpmaster01:2181 --delete --topic gtom.poli.pri.proc Topic gtom.poli.pri.procis already marked for deletion and now we want to revert it what…
jango
  • 59
  • 2
  • 3
  • 12
1
vote
0 answers

Operation APPEND failed with HTTP500?

package org.apache.spark.examples.kafkaToflink; import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.OutputStream; import java.io.PrintStream; import java.nio.charset.StandardCharsets; import java.util.Properties; import…
Anubhav
  • 11
  • 2
1
vote
1 answer

Cannot use supervisor with Apache Kafka

I have an Ubuntu 16.04 machine where Apache Kafka is installed. Currently, I can make it work flawlessly by using a start_kafka.sh script with the following contents: JMX_PORT=17264 KAFKA_HEAP_OPTS="-Xms1024M -Xmx3072M"…
Matheus Portela
  • 181
  • 1
  • 9
1
vote
0 answers

Kafka module missing on syslog-ng.

I'm trying to install syslog-ng and its kafka module. I'm using docker debian:latest for this if that matters. After running 'apt-get install syslog-ng' the syslog-ng can't reach the kafka module even though they are both installed as I can see in…
kub1x
  • 111
  • 5
1
vote
2 answers

Rsyslog. Disk-assisted queue for omkafka

I have following pipeline: nginx -> unix_socket -> rsyslog -> omkafka module -> kafka For omkafka I am using following config: module( load="impstats" interval="10" # how often to generate stats resetCounters="on" # to get…
Samriang
  • 163
  • 1
  • 11
1
vote
1 answer

Securing Kafka REST API endpoints

We plan to use Kafka for publishing updates to our customers. The plan is to use the dedicated topic for each client. To makes things simpler for them we want to utilize REST API. So that user1 will be able to get updates from /api/topic1 and won't…
the_joric
  • 113
  • 4
1
vote
1 answer

Kafka acl without kerberos

I'm trying to set up a single kafka server with certificate authentication and access controls based on those certificates without the bother of setting up a kerberos service. Setting upp the certificate authentication is easy. But I enable the acl…
Elvar
  • 325
  • 2
  • 7
1
vote
0 answers

Kafka not stopping consumers when deleting group

I am sending logs from Logstash as the producer to Kafka and then consuming them from Graylog. This works as expected. However, when stopping the Graylog input, thus sending a LeaveGroup request, the consumer is not unregistered and even when the…
dmuensterer
  • 143
  • 1
  • 6
1
vote
1 answer

Display metric for multiple servers on a single grafana panel, using a Snowflake datasource populated from Kafka

I have a collection of servers which submit metrics to a single kafka topic. This data is stored as a row in a snowflake database in this format: { "disk_util": 47.7, "location": "location1", "timestamp": "1683703169378" } I'm using this…
doublespaces
  • 121
  • 3
1
vote
1 answer

How to forward port to connect multiple MSK Kafka brokers through bastion?

I have MSK Kafka cluster on AWS with 3 brokers: b1.amazonaws.com:9092, b2.amazonaws.com:9092, b3.amazonaws.com:9092. Brokers are not public, so I have opened 3 tunnels through bastion: ssh -L 9092:b1.amazonaws.com:9092 -L 9093:b2.amazonaws.com:9092…
michaldo
  • 111
  • 4
1
vote
1 answer

kafka cluster RAID vs JBOD with new hardware and kafka versions

we want to design production kafka cluster with 7 kafka machines , on RHEL OS kafka version should be - 2.7 usually we are installing the kafka machine/s disks as RAID 10 ( as confluent recommended ) but as maybe some of you know Kafka disks, can…
King David
  • 549
  • 6
  • 20
1
vote
1 answer

Kafka - Get Ip address or hostnames connected to the cluster

I have a test Kafka Cluster in AWS MSK with three brokers. I'd like to know how to get information about who is connecting to the cluster either to produce or consume messages. For instance, MSSQL Server logs successful connections: Login succeeded…
Luis de Haro
  • 133
  • 6
1 2
3
4 5 6 7