Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [kafka]

Kafka is a distributed messaging queue application written in java.

Kafka website: http://kafka.apache.org/

When asking a question about provides relevant information like the kafka version and the version. The design of your cluster should be mentioned with the number of brokers, the topic configuration.

When you have a bug on a particular topic, perform the same test on another topic with different parameters and observe if you can reproduce.

97 questions
0
votes
1 answer

what is the safe and best way to delete the kafka topic folders

on all our kafka machines ( production machines ) , we see that: ( no free space ) df -h /var/kafka Filesystem Size Used Avail Use% Mounted on /dev/sdb 11T 11T 2.3M 100% /var/kafka and under /var/kafka/kafka-logs we see all topic…
jango
  • 59
  • 2
  • 3
  • 12
0
votes
1 answer

Rsyslog. How to count messages sent to kafka

I am using rsyslog for sending data from nginx to kafka. Pipeline looks like following: nginx -> unix_socket -> rsyslog -> omkafka module -> kafka Omkafka module for rsyslog uses tcp for transferring messages, so I can easily count bytes of…
Samriang
  • 163
  • 1
  • 11
0
votes
0 answers

production kafka clusters + How to optimize Page Cache

one of the importance things in Kafka production cluster is the page cache here is a good explains what is page cache PageCache is a typical read/write cache. The operating system uses the free physical memory to cache files. This cache is called…
King David
  • 549
  • 6
  • 20
0
votes
0 answers

Kafka Issue : Request joining group due to: group is already rebalancing

The pod is not getting restarted, No ongoing rebalance to finish consumer group experiences no frequent changes, but still rebalancing issue occurs. is there anything wrong with the configuration? This is the Kafka…
Spartan
  • 101
  • 2
0
votes
0 answers

Expose MSK serverless cluster to internet

I followed the https://docs.aws.amazon.com/msk/latest/developerguide/public-access.html guide, but the Properties → Network settings → Edit public access option doesn't appear at all. Could it be that it only applies to provisioned clusters? Or I…
Aleksei Zyrianov
  • 101
  • 3
0
votes
0 answers

Very old segments in Kafka

I have set my segment retention via (log.retention.ms) for topic to 2h. However I still have a lot of old messages in the message queue and some of the segment data files (data/topic-0/00000000000000658568.log) are older than two months. I was…
Konrads
  • 870
  • 2
  • 20
  • 40
0
votes
0 answers

Vertical Scaling resources (CPU, RAM) for Kafka Production cluster

Currently we are using three node kafka Cluster in Prod with hardware specifications : Zookeeper : CPU 2 Core Ram 4GB Kafka : CPU 4 Core Ram 8GB We want to vertically scale our Kafka cluster to double the resources we currently have. Could anyone…
Arvinder Pal
  • 1
0
votes
0 answers

Can Apache MirrorMaker2 run on only one server?

I have a couple of test clusters (3 nodes in each) and I’m trying to configure them such that one runs in one country and another in another country (hypothetically). I have read that remote consume and local produce is the pattern to follow. With…
RaiseError
  • 1
0
votes
0 answers

Kafka Connect cluster showing erratic behaviour

We have a running KafkaConnect cluster (Strimzi distribution), deployed in an Openshift (Kubernetes for the matter) cluster that is showing an erratic behaviour. The REST API of Kafka Connect is randomly slow, very slow for some endpoints, even…
jmoreno
  • 101
  • 1
0
votes
1 answer

Rsyslog server - conditionally forward logs

The idea is to receive the logs on the Rsyslog server and then send them to a specific Kafka server depending on the log contents. For example, Logs are being received on a single port, 514 Forwarding received logs to Kafka topic #1 and lines…
Iggy
  • 15
  • 1
  • 4
0
votes
1 answer

Handle 150 req/s and 2TB of logs for 7 days of retention available by API

I would like to get advise from experienced people to build a HA infrastructure to log 2To of data in JSON format every week. I need to have a retention time of 7 days and need to be able to requests these data by API. The global requirements are…
michbr010
  • 1
0
votes
0 answers

How to get Kafka consumer metrics with JMX exporter for Prometheus?

Kafka version: 3.1 I didn't find a way to get Kafka consumer metrics as described in Kafka documentation from my Kafka servers. Even if I leave the "rules" section empty in the config of the JMX exporter there are no "kafka_consumer_*" metrics…
André Veelken
  • 5
  • 2
0
votes
0 answers

Ansible playbook to post message into kafka topic

Playbook 1: --- - name: Message into topic hosts: web1 become: yes tasks: - name: post message expect: shell: "/usr/local/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testTopic" responses: …
Dev_Sri
  • 1
  • 2
0
votes
0 answers

Kafka: monitoring length of queues / topics

We have several producers and several consumers connected via Kafka - basically, it's batch processing jobs that are created on demand, placed on Kafka in several queues, and the batch processors pick them up via Kafka and process them one by one. I…
Florin Andrei
  • 1,208
  • 1
  • 12
  • 18
0
votes
0 answers

How to remove escape character when using kafka console consumer?

For compliance purposes, I have to save all kafka raw documents and keep them one year. To do this, I use the following : kafka-console-consumer.sh --bootstrap-server kafka1:9092,kafka2:9092,kafka3:9092 --topic test1 --consumer.config…
Atreiide
  • 1
1 2 3 4 5
6
7