Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [kafka]

Kafka is a distributed messaging queue application written in java.

Kafka website: http://kafka.apache.org/

When asking a question about provides relevant information like the kafka version and the version. The design of your cluster should be mentioned with the number of brokers, the topic configuration.

When you have a bug on a particular topic, perform the same test on another topic with different parameters and observe if you can reproduce.

97 questions
1
vote
1 answer

zookeeper ssl alert "alert bad certificate"

I am using Kafka (version 2.3.0) and Zookeeper (version 3.5.5-3) - the stable version is 3.6.3. When I test the SSL of my Zookeeper using this command: openssl s_client -showcerts -connect 127.0.0.1:2280 -CAfile /certs/ca-chain.cert.pem and I am…
John Mark
  • 13
  • 2
1
vote
0 answers

Kafka installation with Confluent vs Apache Kafka

for now we have HDP 2.6.5 version , with Kafka cluster with Kafka version 0.1 all Kafka machines are installed on RHEL version 7.6 all machines with RHEL 7.6 are on DELL physical machines we want to move to Apache Kafka or Confluent Platform The…
King David
  • 549
  • 6
  • 20
1
vote
2 answers

The Debezium Kafka based KAFKA_ADVERTISED_LISTENERS property setting is ignored

I have a Docker Compose based Swarm with a Zookeeper service that starts fine, but the Kafka service cannot start. It keeps using the default KAFKA_ADVERTISED_LISTENERS property: debezium_kafka.1.5hhrqfp5kqts@stephane-pc | Using…
Stephane
  • 227
  • 3
  • 12
1
vote
2 answers

PUB/SUB Message Queue with server side filtering?

I am looking for a message queue to realize a chat application. The most important requirement is: when a subscriber specifies a filter criteria, all the messages need to be filtered on the server side. Unfortunately I don't find definite answers to…
Purple Alien
  • 11
  • 1
1
vote
0 answers

How can I further analyze ram usage in a possible leak? [debian@HyperV + Kafka + Zookeeper]

We have a 3-box cluster Kafka+Zookeeper, in a virtualized enviroment (in Windows 2019 HyperV). In the machines' config dynamic memory is disabled, and they have 4G RAM each (I know it's a very few amount for this kind of cluster, but the usage is…
Mise
  • 11
  • 2
1
vote
0 answers

Confluent Kafka Connect S3 re-processing data due to offset out of range

I'm experiencing a strange situation. My Confluent Kafka Connect S3 is processing around 8K messages/sec. What started happening randomly every few days is: The Fetcher class would try to fetch an offset (e.g. 158473047). It would get a "Fetch…
Yasitha Bogamuwa
  • 433
  • 1
  • 3
  • 13
0
votes
1 answer

kafka in kubernetes pod

I am new at working with kubernetes setup, and are trying to run kafka pod with a persistence volume, so in case a pod goes down, the memory is not lost, and I can spin up a new cluster, using the persisted memory. The problem with this I tried…
kube
  • 1
  • 1
0
votes
1 answer

mkfs + xfs + what is the right mkfs cli in order to create xfs file-system on huge disk

We need to create xfs file-system on kafka disk The special thing about kafka disk is the disk size kafka disk have 20TB size in our case I not sure about the following mkfs , but I need advice to understand if the following cli , is good enough to…
shalom
  • 461
  • 13
  • 29
0
votes
0 answers

zookeeper server due unable to load database on disk

We have kafka cluster with 3 nodes , each kafka include zookeeper server and schema registry We get the following error on one of the zookeeper server [2019-11-12 07:44:20,719] ERROR Unable to load database on disk…
shalom
  • 461
  • 13
  • 29
0
votes
1 answer

Can logstash be used as a kafka forwarder?

I am looking at the plugins for Logstash and I noticed it had both an input and output plugin for kafka. For a single topic, can Logstash be used to forward kafka topic messages to another kafka server or should I expect there to be some data loss?
Archimedes Trajano
  • 505
  • 2
  • 8
  • 20
0
votes
1 answer

Way to forward remote internal network ip and port to same ip and port on local mac

For development reasons I want to connect my local machine (macos) to a remote private network of a kafka/zookeeper cluster. When I do port forwarding with ssh root@000.000.000.000 -N -L 10.0.0.8:9092:10.0.0.8:9092 my mac is not doing this…
Jurudocs
  • 339
  • 1
  • 3
  • 11
0
votes
1 answer

Creating RabbitMQ cluster (by Bitnami) and Kafka cluster (by Bitnami) in Azure and connecting to them from Kubernetes in another v-net

I'm trying to create an architecture with: 1. Kafka cluster (by Bitnami) 2. Rabbitmq cluster (by Bitnami) 3. Kubernetes cluster (AKS) that can use both Kafka and RMQ When I created both clusters (Kafka and RMQ) in Azure, I was asked to create a…
Leeran Setton
  • 1
0
votes
1 answer

Can I control which terraform instances are discarded when reducing count?

When having a multi-instance provisioning that needs to be reduced - can I control which of the existing resource will be killed? I have a Kafka cluster with variable load, so I would like to provide broker clients a list of currently working…
mjahr
  • 3
  • 4
0
votes
0 answers

Message Forwarder to MapR Streams/Kafka

We have an application that forwards data to MapR Streams (Kafka). Problem is if the network link goes down then data is lost as the application itself doesn't keep a history. We are looking at installing a message bus of some sort on the same VM so…
MikeKulls
  • 336
  • 1
  • 2
  • 16
0
votes
1 answer

Expose Confluent Kafka outside of the cluster but inside of the GCP

I've installed Confluent Kafka OSS using their official helm chart: https://github.com/confluentinc/cp-helm-charts However, I would like to consume Kafka topics in a GCP Dataflow so this mean, I guess, that I need to make the Kafka pods available…
Gabriele B
  • 163
  • 4
1 2 3
4
5 6 7