Stack Overflow
Stack Overflow

Questions tagged [kcat]

kcat is a single-binary Apache Kafka CLI tool

kcat is a single-binary Apache Kafka CLI tool for doing Unix cat-like operations against topics, such as reading (consuming), writing (producing), as well as Kafka related network troubleshooting.

https://github.com/edenhill/kcat

49 questions
2
votes
1 answer

kafkacat's -o (offset to start consuming from) option

Will execute the following kafkacat command with the -o (offset to start consuming from) option but without the -G (group id) option affect other consumer groups? kafkacat -C -b 10.52.1.1:9092,10.52.1.2:9092,10.52.1.3:9092 -t MyTopic -o beginning
ytw
  • 1,335
  • 2
  • 20
  • 42
2
votes
4 answers

Kafka deployment on minikube

Hi i am new to kubernets, i am using minikube single node cluster for local development and testing. Host: Ubuntu 16.04 LTS. Minikube: Virtual box running minikube cluster My requirement is i need to deploy kafka and zookeeper on minikube and…
user1184777
  • 987
  • 2
  • 19
  • 38
1
vote
1 answer

Kafka partition leader -1 ONLY over SSL

I'm working on moving all our Kafka traffic over SSL. We have two clusters in each region. Using Kafka version 2.7.0. All regions and all clusters work fine over SSL except one cluster. Among other tools, I use kafkacat to probe the cluster. When…
Capt. Crunch
  • 4,490
  • 6
  • 32
  • 38
1
vote
2 answers

Problem with ADVERTISED_LISTENER on macos

I start kafka with this docker-compose.yml on my Mac: mydb: [...] environment: kafka_bootstrap_servers: kafka:9093 kafka: image: wurstmeister/kafka:2.13-2.6.0 ports: - "9092:9092" environment: …
Frank
  • 129
  • 1
  • 6
1
vote
1 answer

Configuration of a specific topic. Kafkacat

I have a topic "topic-one" and I want to know if it has "log.cleanup.policy = compact" configured or not. Is it possible with kafkacat, extract the properties and / or configuration of a specific topic?
Jose
  • 1,779
  • 4
  • 26
  • 50
1
vote
2 answers

How to install kafkacat on kafka container resulted from docker-compose

My context is I am trying to create a docker-compose which will start few containers for running ELK+FileBeats and 3 Kafka containers. The whole docker compose and its yml configuration files can be found in GitHub docker-compose To summarize, this…
Jim C
  • 3,957
  • 25
  • 85
  • 162
1
vote
1 answer

trying to connect to heroku kafka from local computer using kafkacat: ssl.ca.location failed: No error

I am trying to follow these instructions for connecting locally to my heroku kafka using kafkacat: https://gist.github.com/crcastle/cb21c2148fc57ad89753bf28b561d910 I am creating an env file like this: heroku config -s > .env and then running this…
Alex028502
  • 3,486
  • 2
  • 23
  • 50
1
vote
1 answer

Performance issues running kafacat over slow speed link

I have weird performance issues with fetch.max.message.bytes parameter in librdkafka consumer implementation (version 0.11). I run some tests using kafkacat over slow speed network link (4 Mbps) and received following results: 1024 bytes =…
meaclum
  • 96
  • 5
0
votes
1 answer

kcat get latest message from a topic with a specified key

I am doing integration tests for Kafka and I produce a message with a key integration-test-sub-key and I want to read it later and compare result to what is expected. I tried doing $ kcat -b localhost:9092 -t topic-name -C -o -1 -c 1 -k…
comonadd
  • 1,822
  • 1
  • 13
  • 23
0
votes
1 answer

How to use kafkacat -f option?

I am trying to consume Kafka message using below options [format token as requirement] kafkacat -C -b localhost:9092 -t test-topic -p 0 -f 'Topic %t [%p] at offset %o: key %k: %s\n' -o -1  -e | jq . But getting below error, Error: file/topic list…
Ayan rahate
  • 21
  • 4
0
votes
3 answers

kcat fials to connect to broker

I've been trying to consume messages from broker using kcat, however for some reasons kcat tries to connect to localhost:9092 instead of the broker I've supplied to it using the -b switch. I've been through all the official documentation and several…
Syn
  • 73
  • 1
  • 9
0
votes
1 answer

How can I see which partitions in the apache Kafka broker?

Is it possible in Apache kafka through the kafkacat command to see which partition is located in broker? for example: broker 0 partition 1,2 broker 1 partition 0,1,2 Many thanks I use the .\bin\win\kafkacat -L -b hostname:port command it shows…
Bigsol
  • 1
  • 2
0
votes
2 answers

kcat protobuf deserialization

I'm using kcat to check the content of kafka topics when working locally but, when messages are serialized with protobuf, the result I get is an unreadable stream of encoded characters. I'm aware of the existence of some other kafka-consumers tools…
frblazquez
  • 115
  • 1
  • 10
0
votes
1 answer

Docker compose behaviour affected by directory structure and user

Here I have a network of Docker containers: Docker-compose.yml: version: "2" services: zookeeper: image: zookeeper container_name: zookeeper environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 broker: …
Jack Rowntree
  • 193
  • 1
  • 5
0
votes
1 answer

Kafkacat consume between timestamp giving wrong results when counting records

I want to count the number of messages in a given Kafka topic between two timestamps. I tried doing this using kafkacat, using the following command: # START_DATE = 01.04.2022 02:00:00Z # END_DATE = 01.04.2022 02:05:00Z $ kafkacat -C -b broker:9092…
filpa
  • 3,651
  • 8
  • 52
  • 91
1
2
3 4