Stack Overflow
Stack Overflow

Questions tagged [kafka-topic]

202 questions
1
vote
1 answer

Springboot with AWS SNS

Actually Iam trying to set email subscription to SNS topic in AWS through springboot and publish the message to SNS topic. But facing issue in configuring the AWS access key and secret key. Do I need to create a separate IAM user for accessing SNS…
Saipriyadarshini Bandi
  • 291
  • 2
  • 8
1
vote
0 answers

Kafka RETENTION.BYTES | Manual acknowledgement

In case CLEANUP policy is COMPACT,DELETE and our acknowledgement is MANUAL_IMMEDIATE, and we reach the retention.bytes -> while we were in middle of processing a message, once the logs are cleaned by cleanup policy, would the message be repicked? or…
Shreya Garg
  • 51
  • 7
1
vote
0 answers

Kafka Cluster : Zookeeper nodes show different topic configuration information

I have a kafka cluster with 5 zookeeper nodes and 6 broker nodes. Kafka version: 2.4.0 Zookeeper version: 3.5.6 I recently changed the partition count on a topic from 12 to 36. Now that I check my zookeeper nodes, the nodes show different topic…
Ishrat Jahan
  • 136
  • 5
1
vote
1 answer

Kafka Consumers subscribed to different topics in same consumer group

I am starting with kafka , have a question on the consumer groups. We have an application where we want different consumers from same group subscribing to different topics. The grouping is done based on some business criteria . To be specific…
user2254608
  • 33
  • 1
  • 6
1
vote
0 answers

Kafka Connect MQTT: is it possible to have a 1:1 map between MQTT and Kafka topics?

I've been working with MQTT for a while, but I'm a newbie in Kafka. I'm dealing with a scenario in which N MQTT clients are publishing messages on M topics, where N and M can freely range from half a dozen to several hundred. They publish their…
DaleCooper
  • 11
  • 2
1
vote
0 answers

How to programmatically create a topic in Apache Kafka using C++ client librdkafka

So far I haven't seen a c++ client librdkafka that implements the creation of a topic explicitly without using the configuration option to create automatically the topics.
lakshmiravali rimmalapudi
  • 355
  • 6
  • 19
1
vote
1 answer

Kafka topic returns 0 messages

I have a real-time process that streams events occuring in MongoDB directly to Snowflake using Confluent Platform. Confluent is established on my local machine as a docker image and I see no issues: Name Command …
marcin2x4
  • 1,321
  • 2
  • 18
  • 44
1
vote
2 answers

Get the partition number of a record before producing in Kafka

I need to know the partition number of Kafka topic to which a Kafka record goes. Before the execution of producer.send(record); Is there any way to know to which partition that record goes?
Harsha Chittepu
  • 115
  • 1
  • 8
1
vote
1 answer

Does Kafka maintain the In-sync replication factor after leader goes down?

Can someone please explain the in-sync replication factor maintainance in Kafka in the below scenario: Scenario: Consider, initially we have replication factor as 3. In Kafka if the leader goes down one of the in-sync replicas becomes the new…
Nilay Narlawar
  • 15
  • 8
1
vote
1 answer

How to check kafka topic's partitions memory usage?

how can I check memory usage on specific topic's partitions in kafka? I tried this one "kafka-log-dirs.sh --describe...", but I'm not sure what output means: "partition": "simpleTopic-8", …
Rapiernik
  • 111
  • 3
  • 11
1
vote
1 answer

What happens to the offsets that are compacted in a kafka partition?

Let us say, I have a partion-0 with 2 log segments. The first segment contains keys v1, v2 and v3 at offsets 0, 1 and 2. The second segment has keys v3, v4, v1 at offsets 3, 4 and 5. When the log compaction happens, the keys at offset 0 and 2 in…
swetha
  • 91
  • 2
  • 11
1
vote
1 answer

Does Kafka consumer reads the message from active segment in the partition?

Let us say I have a partition (partition-0) with 4 segments that are committed and are eligible for compaction. So all these segments will not have any duplicate data since the compaction is done on all the 4 segments. Now, there is an active…
swetha
  • 91
  • 2
  • 11
1
vote
1 answer

Does Kafka chose an alternative partition in the same topic if the partition I want to send message to was full of disk?

My topic has 3 partitions (in 3 different brokers). I want to send message to particular partition 1 (add partition id or specify a key in my message). Assuming the partition 1 became full of disk, does an alternative partition (partition 2 or 3 in…
Hieu Doan
  • 115
  • 1
  • 7
1
vote
1 answer

How uber-go/zap logger logs to a Kafka topic?

Can I get an example to publish the logs to kafka topic. I am currently using uber-go/zap logger and using https://github.com/confluentinc/confluent-kafka-go to publish.
keerthi reddy
  • 49
  • 3
1
vote
3 answers

Kafka internal topic : Where are the internal topics created - source or target broker?

We are doing a stateful operation. Our cluster is managed. Everytime for internal topic creation , we have to ask admin guys to unlock so that internal topics can be created by the kafka stream app. We have control over target cluster not source…
Shweta Gulati
  • 566
  • 1
  • 7
  • 17
1 2 3
…
13 14