Stack Overflow
Stack Overflow

Questions tagged [kafka-topic]

202 questions
0
votes
1 answer

How to align my consumer to consume 2 depending topics

I have this use case that I have 2 topics Topic 1 (Units) -> P0 / Topic 2 (Reservations) -> P0 I have a single consumer that needs to have the up to data from both topics/partitions in order to take the correct decision (either delete a unit if not…
Ahmed Alaa El-Din
  • 1,813
  • 1
  • 16
  • 19
0
votes
1 answer

How do I ask kafka which topics have extraordinary overrides?

I understand my default retention period, but I want to see if any topics have special overrides for the retention period. I've tried using the kafka-topics, but it doesn't connect with zookeeper. kafka-topics --zookeeper localhost:2181 --describe…
Micah Rothenbuhler
  • 66
  • 4
0
votes
1 answer

source kafka connect not publish with timestamp mode

I try to create a source Kafka connect with timestamp mode in Windows. I have this table: CREATE TABLE "usu"."mytable" ( "first_name" CHAR(8 BYTE) DEFAULT ' ', "last_name" CHAR(8 BYTE) DEFAULT ' ', "regist" TIMESTAMP (0) DEFAULT…
Lanre
  • 101
  • 2
  • 8
0
votes
1 answer

Kafka: Topic vs Partition data

Going through Kafka documentation and various other resources, I understand that messages in Kafka are organized into topics. Also, a topic can be broken down into partitions and each partition can be hosted on a different server. This gives…
Mandroid
  • 6,200
  • 12
  • 64
  • 134
0
votes
1 answer

Delete messages and reset topic offset to 0 irrespective of consumer group

I need to create a Kafka setup where at the start of each day, all messages in a topic must get deleted and its offset reset to 0. I have tried to do this by setting log.retention to 1 day but it didn't seem to work. I also ran…
SRMara
  • 11
  • 5
0
votes
1 answer

How to mount volume for kafka through docker-compose file?

I am using "wurstmeister/kafka" docker image with latest tag. Whenever I tried to stop & start the kafka container, it will start container with default configuration. How can I mount volume, so that data persists even when container stops or…
Bhavik Shah
  • 31
  • 1
  • 6
0
votes
0 answers

NestedError: topic/partition change check failed

Suddenly I got an issue in running process of apache kafka consumer group. Anyone please share your thoughts on why this issue happens. NestedError: topic/partition change check failed at _checkTopicPartitionChange…
Bhavik Shah
  • 31
  • 1
  • 6
0
votes
2 answers

High Scalability Question: How to sync data across multiple microservices

I have the following use cases: Assume you have two micro-services one AccountManagement and ActivityReporting that processes event U. When a user registers, event U containing the user information will published into a broker for the two…
F.O.O
  • 4,730
  • 4
  • 24
  • 34
0
votes
1 answer

io.confluent.ksql.exception.KafkaTopicExistsException: when launching ksql-server-start ksql-server.properties

I'm working with ksql from quite some time. Kafka cluster if of 3 nodes. I've been using udf as well and all looks good until I stop the servers and start them again. On server start I'm seeing the following in the logs: [2019-04-03 11:29:54,381]…
srikanth
  • 958
  • 16
  • 37
0
votes
1 answer

Specify default partitions and replication factor for regex based topics in kafka

Background: I have set default number of partitions=1 and replication factor=3 in my kafka server.properties file and have enabled topic creation. Kafka cluster is running with both authentication and authorization. There are topic for which I have…
xabhi
  • 798
  • 1
  • 13
  • 30
0
votes
2 answers

Clear kafka topic programmatically using C#

I need to clear or delete Kafka topics programmatically using C# language. Currently, I have used Confluent.Kafka library for publishing and consuming Kafka topics. I can delete Kafka topics using the command line like this kafka-topics.bat…
Abdul Manaf
  • 4,933
  • 8
  • 51
  • 95
0
votes
1 answer

How can I push multiple messages to a topic in a single asynchronous call

I am using Google Cloud Platform. I need to push messages from different source to a topic, but it should be done using a single asynchronous call.
Ashish Pandey
  • 1
0
votes
1 answer

Apache kafka Error while appending records to topic

I am trying to consume 10 millions rows csv file of size(600MB) via connect api. The connection starts consuption completes for 3.7million records .After that i am getting the below error. [2018-11-01 07:28:49,889] ERROR Error while appending…
souradeep majumdar
  • 137
  • 1
  • 1
  • 17
0
votes
0 answers

Unable to update metadata after ____ ms | Java application unable to get metadata from kafka

I was trying a producer that I created in Java to send a message to a topic on Kafka broker. I am able to send messages to the topic via console i.e. by using kafka-console-producer.sh. But when I am trying to do the same with a producer that I…
Utkarsh Upendra
  • 55
  • 5
0
votes
4 answers

Microservice input and output domain models

I am using Kafka to decouple my services, but I'm having some seconds thoughts on the way services consume and produce inputs and outputs. If I have a service A, which takes data from some external service out of my control, I am forced to adapt to…
ioreskovic
  • 5,531
  • 5
  • 39
  • 70
1 2 3
…
13
14