Stack Overflow
Stack Overflow

Questions tagged [confluent-kafka-dotnet]

A Kafka .NET client, written by Confluent.

Source: https://github.com/confluentinc/confluent-kafka-dotnet

138 questions
0
votes
0 answers

Why is there always an error: Local value serialization error in kafka?

I have a problem with Kafka "Local: Value Serialization error". var producerConfig = new ProducerConfig { BootstrapServers = bootStrapService, MessageTimeoutMs = 5000 }; producerConfig.SecurityProtocol…
Zora-Luo
  • 1
0
votes
0 answers

Why offset duplicates in Confluent.Kafka at Consumer

I'm using Apache Kafka as a Message Processor and using Confluent.Kafka in Asp.Net Core as a consumer. I have to use a couple of consumers to process all the messages from the topic. My code works perfectly when I don't have any exceptions. 2…
Phil
  • 82
  • 8
0
votes
0 answers

How to get the number of unread messages in a topic in kafka

how can I get the number of unread messages on a message topic in the confluent kafka dotnet?
Gülsen Keskin
  • 657
  • 7
  • 23
0
votes
0 answers

confluent kafka dotnet consumer retry

If any message in kafka fails to be read, I want to save these messages in a new topic and retry the reading process a certain number of times. Meanwhile, I need to pause the reading process at the original location. How can I do that?
Gülsen Keskin
  • 657
  • 7
  • 23
0
votes
3 answers

How can I write messages in producer at once and read 1 message per minute with consumer?

How can I write messages in producer at once and read 1 message per minute with consumer? config properties I can use note : please "max.poll.records" note that I cannot use the method My consumer class: var settings =…
Gülsen Keskin
  • 657
  • 7
  • 23
0
votes
0 answers

Confluent Kafka with .net standrad 2.1 +

I wonder how is it possible to integrate the ConfluentKafka NuGet in a .net standard 2.1 proj? In the description at nuget.org, it seems that it can run when projects' target framework is up to .net standard 2.0 / .net core 2.1. Practically I've…
Itay Barber
  • 63
  • 2
  • 9
0
votes
1 answer

Consume only a specific number of messages from the queue

Is it possible to consume only a specific number of messages like 10 or 50 or 100 messages from the 1000 that are in the queue? I was looking at 'fetch.max.bytes' config, but it seems like it is for a message size rather than number of messages. I…
Gülsen Keskin
  • 657
  • 7
  • 23
0
votes
1 answer

kafka confluent .NET 1.8.2 How to override max.poll.interval.ms in the consumer config?

My application has to consume messages from a kafka topic at specific time or when triggered. So when my consumer sits there without consuming messages, it gets removed from the consumer group. Application maximum poll interval(300000ms) exceeded by…
Bel Vedere
  • 3
  • 4
0
votes
1 answer

Using .NET Core to consume Protobuf serialized log messages in Kafka that were produced by OTEL agent

I'm using the OpenTelemetry collector agent to send logs to a Kafka streaming layer. I want to consume these log messages in a .NET Kafka consumer (for now), but I'm having a few issues. OTEL appears to use Protobuf serialization which is making it…
Ryan.Bartsch
  • 3,698
  • 1
  • 26
  • 52
0
votes
0 answers

Multithreaded Kafka Consumer in .NetCore

I want to implement multi-threaded Kafka consumer in .NetCore which is given in the image below. In java, we can retrieve multiple records using Poll method and can get records of each partition. I want to implement the same thing in .net. Java…
esdoodle
  • 247
  • 2
  • 21
0
votes
1 answer

Kafka .NET Client custom timestamp

I have been working with the Kafka .NET Client, everything is working but right now the timestamp is automatically set and what I need is to be able to send my own timestamp. var kp = new kafkaDatapoint { value = p.Value, tags =…
GhzNcl
  • 149
  • 1
  • 4
  • 13
0
votes
0 answers

How to enhance the speed of consumer cosume message in kafka use confluent-kafka-dotnet?

Test Result Picture When I use confluent.kafka-net, and I want to ensure the order of kafka message in .netcore, I create a mschannel, _msChannel = Channel.CreateUnbounded(); when listening, I got a consume result and a received message, and if…
Boi Fox
  • 45
  • 5
0
votes
1 answer

Topic partition on basis of partition Key

I am trying to split a stream into multiple partition on the basis of a partition key but apparently its not working. The implementation is such that I have a Class lets say Metrices public class Metrice { public string MetriceType { get; set;…
user_19240589
  • 97
  • 1
  • 14
0
votes
1 answer

why Implementing request/response pattern is not recommended with Kafka

Some people don't recommend to use kafka to implement request/response pattern in micro-service world. but I am not yet clear why it is not recommended ? At the same time, I see spring kafka is supported this pattern out of the box while .net…
KDS
  • 99
  • 1
  • 16
0
votes
1 answer

Implement Request Response pattern using Kafka + .Net Core + MassTransit

Can someone tell me how to implement request response pattern using kafka with .net core (2.1,2. or 3.1). I prefer to implement this pattern using MassTransit which is light weight message bus. I was not able to find reference implementation among…
KDS
  • 99
  • 1
  • 16
1 2 3
…
9 10