A Kafka .NET client, written by Confluent.
Source: https://github.com/confluentinc/confluent-kafka-dotnet
Questions tagged [confluent-kafka-dotnet]
138 questions
1
vote
1 answer
Kafka: Consume partition with manual batching - Messages are being skipped
I am using Confluent Kafka .NET to create a consumer for a partitioned topic.
As Confluent Kafka .NET does not support consuming in batches, I built a function that consumes messages until the batch size is reached. The idea of this function is to…
Felipe Correa
- 654
- 9
- 23
1
vote
2 answers
How to serialize/deserialize from ksql avro format to c# using confluent platform
I am using KsqlDb a table with the following form:
KSQL-DB Query
create table currency (id integer,name varchar) with (kafka_topic='currency',partitions=1,value_format='avro');
C# model
public class Currency
{
public int Id{get;set;}
public…
Bercovici Adrian
- 8,794
- 17
- 73
- 152
1
vote
1 answer
Kafka message key is null using Confluent.Kafka consumer library
I am using Debezium to perform CDC against my DB to create messages in Kafka. Using tools like kafdrop and OffsetExplorer I am able to see the message key and value. However, from a .NET framework application, using the Confluent.Kafka libraries,…
Kyley Jex
- 53
- 8
1
vote
1 answer
connecting to localhost Kafka pod
I am new to k8s, have installed Kafka to the local cluster using helm install command and it is installed successfully and shown it using
helm list
and using
Kubectl get all -A
Command as running.
I installed Confluent.Kafka nugget package in my C#…
AB1994
- 21
- 6
1
vote
0 answers
Get producer List for topics in Confluent.kafka nuget package
Is it possible to get list of producers that have sent messages to topics? I am using Confluent.Kafka nuget package for C#.
Surbhi Agarwal
- 11
- 1
1
vote
1 answer
Configure Avro with MassTransit and Kafka
How to configure MassTransit to serialize/deserialize using Avro when producing to, and consuming from Confluent Kafka topics? I saw that Avro serializer/deserializer are in the package Confluent.SchemaRegistry.Serdes. Some code examples would be…
Rafał Ryszkowski
- 365
- 2
- 15
1
vote
0 answers
Confluent Kafka consumer consumes messages only after changing groupId
I have a .Net core console application, that uses Confluent.Kafka.
I build a consumer for consuming messages from specific topic.
the app is intended to run a few times every-day, consume the messages on the specified topic and process them.
It took…
tovale
- 23
- 5
1
vote
1 answer
ASP.NET Core WebApplicationFactory CreateClient Booting Server Blocking on GitLab CI When Run In Docker Compose Environment
I have a functional tests project for my ASP.NET Core 3.1 API that uses WebApplicationFactory to create a test server for the API. The tests pass locally and within a local docker-compose environment.
However when run on the GitLab CI server within…
anon_dcs3spp
- 2,342
- 2
- 28
- 62
1
vote
1 answer
strimzi operator 0.20 kafka 'useServiceDnsDomain' has no effect
PROBLEM:
For some reason client pod can only resolve fully qualified fully-qualified DNS names including the cluster service suffix.
That problem is stated in this quesion:
AKS, WIndows Node, dns does not resolve service until fully qualified name…
Savin
- 131
- 5
1
vote
1 answer
How do I create a Kafka topic on the fly / on startup for the producer to send to?
I'm starting to use the Confluent .NET library for Kafka and am attempting to implement a pattern I used to use against Azure Service Bus, to create the topic upon startup of the producer application (create if not exists). How would this be done…
Kyle J V
- 563
- 5
- 21
1
vote
0 answers
Blazor Server - How to notify user sessions of each kafka message consumed and updated web page?
I am new to Kafka and am using Kafka Dotnet Confluent, considering the following architecture within ASP.NET Core Blazor Server web application
Due to the expense associated with a consumer, I have read that it is recommended to have a…
dcs3spp
- 493
- 1
- 9
- 23
0
votes
0 answers
Azure Eventhub Consume Message throttling in metrics
Problem Statement
Single consumer is consuming the message from the single topic partition. The consuming topic partition has 8578 messages The observation from Azure Eventhub Metrics, Outgoing message is high and seeing the Throttling Request.
This…
Vishnu Kumar K S D
- 21
- 4
0
votes
0 answers
Kafka producer request.timeout.ms does not appear to be working
I have a Kafka producer with a 250 ms request.timeout.ms. All other configs are set to their default.
var producerConfig = new ProducerConfig
{
Acks = Acks.All,
RequestTimeoutMs = requestTimeoutMs
…
Imran
- 385
- 1
- 2
- 8
0
votes
0 answers
Azure Function KafkaTrigger (.NET) consuming messages with Avro array schema as root
I have an Azure Function KafkaTrigger (using azure functions kafka extension: https://github.com/Azure/azure-functions-kafka-extension) which is set to consume messages from Kafka with Avro schema that looks like this:
{
"type": "array",
…
michn
- 226
- 3
- 11
0
votes
1 answer
Apache Kafka time difference between when a message is consumed and produced
We have a topic with 6 partitions. Traffic is not excessive by Kafka terms. We are using a 3-broker cluster in AWS MSK. We are using the Confluent Kafka package for .NET.
When we consume the topic, I take the difference between the produce…
Matthew Allen
- 538
- 2
- 7
- 14