Stack Overflow
Stack Overflow

Questions tagged [confluent-kafka-dotnet]

A Kafka .NET client, written by Confluent.

Source: https://github.com/confluentinc/confluent-kafka-dotnet

138 questions
1
vote
0 answers

Kafka Transaction Scope in C#

Trying to persist some data in MongoDB and publish a couple of messages to Kafka everything inside of a transaction. I was thinking of using the dotnet TransactionScope but I do not see how to apply it to the Confluent Kafka packages. var…
101000101
  • 23
  • 3
1
vote
0 answers

Kafka Transaction INVALID_PRODUCR_EPOCH Error while Commit / Abort Transaction

I am facing the INVALID_PRODUCER_EPOCH error while committing or aborting the transaction. Am running the my application in Windows 10 machine. I have a single Producer and TransactionId is Random Guid, enabled Idempotence in producer as well. No…
Vishnu Kumar K S D
  • 21
  • 4
1
vote
1 answer

Kafka How to not move offset until message processing is succeeded

We are reading messages from a Kafka-topic. I was under the (false) impression that by setting the EnableAutoOffsetStore/enable.auto.offset.store = false you, as a consumer could choose when you wanted to move the offset. We were using this like…
Cowborg
  • 2,645
  • 3
  • 34
  • 51
1
vote
1 answer

How to use truststore.jks & keystore.jks files to build a producer application in .NET using Confluent Kafka

I have received truststore.jks & keystore.jks file from a team who manages kafka. We are required to build a producer application in .NET using these .jks files. I have used below commands to convert .jks files to required format as .jks is not…
Dhruv Sanan
  • 36
  • 4
1
vote
1 answer

Kafka Dotnet Builders best practices

I am using dotnet to build a Kafka application and I found that this NuGet package from Confluent. The documentation is okay, but I was wondering the following: I have a controller, and I will receive multiple requests, where I will use a producer…
Vivere
  • 1,919
  • 4
  • 16
  • 35
1
vote
1 answer

Get event/message on Kafka when new file on S3

Im quite new to AWS and also new to Kafka (using Confluent platform and .NET) . We will receive large files (~1-40+Mb) to our S3-bucket and the consuming side of this should process these files. We will have all our messaging over Kafka. Ive read…
Cowborg
  • 2,645
  • 3
  • 34
  • 51
1
vote
1 answer

Can't send message to kafka topic use Confluent.Kafka C#

We have 3 different environments: test, cert and prod. These environments have topics configured using the offset explorer. The problem is that I can send messages to cert and test, but I can't send to prod until the topic in prod is marked for…
Wafeelka
  • 41
  • 1
  • 6
1
vote
0 answers

Connect to AWS MSK Kafka cluster from azure service, Private key check failed. no certificate assigned

I am trying to connect to an AWS MSK cluster from an Azure Service. We are using confluent kafka dotnet libraries in the code and also mTLS for the connection as below. var consumerConfig = new ConsumerConfig { …
Priyan Perera
  • 560
  • 1
  • 7
  • 20
1
vote
0 answers

How can I find large Kafka messages?

The problem My app, reading message from several topics by mask, crashes with error: Confluent.Kafka.ConsumeException: Broker: Message size too large What I've tried The first problem is that I followed the guidelines for handling large messages:…
Makrushin Evgenii
  • 953
  • 2
  • 9
  • 20
1
vote
0 answers

Need help Confluent Kafka .net Core client trying to produce/consume message to and from Kafka cluster(Linux server) using SASL Kerberos authenticatio

I am trying to work on the Kafka connectivity issue. Basically I am trying be produce/consume from .net core confluent kafka to kafka cluster( linux server) and I am facing various issues. I am using Confluent.Kafka(.net) 1.7.0 version to connect to…
DKaul
  • 11
  • 2
1
vote
1 answer

How can I access the list of uncommitted messages in a topic in kafka?

How can I access the list of all uncommitted messages in a topic in kafka
Gülsen Keskin
  • 657
  • 7
  • 23
1
vote
0 answers

Producer terminating with 1 message (39bytes) still in queue or transit. Kafka

How do I solve this terminating with kafka? I have the following error: https://i.stack.imgur.com/bY2j3.png Code: public class KafkaHelper { public static async Task SendMessage(KafkaSettings settings, string topic, string key, string…
Arizal Pratama Tanu Andara
  • 13
  • 4
1
vote
1 answer

How to create a single producer?

I have a message list coming from API, I need to map the elements in this list, send them to the producer and read them one by one. Multiple users can send message lists at the same time. I need to map the lists in order and not read the other list…
Gülsen Keskin
  • 657
  • 7
  • 23
1
vote
1 answer

Custom Connector for Apache Kafka

I am looking to write a custom connector for Apache Kafka to connect to SQL database to get CDC data. I would like to write a custom connector so I can connect to multiple databases using one connector because all the marketplace connectors only…
Learn AspNet
  • 1,192
  • 3
  • 34
  • 74
1
vote
0 answers

Dotnet Kafka Consumer pause and resume on schedule/on-demand

We have a Dotnet BackgroundService(Microsoft.Extensions.Hosting) running from a WebApi project. There is a Kafka Consumer(fairly new to Kafka world) running from within the ExecuteAsync method. We would like our Kafka consumer to stop/start…
AJ_NY
  • 59
  • 1
  • 8
1 2
3
…
9 10