0

I am trying to come up with a solution that involves Kafka with .Net

This video gives a nice introduction and also suggests configuring for Throughput or Latency.

The related code here has configuration values like so:

config.Add("batch.num.messages", "50000")
config.Add("queue.buffering.max.ms", "300")

There's also the mention of using

Collections.Concurrent.BlockingCollection

in the video.

What I currently have is this:

  1. A Web API call writes data into a table (MS-SQL) which has a column named PublishedStatus whose value is set to 1 (ReadyForPublishing). The data is also added to a List.
  2. Apart from doing the above the API uses two classes one deriving from BackgroundService and the other from IHostedService

2.1 The one using BackgroundService takes the list from Step 1 and tries to Publish them to Kafka

2.2 The one using IHostedService is for resiliency and will query the table from Step 1 for all the records in certain time range which could not be published due to some failure and tries to publish them again.

  1. Records are marked as Published or ErrorPublishing based on the response of the ProduceAsync call.

After watching the video I was looking for examples that could show the usage of BlockingCollection because the target is Throughput here.

Question 1: Can someone please point to a resource that I can get started with using BlockingCollection? If BlockingCollection is not recommended what are my other options? When we use the batch option, what is queue or collection that this library uses in which I can add the data instead of to a list in Step1 above?

Question 2: Can a framework like Polly be helpful here when designing for resiliency?

I have an option here to redesign/refactor the existing implementation as described above, hence these queries.

Appreciate any help/guidance.

Peter Csala
  • 17,736
  • 16
  • 35
  • 75
Codehelp
  • 4,157
  • 9
  • 59
  • 96
  • Have you considered KafkaFlow: https://github.com/Farfetch/kafka-flow There's even an extension library that gives you features like Polly (https://github.com/Farfetch/kafka-flow-retry-extensions). – Gui Ferreira Aug 16 '22 at 10:11
  • Could you please share with us the above mentioned two classes' source code? – Peter Csala Aug 26 '22 at 14:06
  • This seems like the OutBox pattern (At least half of it). There is a library that is called [DotnetCore.Cap](https://github.com/dotnetcore/CAP) which abstracts all those things away from you. – Eldar Aug 26 '22 at 14:06

0 Answers0