0

Some people don't recommend to use kafka to implement request/response pattern in micro-service world.

but I am not yet clear why it is not recommended ? At the same time, I see spring kafka is supported this pattern out of the box while .net doesn't have this feature available out of the box via any libraries (neither via confluent kafka nor MassTransit). If it is not recommended via kafka do we have any other standard methodology to achieve this scenario via .net core. Can someone provide a reasonable technical answer for me to convince my-self?

UPDATE

I am editing my question to explain the business use case Let's assume I need to display my account balances. I make a request [http request] from front end to api controller, Then there will be three events will be triggered to get savings account balance/ Fixed account balance / current account balances.

These events will be posted to the three kafka topics it will then be listened by three consumers. These consumers process the requests separately and publish to another three topics. Initial http call will be waited till these topics are get consumed. Once it's consumed http call respond to the front end.

    [HttpPost]
    [Route("ResponseWhenAny")]
    public async Task<string> AggregateResponseWhenAny([FromBody] string value, string correlationId)
    {
        string resultOne = "No-Message";
        string resultTwo = "No-Message";

        correlationId=(string.IsNullOrEmpty(correlationId)) ? Guid.NewGuid().ToString() : correlationId;
        var resultOneTask = RequetOne(value, correlationId);
        var resultTwoTask = RequetSecond(value, correlationId);

        var requestResponseTasks = new List<Task> { resultOneTask, resultTwoTask };


        while (requestResponseTasks.Count > 0)
        {
            Task finishedTask = await Task.WhenAny(requestResponseTasks);
            if (finishedTask == resultOneTask && resultOne == "No-Message")
            {
                Console.WriteLine("First response Received....");
                resultOne = await resultOneTask;


            }
            else if (finishedTask == resultTwoTask && resultTwo == "No-Message")
            {
                Console.WriteLine("Second response Received....");
                resultTwo = await resultTwoTask;
            }

            requestResponseTasks.Remove(finishedTask);
        }

        string newResponse = resultOne + resultTwo;
        return newResponse;
    }

resultOneTask and resultTwoTask should be waited till response topics are consumed. If resultOneTask and resultTwoTask are http calls there is no issue of implementing it. Even RMQ is also supported for this use case.But I am not able to find any standard implementation using confluent kafka or MassTransit for kafka .

Thanks In advance

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
KDS
  • 99
  • 1
  • 16
  • 1
    If you want request/response you don't need Kafka. HTTP and hence ASP.NET Core is already synchronous request/response. So it TcpListener or gRPC. People use Kafka to get *asynchronous messaging*. – Panagiotis Kanavos Sep 27 '21 at 16:20
  • 1
    What do you mean by `request/response` in the first place? Why do you want to implement this on top of Kafka? – Panagiotis Kanavos Sep 27 '21 at 16:21
  • Request /response pattern is simply User makes a request and wait for a response. I know this can be achieved by simple REST calls. But let's assume we have more than 50 micro-services , then REST will add some pressure on your micro-services , Instead of giving that pressure to your micro services I want to give it to Kafka. I am already using kafka for Async messages , I want to use Kafka for Sync messages as well using request / Response pattern , But I am not able to understand why it is not recommended and why .net core libraries doesn't have that feature out of the box – KDS Sep 27 '21 at 16:37
  • There's not really a way to make a Kafka consumer synchronous with HTTP calls in any language. And synchonous Kafka producers just ack; there is no other response. – OneCricketeer Sep 27 '21 at 16:55
  • 3
    @KDS you misunderstand what request/response is, or what Kafka does. Even what .NET does. Of course .NET has this out of the box. It had it since 2002. All TCP requests are request/response. WebForms was request/response, so was SOAP and later WCF. There was even messaging support (in a way) through MSMQ. What you call `request/response` over Kafka though would require someone to send a "request" message then *artificially* block waiting for another message that would be considered a response. You can do that over *any* messaging system. People don't do it because it's wasteful – Panagiotis Kanavos Sep 27 '21 at 17:02
  • You got my point , Even it is wasteful let's assume we have to implement it for the moment. Can we do it using any of standard library with Kafka . I know MassTransit is not supported for this use case. I hardly managed by it using Confluent Kafka, But it is custom implementation. Can you help me out to implement this use case using confluent kafka. Do you have any reference implementation ? – KDS Sep 27 '21 at 18:25
  • Adding more information - I am sharing java implementation using spring kafka , I would need same implementation using Confluent kafka library (.net). I had custom implementation , But it is not too sure in high available environment "https://dzone.com/articles/synchronous-kafka-using-spring-request-reply-1" , You can refer this git hub issue which does not have a reply , "https://github.com/confluentinc/confluent-kafka-dotnet/issues/1403" – KDS Sep 27 '21 at 18:37
  • As mhowlett mentioned in those Github comments, any such patterns need built **externally** from the Kafka client code itself, similar to how Spring has done. Overall, seems like gRPC is a better solution for what you want between microservices – OneCricketeer Sep 27 '21 at 18:57
  • Yes , I am also looking for a higher level library, "we may make a higher level library that provides this out of the box". But I haven't fount this kind of library so far. That is why I posted a question here. – KDS Sep 27 '21 at 19:52
  • Hey @KDS, Did you ever find a solution? I am also looking at adding request-response to an architecture with lots of real time data streaming using .NET 6. Cheers – Matt Sep 02 '22 at 10:54
  • 1
    Hey @Matt - I was looking for out of the box solution , But i haven't found any in C#. Therefor i gave up. As per my readings implementing request response pattern using Kafka is not a good idea. – KDS Sep 26 '22 at 12:44

1 Answers1

1

Generally a message queue and/or event streaming platform is not needed to implement request/response, and only serves to complicate the architecture.

Exposing and calling an http endpoint (often called WebAPI in .Net) is a much simpler solution.

Phil Sandler
  • 27,544
  • 21
  • 86
  • 147
  • I know this can be achieved by using simple REST calls. But let's assume we have more than 50 micro-services , then REST will add some pressure on your micro-services , Instead of giving that pressure to your micro services I want to give it to Kafka. I am already using kafka for Async messages , I want to use Kafka for Sync messages as well using request / Response pattern , But I am not able to understand why it is not recommended and why .net core libraries doesn't have that feature out of the box . – KDS Sep 27 '21 at 16:42
  • 1
    My answer explains why it's not recommended--it's much more complicated to try to do request/response using a queue/stream. It's just not designed for that purpose. Also, when you say "why .net core libraries doesn't have [request/response] out of the box" you're going to have to explain what you have in mind. You can do http out of the box, which is perfectly suited to request/response. – Phil Sandler Sep 27 '21 at 16:48
  • I updated my question with more details. Out of the box means there is no standard library to support this feature embeded to their library [MassTransit,Confluent Kafka]. – KDS Sep 27 '21 at 18:20
  • Adding more information - I am sharing java implementation using spring kafka , I would need same implementation using Confluent kafka library (.net). I had custom implementation , But it is not too sure in high available environment "https://dzone.com/articles/synchronous-kafka-using-spring-request-reply-1" , You can refer this git hub issue which does not have a reply , "https://github.com/confluentinc/confluent-kafka-dotnet/issues/1403" – KDS Sep 27 '21 at 18:37