0

I'm having an architecture of microservices where each service's producer write to the same topic. I have two instance of kafkaRestproxy each listen to that topic but the problem here is that : Suppose a request come to instance-1 of a restproxy and it will redirect to the microservice and that service done with the job and write the response to the topic but the response is consumed by the second instance of the restproxy let say instance-2.

What should I do to solve this? Is their any kind of application_id we can attach to the request so when that microservice done with the job and if another instance of restproxy consumed that response then we can redirect the response to that instance of restproxy which gets that request?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Madhukar Choudhary
  • 15
  • 4
  • You can use the Kafka partition key, I am assuming both the consumers belong to the same consumer group, In this case you can attach a partition key to the message so the messages with the same partition key goes to the same consumer. – Pharaoh Mar 01 '20 at 13:19
  • Please refer to this documentation: https://kafka.apache.org/intro#intro_producers – Pharaoh Mar 01 '20 at 13:26

1 Answers1

1

Your proxies form a Kafka Consumer group, just as any other application. When you request records, you give both the consumer group and the consumer instance name (such as a host of the HTTP client) GET /consumers/(string:group_name)/instances/(string:instance)/records

You should generally not try to strictly control which consumers get which information beyond assigning a unique instance to each request, to allow for parallel consumption (assuming this is what you want).

Also, the rest proxy isn't consuming anything unless you have another application that's requesting that information, e.g. the GET request above.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245