0

I want to be able to test kafka-consumer flows after a canary deployments. Assume we have service which consumes a kafka topic. I have 3 instances in production and 3 partitions on kafka.

At T0 they have the V1 of the application.

  • Server #1 (v1)
  • Server #2 (v1)
  • Server #3 (v1)

I start a canary deploy, and now I have the new version on Server #1

  • Server #1 (v2)
  • Server #2 (v1)
  • Server #3 (v1)

With some routing trick, I can ensure that my API tests are hitting the new instances. But, I couldn't find a feasible way to test kafka-consumer flows for Server #1. There is no guarantee that the test message I emit, will be consumed by Server #1.

I am looking for testing strategies, but I couldn't find a good way without changing the kafka topology, or application logic. I wonder what is your thoughts on this problem.

ygk
  • 550
  • 1
  • 7
  • 17

1 Answers1

1

Kafka doesn't really have such an approach since v2 will likely maintain the same consumer group id, and there's no guarantee it'll consume the same partition as before after a rebalance

You can use embedded Kafka or test containers to do integration tests with Kafka clients, or use builtin MockConsumer. Don't depend on external infrastructure to run your tests

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • thanks @oneCricketeer For unit or component testing, using test containers/mocking makes sense. But in my case, it is important to verify that core flows are working before rolling out to other production instances. Testing with mocks doesn't give the same confidence. – ygk Jul 24 '23 at 13:08
  • If you use external infrastructure, you'll need to use unique consumer groups, or disable offset commits. Neither will help maintain a canary deployment strategy. Plus, if the Kafka server is having maintenence while running tests, you'd be blocked – OneCricketeer Jul 24 '23 at 13:27