I'm implementing CDC for a PostgreSQL azure database. And I want the events to be sent to azure event hub. My current plan is to use Debezium Server with the Event Hub sink to do this. However I want to enforce order of events by table. From this article I know I can do this by having a single topic with multiple partitions but only sending events from a single table to a specific partition every time.
However it seems like debezium doesn't provide a nice way to handle this. You can specify the partition key for all events to be sent to, but not dynamically per event. The only other thing I saw that could solve this is a custom sink implementation or a custom EventHubProducerClient implementation passed into the config.
What are my options for handling this? Is there another way to architect this solution so that I don't have to use partition keys? Or is a custom sink implementation going to be my best bet? Or should i just drop debezium and write a custom listener/publisher?
Context / requirements
- typically to run debezium you need a kafka instance running, however if possible I don't want to use kafka as I'm already planning on using event hub, and it seems duplicitous, and it is another service that needs to be maintained.
- FIFO ordering of events by table when read by consumers of the event hub
- all logical database changes are turned into events
- no java developers on the team so the custom (java) implementations will be a stretch to our expertise.
