We have an application that handles various pipelines modeling real-world workflows, each pipeline being composed of multiple different consumers. For example, in Pipeline 1, we might have Consumer A -> Topic 1 -> Consumer B -> Topic 2 -> Consumer C. In Pipeline 2, we might have Consumer A -> Topic 1 -> Consumer B -> Topic 5 -> Consumer D.
We are trying to limit consumers to one external dependency and re-use them in different pipelines/workflows.
Right now, most of our consumers call a service that handles the business logic, and that business logic will often times determine what is the correct next topic to produce to based on something in the message. This determination is inferred from some information, rather than explicitly set on the message.
The coupling of the business logic to the pipeline structure is something I would like to improve. What are some recommendations for how to "compose" a pipeline more flexibly but still reuse consumers across multiple pipelines? One idea I had was adding a "plan" to the message that contained an ordered list of topics this message needed to go to, it could be generated at the start of the pipeline and then each consumer could produce to the right topic without involving the business logic.
We are using Apache Kafka in python to make our pipelines.
