Workflow engine with events

Question

I'm researching a workflow engine to organize some system flows.

As part of the research I want to build a small workflow poc:

The workflow should listen to some event and than start (lets say customer A created)

The flow is:

1. Publish a work message for customer A
2. Receive a work message response for customer A
3. Publish a second work message for customer A
4. Receive a second work message response for customer A
5. End workflow

Assuming multiple workflows exist at the same time, How do I make sure each response goes to the correct workflow?

I thought of a few solutions (none I think are good enough)

1. Create dynamic queues in Rabbit per workflow and as part of the message add the response queue
2. Work with Kafka and read every message discarding none relevant messages.
3. Responses would be posted to the api of the workflow engine.

I would love to hear better suggestions.

Answer 1

That's an interesting question, but it's quite generic and the right way of analysing a potential solution is by considering a number of aspects:

• What's the scale at which you want to operate?
• What's frequency of creating the workflows? The messages are something like user profile updates, or general user interactions with your website, for example? If the workflow fires once a day, you would need a completely different architecture compared to if the workflow fires every second.
• You want the system to be multi-tenant? If that's the case, you need to consider some isolation strategies, so that one customer's workflows doesn't affect the ones of other customers.

I would assume that, as in any architectural decision, you aim for a good balance between scalability and costs, and I believe that considering the above points will help you to navigate through tradeoffs.

For cost-efficiency, I believe it would help if you check a cloud-based queueing service, like Amazon SQS, because you only pay for the data-transfer, and not for the queue itself.

Also you can check Amazon Lambda for serverless computing, which again is cost-efective for event processing, since you don't pay for a VM that is always up and running.

Answer 2

Option 3.

The standard approach would be to create a workflow instance per customer. Then when sending an event send it to a particular workflow instance by its id.

Note that in most workflow engines you don't need to deal with messages directly in the case you described. The workflow code invokes activities that perform actual work. Activities are invoked asynchronously and can take as long as needed.

I would recommend checking the temporal.io that can be used to implement your use case trivially. Disclaimer: I'm the lead of the project.