3

I have to implement a document management service as an orchestration layer that orchestrates between underlying service such as storage , parsing , anti virus scan etc. The requirement is to make the layer flexible so that different flows for different kinds of documents can be implemented quickly
One approach is to model this as a event driven system and implement processing pipeline on events using framework like Apache Flink.
Another way to think about it is - workflow . Design this as a workflow that runs on a workflow engine like Apache Airflow or Uber Cadence.
What will be a better approach.

redzedi
  • 1,957
  • 21
  • 31

2 Answers2

8

Disclaimer: I'm a tech lead of Temporal Workflow and former tech lead of Cadence Workflow. But I know something about Flink and Airflow :).

It would be pretty hard to implement orchestration using Flink. The reason is that Flink is a stream processing solution that is optimized for very fast processing of each request. Orchestration should deal with slow requests or services being down for a long time. Flink doesn't support such scenarios out of the box. It also models processing as a static graph while orchestration is frequently a very dynamic state machine.

Airflow is also based on a static graph. So it is not really suitable for complex scenarios that do not fit into its DAG paradigm. The even more severe problem with Airflow is its very limited scalability. That's why I've never heard it being used for service orchestration outside of the data pipeline control plane where scalability is not really needed.

Temporal Workflow was built from the beginning as a very dynamic and highly scalable orchestration system. There are many high throughput service orchestration use cases in production at many companies that rely on Temporal. Some of them are listed in the case studies.

Maxim Fateev
  • 6,458
  • 3
  • 20
  • 35
  • how do u make the distinction between a static vs dynamic graph ? can you illustrate with an example ?? – redzedi Oct 23 '19 at 18:34
  • For example the first activity returns the list of operations to execute and then workflow executes those operations. But even more simple example would be ML learning when the number of iterations of the algorithm is not known in advance. – Maxim Fateev Oct 23 '19 at 22:44
  • Can you elaborate on your first paragraph? When you say "slow requests or services being down for a long time," it sounds like that means it's best for events in the hundreds per second and not millions, but I'd imagine the "tip driver" occurs at a high rate. And what do you mean by services being down for a long time? My initial take is that it sounds like Cadence is best for adding reliability and stability to a company's existing microservices, but isn't going like using spring boot microservices to try to do large real time, event-time based processing tasks, for example. – Jicaar Jun 23 '21 at 19:01
  • @MaximFateev Basically, what is the ideal use case for Cadence that Flink isn't built to handle? – Jicaar Jun 23 '21 at 19:05
  • For example money transactions. With possible long outages of participating services. – Maxim Fateev Jun 23 '21 at 20:53
  • Hey @Maxim, I noticed that you edited the post above in a way that gives the impression that Temporal is being used at Uber. Could you please revise it to make it clear that Uber continues using & developing Cadence? Thanks! – Emrah Seker Oct 01 '21 at 18:45
  • Fixed, thanks for letting me know that the edit was ambiguous. – Maxim Fateev Oct 01 '21 at 20:30
4

There are folks building platforms on top of Flink that support use cases more like what you are describing than what is straightforwardly done with Flink itself. In particular, I want to suggest you take a look at Stateful Functions, which is a new open source library from Ververica, the original creators of Flink. From what I've understood of your requirements, this seems like it might be a good fit.

Cogynt is another example of this trend of building platforms on top of Flink to support use cases that have been difficult to implement with Flink itself, while leveraging its high-performance, fault tolerant, exactly-once stream processing engine.

Disclaimer: I work for Ververica.

David Anderson
  • 39,434
  • 4
  • 33
  • 60