0

Is there a possibility to run Spring Cloud Data Flow in cluster mode ? I have many apps running on SCDF, I wanted to ensure that SCDF is always up and running just like a kafka cluster.

Jibit George
  • 76
  • 4

1 Answers1

0

As far as you have multiple instances of SCDF applications, you can plug an LB in front to mediate the traffic. That's a common practice.

That said, though, SCDF itself doesn't get huge traffic from requests/responses per see. The traffic really only occurs when you are using the SCDF Dashboard or exercising the SCDF APIs. Even for any reason, the SCDF App crashes, the platform (eg: K8s) where it runs will bring it back up in <20secs. SCDF is a Boot App, and roughly this much time it takes to start - in the future, there'll be a Graal Native compiled version of SCDF, which would dramatically reduce the startup time.

Given the above explanation, you won't have a noticeable downtime. This is one of the reasons to run SCDF in container orchestration platforms like Kubernetes, so you can leverage the platform's resiliency functionality.

Sabby Anandan
  • 5,636
  • 2
  • 12
  • 21
  • There are situations where the server pod may go down while client applications are trying to programmatically launch tasks (either using the Java task DSL or by using Spring Batch partitioning with `DeploymentPartitionHandler`). If we cannot have more than one SCDF server pod, then we loose the fault tolerance that is an important feature of distributed systems? Without fault tolerance, client programs would have to have to handle failed attempts of launching tasks (404 error for example). This somehow does not seem like something clients programs should be responsible for IMO. – Chetan Kinger Jun 13 '21 at 15:58