3

I am trying to use spring-cloud-dataflow to create streams over an existing set of servers. It is not a production environment, I am aware of the fault tolerance issues with the "local" deployer.

I have in the past created this infrastructure by:

1-Created a "scdf-server" by implementing the provided interface (ui and shell worked fine with no changes) . This server forward the request to an application (2) running on each server using REST calls

2-On each server I have an application that encapsulates the local deployer and receives the REST calls from the server passing the along to the encapsulated local deployer.

This approach worked perfectly, but now I need some of the new features in the new SCDF version and with the introduction of skipper I am not sure this is the best approach, or if it is even possible, need advice.

TIA

ka2
  • 151
  • 2
  • 12
  • Because my apps wont be using the standard pipes (they have more than one input or outputs) I am wondering if I should use skipper directly, the only benefit of SCDF would be the UI, and I wont be using most of it anyway. – ka2 Feb 08 '20 at 05:35
  • I think that would need an instance of skipper server on each box, and then on the SCDF server somehow have a different implementation of "SkipperStreamDeployer" class that can route the requests to a skipperClient corrsponding with each skipper server, and keep track there of what skipper client is running what application. – ka2 Feb 08 '20 at 06:07
  • Ignoring the local deployment, suppose there is a need to deploy part of an stream into a cloud and another part of the same stream in a different cloud, what would be the best way to accomplish this? – ka2 Feb 19 '20 at 00:11

1 Answers1

0

Looks like you have already explored the code to have better insights on how things are implemented.

As you stated, the local deployer implementation is never meant for production deployments. Hence, the SCDF team has dev-centric design goals on local deployer implementation and also limited its scope on extensions as well.

Given this, the way you designed your setup based on the needs, going with the Skipper server embedded into each app wouldn't work as Skipper server is expected to have a 1-on-1 relationship with SCDF server. Moreover, Skipper server also maintains a local State machine that might go stale when you have a different instance of Skipper server that listens to the same SCDF server. This makes the whole design very complex indeed.

Ilayaperumal Gopinathan
  • 4,099
  • 1
  • 13
  • 12
  • As you said, I have done my research. I understand local deployer does not handle resiliency by the apps, but where exactly in the documentation does it say that it is to be use for development only? and not for production? – ka2 Feb 18 '20 at 13:56
  • It is mentioned in the README here: https://github.com/spring-cloud/spring-cloud-deployer-local – Ilayaperumal Gopinathan Feb 18 '20 at 15:58
  • it says"any use of this deployer in a production environment should be accompanied with additional monitoring at the app level" ... it does not say it should not be used in production, just that you should handle those things yourself.... – ka2 Feb 18 '20 at 20:41
  • Ignoring the local deployment, suppose there is a need to deploy part of an stream into a cloud and another part of the same stream in a different cloud, what would be the best way to accomplish this? – ka2 Feb 18 '20 at 20:43