How do you automate deploying Spring Data Flow?

Question

I've got an instance of Spring Cloud Data Flow stood up with my apps and streams that I need. I even wrote scripts that work within the command shell to deploy all this stuff at once. The problem is that I have a manual step now that I need to do rather than just have it run out of the box.

With both Windows and Linux I know how to run a script through a shell and have it end when the script is run.

The problem is that I can's seem to do the same with the Spring Data Flow Shell. I'd like to do something like this:

java -jar spring-cloud-dataflow-shell.jar --runScript my-awesome-script.shell

However, I can't see any documentation that says something like that exists. Every option I could find didn't run the script and kept the shell open. Or is there another option completely that I am not aware of. My 15 years of Java experience was interrupted by doing .Net the past 6 years, and Spring looks completely different than it used to.

score 4 · Accepted Answer · answered Jun 16 '17 at 12:50

Sometimes I am such a numskull. After searching for it all day yesterday, it dawned on me to use the --help option to get the command line options for the shell.

C:\Dev>java -jar spring-cloud-dataflow-shell-1.2.1.RELEASE.jar --help
Data Flow Options:
  --dataflow.uri=<uri>                              Address of the Data Flow Server [default: http://localhost:9393].
  --dataflow.username=<USER>                        Username of the Data Flow Server [no default].
  --dataflow.password=<PASSWORD>                    Password of the Data Flow Server [no default].
  --dataflow.credentials-provider-command=<COMMAND> Executes an external command which must return an OAuth Access Token [no default].
  --dataflow.skip-ssl-validation=<true|false>       Accept any SSL certificate (even self-signed) [default: no].
  --spring.shell.historySize=<SIZE>                 Default size of the shell log file [default: 3000].
  --spring.shell.commandFile=<FILE>                 Data Flow Shell executes commands read from the file(s) and then exits.
  --help                                            This message.

When I tried the --spring.shell.commandFile option, it did exactly what I needed the shell to do: run the script and end.

java -jar spring-cloud-data-flow-shell.jar --spring.shell.commandFile=my-awesome-script.shell

NOTE: The documentation on the Spring website does not mention this at all. If this is something that can be done via Spring Config Server I would also like to know about it.

score 1 · Answer 2 · answered Jun 16 '17 at 15:18

1

Apart from the --spring.shell.commandFile option, you could also use the dataflow:>script --file <YOUR_AWESOME_SCRIPT> that include the various stream/task create and deploy commands. I've seen users versioning the submitted file artifact for different environments, too. Sadly, both the options are not documented and we will fix it (spring-cloud/spring-cloud-dataflow#1534).

With that said, for SCDF 2.0, we are going to add native CI/CD support inspired by Kubernetes Helm. This will include primitives in SCDF's shell/gui for app level canaries, versioning of apps, and as well as definitions backed by config-server.

answered Jun 16 '17 at 15:18

Sabby Anandan

5,636
2
12
21

The `script --file my-awesome-script.shell` command within the shell was where I was before I asked the question. I just want to be able to automate deployment and configuration without having to add an additional step. Thanks for opening that ticket for me. Is there a satisfying story with the Spring Configuration Server to essentially deploy apps and stream? – Berin Loritsch Jun 16 '17 at 18:19
On an unrelated note: when I Googled SCDF the first response was Singapore Civil Defense Force, and the second was Southern California Development Forum... We're running out of acronyms :) – Berin Loritsch Jun 16 '17 at 18:22
1

If you're to automate the deployment of streams/tasks, the currently available options are either the REST-APIs or the DataflowTemplate, which you'd have to integrate directly with the CI systems. A few customers instantiate the deployment via Jenkins/Concourse. For each app, the app and the binding properties are configured in the versioned YAML file backed by config-server. – Sabby Anandan Jun 16 '17 at 21:04
2

In 2.0, though, we will support the individual app lifecycle directly from the DSL/UI/APIs. Thanks for the feedback on Google search results. – Sabby Anandan Jun 16 '17 at 21:08

score 1 · Answer 3 · answered Oct 30 '18 at 09:38

//Start spring cloud dataflow server
    //create the file 
    //file name:Task_scripts.shell and inside file bas the below command
    app register --name PredictionBatchTest --type task --uri file://D:/PredictionBatch/target/PredictionBatch-0.0.1-SNAPSHOT.jar
    //Create the shell script file and add below command and execute the shell script
    java -jar scdf-shell-1.6.3.RELEASE.jar --spring.shell.commandFile=D:/my-awesome-script.shell

How do you automate deploying Spring Data Flow?

3 Answers3