I am trying to launch a Spring Batch Remote Partitioning with Spring Cloud Dataflow. I have a remote partitioning that is based on https://github.com/mminella/LearningSpringBatch/tree/master/src/remotePartitioning
The Spring Batch (Master and Slave) will run in Kubernetes, currently running in minikube.
The SCDF is running in minikube.
Then launch the slave:
java -jar -Dspring.profiles.active=slave target/remote-partitioning-0.0.1-SNAPSHOT.jar
Launch the master:
java -jar -Dspring.profiles.active=master target/remote-partitioning-0.0.1-SNAPSHOT.jar time=$(date +%s)
In the example https://dataflow.spring.io/docs/batch-developer-guides/batch/data-flow-spring-batch/ one task is registered and launched. And Steps associated to the job stays under the job.
Question is, how do I launch the master and the slave? Register a task for the master (docker image that runs with profile master) and another for the slave (docker image that runs with profile slave)?
How/Where do I configure the Job Parallelism for the slaves? In my case I am planning to run 3-5 slaves.
Where do I assign/config the Kubernetes Cluster (in my case minikube) that the Job/Steps will run?
Another question that I have is how to pass the argument between the SCDF, docker container to the java launcher. How to pass and argument from Spring Cloud Dataflow Task to a Spring Batch Job Running in Kubernetes
I just could not find any example/instructions on how to do this for remote partitioning. Any help/lead is appreciated.
Thx, Markus
