0

How do these Spring components relate/differ to/from each other? What does each represent conceptually? Would one use them together or are they competing projects?

  • Spring Cloud Data Flow
  • Spring Cloud Stream
  • Spring Cloud Task
  • Spring Cloud Task App Starters
  • Spring Batch

From my understanding, SC Tasks are just "units of work" to execute, a processing unit in the form of a short-lived/task-based microservice. SC Data Flow is orchestration for the tasks. These two I (think I) understand how they relate and what they represent conceptually, but a lot of documentation and examples talk about the other projects in the same context.

I also thought that SC Task was a replacement for Spring Batch but in some examples they seem to imply that Spring Batches are executed inside SC Tasks

Hilikus
  • 9,954
  • 14
  • 65
  • 118

1 Answers1

2

Thanks for your interest in Spring Cloud projects! Find below the high-level introductions for the primary projects involved in Spring Cloud Data Flow (SCDF) ecosystem. The launch blog covers the backstory and among other details.

  • Spring Cloud Stream is a lightweight event-driven microservices framework to quickly build applications that can connect to external systems (eg: Kafka, Cassandra, MySQL, Hadoop, ..).

  • Spring Cloud Task is a short-lived microservices framework to quickly build applications that perform finite amounts of data processing (eg: batch-jobs, ..). The connection with Spring Batch framework is explained in the launch blog linked above.

  • Spring Cloud Data Flow provides the orchestration mechanics to deploy applications built with Spring Cloud Stream and Spring Cloud Task programming model to a variety of runtime platforms including Cloud Foundry, Apache Yarn, Apache Mesos and Kubernetes. There's community developed SCDF implementations for OpenShift and Nomad, too. More details here.

The building blocks visual from the project site should cover the high-level interaction between the various projects in SCDF's ecosystem.

Sabby Anandan
  • 5,636
  • 2
  • 12
  • 21
  • Thank you @Sabby, the launch blog was very useful. I am still confused wrt Spring Cloud Task App Starters. Is it just a "control API" used by Data Flow to spawn SC Tasks? – Hilikus Nov 02 '16 at 15:34
  • 1
    The "Spring Cloud Task App Starters" project provides OOTB ready to use Task/Batch applications such as `sqoop-job`, `spark-job`, `jdbchdfs` etc. We also provide [bulk import links](http://cloud.spring.io/spring-cloud-task-app-starters/#task-app-starters-and-spring-cloud-data-flow-) to use the already built artifacts in [SCDF directly](http://docs.spring.io/spring-cloud-dataflow/docs/current-SNAPSHOT/reference/htmlsingle/#_registering_a_task_application). – Sabby Anandan Nov 02 '16 at 16:08
  • Similarly, we also have "Spring Cloud Stream App Starters" project that provides OOTB ready to use [streaming applications](https://github.com/spring-cloud/spring-cloud-stream-app-starters). See the [reference guide](http://docs.spring.io/spring-cloud-stream-app-starters/docs/1.0.4.RELEASE/reference/html/) for more details and there's [bulk import links](http://cloud.spring.io/spring-cloud-stream-app-starters/#stream-app-starters-and-spring-cloud-data-flow-) to register them in SCDF, too. – Sabby Anandan Nov 02 '16 at 16:10