3

Team, Currently I am working on spring-xd and using as a runtime container for data analytics and yarn jobs.

My questions are

1) Can I leverage the same environment setup which I used for spring-xd?
2) From the documentation,I read that it can be deployed as micro services, is it using embedded drivers for stream processing? If it is using embedded drivers, can I use it to deploy in a clustered environment with the same infrastructure leveraged for spring-xd?
3) Is there any specific wrappers built for Apache Spark?

My Environment: Spark 1.6.1, Hadoop 2.7.2, zookeeper 3.6.8, redis 3.2, spring-xd-1.3.1

Any help on this specific queries would be highly appreciated.

Sabby Anandan
  • 5,636
  • 2
  • 12
  • 21
Praveen Kumar K S
  • 3,024
  • 1
  • 24
  • 31

1 Answers1

4

Can I leverage the same environment setup which I used for spring-xd?

Spring Cloud Data Flow (SCDF) relies on spring-cloud-deployer SPI and there are implementations for Cloud Foundry, Apache YARN, Apache Mesos and Kubernetes. Given that you're already having an Hadoop cluster with YARN in use, you could provision the YARN implementation of the SCDF-server.

is it using embedded drivers for stream processing?

Not clear what you mean by this. If you're referring to jdbc-drivers; specifically, when using jdbc as a sink application, we do embed OSS-friendly drivers, yes.

If it is using embedded drivers, can I use it to deploy in a clustered environment with the same infrastructure leveraged for spring-xd?

Perhaps answer to #1 covers this. You could leverage the same infrastructure and provision YARN SCDF-server using the Ambari plugin.

Is there any specific wrappers built for Apache Spark?

We have Spark as a client and cluster applications. You can register them in SCDF to build task/batch pipelines.

Sabby Anandan
  • 5,636
  • 2
  • 12
  • 21
  • ,Thanks for your convincing answer!! I meant embedded drivers as spark-driver which is mavenized along with spring-boot which can be deployed in a runtime container. Kindly confirm if this option is feasible and would be great, if you could send me some references. – Praveen Kumar K S Jun 03 '16 at 15:24
  • 1
    I've created some instructions for running a small Spark test app on YARN. See - https://gist.github.com/trisberg/79d50dc46c11c0fb7020c8ccc72d1234#running-a-spark-application-on-yarn-using-spring-cloud-data-flow. We'll work on improving the docs before the GA release. – Thomas Risberg Jun 07 '16 at 20:09
  • @Thomas, Gone through your samples, really helpful. Also, can you you please post some more examples in Java as well? – Praveen Kumar K S Jun 26 '16 at 03:00