0

I have installed a single-node Apache YARN with Kafka-Zookeeper and Spring Cloud Dataflow 1.0.3.

All is working fine, but when I made some deployment examples, like:

stream create --name "ticktock" --definition "time | hdfs --rollover=100" --deploy

http --port=8000 | log --name=logtest --level=INFO

The stream's status never stays on "deployed". It keeps changing on "undeployed"->"parcial"->"deployed" in a constant loop.

However, the YARN application deploys successfully, but it's like the comunication between Spring Cloud Dataflow server instance and Apache Hadoop it's constantly failing.

What can be the possible issue for this?

Thanks in advance!

Juan Echeverria
  • 1
  • 2
  • When it comes to a loop of deployment failures, the usual suspect is around the memory allocated for app-master and yarn containers. How are you provisioning SCDF server on YARN? Are you using the Ambari plugin? (_if not, why?_). Please review this [section](http://docs.spring.io/spring-cloud-dataflow-server-yarn/docs/current-SNAPSHOT/reference/htmlsingle/#yarn-troubleshooting) on memory requirements. There are [global settings](http://docs.spring.io/spring-cloud-dataflow-server-yarn/docs/current-SNAPSHOT/reference/htmlsingle/#_global_yarn_memory_settings) that you can override, too. – Sabby Anandan Nov 02 '16 at 14:37

2 Answers2

0

Well, there's not a lot of information to go on here, but you may want to check and make sure you have the necessary base directory created in HDFS and that your Yarn user has read/write permission to it.

spring.cloud.deployer.yarn.app.baseDir=/dataflow

franktylerva
  • 91
  • 1
  • 4
  • Hello! Thanks for answering. How I can check the HDFS directory? That is using the command provided by the SCDF console: _hadoop fs ls_ Thanks in advance!. – Juan Echeverria Oct 27 '16 at 22:28
  • Hello again. I created the directory using: _hadoop fs -mkdir -p /dataflow_ And added to the SCDF configuration. The user who executes all the services (including SCDF) has full access and permissions. After doing that and testing a simple deployment, the error persist. What other check I can do? Thanks in advance. – Juan Echeverria Oct 28 '16 at 15:31
  • Can you provide your servers.yml file? – franktylerva Nov 04 '16 at 11:18
0

Thanks to all for your answers! Been busy with other high-priority projects, but I was able to build all using Ambari (including the plugin mentioned by Sabby Adannan). Now all is working great!.

Juan Echeverria
  • 1
  • 2