0

I have a spark cluster in yarn mode, and I installed Spring Cloud Data Flow on the master node directly, followed the simple steps of the document, everything goes well.
I created simple stream and task, deployed them, nothing happened there.
stream create ticktock --definition “time | log” --deploy
task create --name pi --definition "spark-yarn --spark.app-jar=<spark-example.jar> --spark.assembly-jar=<spark-assembly-jar-in-hdfs> --spark.app-class=org.apache.spark.examples.SparkPi"
In webUI I can see the stream is deployed and the task is in unknown status. But in the console of yarn-server, something abnormal printed as following:
2017-02-04 02:26:14.736 INFO 4940 --- [nio-9393-exec-8] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:14.737 INFO 4940 --- [nio-9393-exec-8] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:14.738 INFO 4940 --- [nio-9393-exec-8] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:14.740 INFO 4940 --- [nio-9393-exec-8] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:19.998 INFO 4940 --- [nio-9393-exec-7] o.s.c.deployer.spi.yarn.YarnAppDeployer : Checking status of ticktock:time:application_1486192446476_0003 2017-02-04 02:26:19.998 INFO 4940 --- [nio-9393-exec-7] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:20.000 INFO 4940 --- [nio-9393-exec-7] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:20.005 INFO 4940 --- [nio-9393-exec-7] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:20.007 INFO 4940 --- [nio-9393-exec-7] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:20.008 INFO 4940 --- [nio-9393-exec-7] o.s.c.deployer.spi.yarn.YarnAppDeployer : Checking status of ticktock:log:application_1486192446476_0003 2017-02-04 02:26:20.009 INFO 4940 --- [nio-9393-exec-7] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:20.010 INFO 4940 --- [nio-9393-exec-7] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:20.011 INFO 4940 --- [nio-9393-exec-7] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:20.013 INFO 4940 --- [nio-9393-exec-7] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:25.268 INFO 4940 --- [nio-9393-exec-3] o.s.c.deployer.spi.yarn.YarnAppDeployer : Checking status of ticktock:time:application_1486192446476_0003 2017-02-04 02:26:25.268 INFO 4940 --- [nio-9393-exec-3] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:25.270 INFO 4940 --- [nio-9393-exec-3] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:25.274 INFO 4940 --- [nio-9393-exec-3] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:25.276 INFO 4940 --- [nio-9393-exec-3] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:25.277 INFO 4940 --- [nio-9393-exec-3] o.s.c.deployer.spi.yarn.YarnAppDeployer : Checking status of ticktock:log:application_1486192446476_0003 2017-02-04 02:26:25.277 INFO 4940 --- [nio-9393-exec-3] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:25.279 INFO 4940 --- [nio-9393-exec-3] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:25.280 INFO 4940 --- [nio-9393-exec-3] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa 2017-02-04 02:26:25.281 INFO 4940 --- [nio-9393-exec-3] o.s.c.d.s.y.DefaultYarnCloudAppService : Cachekey STREAMnull found YarnCloudAppServiceApplication org.springframework.cloud.deployer.spi.yarn.YarnCloudAppServiceApplication@72feb9aa

I think it means that the stream and task are not deployed successfully, right? How to solve this problem? Thanks.

Lionel
  • 13
  • 2
  • Your problem description doesn't include all the details. Perhaps you could edit the post and supply some of the details as listed [here](https://github.com/spring-cloud/spring-cloud-dataflow/wiki/Reporting-Issues)? Also, we are addressing the Spark-task issue - see [spring-cloud-task-app-starters/spark-yarn#3](https://github.com/spring-cloud-task-app-starters/spark-yarn/pull/3) for more details. – Sabby Anandan Feb 07 '17 at 22:16
  • @SabbyAnandan, Thank you for your reply. Actually I've got stream work successfully now, but I can not find the log module output using spring cloud dataflow for yarn when I tried the stream " time | log ". However, the stream " time | file " or " http | hdfs " works well. Never mind, I think the stream part works well now, it might be even better if you could tell me why the log module didn't show the log file path. The task part still can't submit it's job to spark-yarn, I'll read the link you supplied and have a try again. – Lionel Feb 10 '17 at 09:01
  • I have one more question about spark-yarn module, I see the modification of spark version 1.6.3, does it mean that it can only submit to spark cluster 1.6.3 in yarn mode or it can also support the latter versions? – Lionel Feb 10 '17 at 09:32
  • Please consider opening a new thread on this. – Sabby Anandan Feb 10 '17 at 14:43

0 Answers0