-1

We installed spring xd on YARN. I see my applications running on YARN and containers getting deployed on YARN however XD-ADMIN is not getting started by YARN Resource Manager.

Just wanted to understand is it an expected behaviour that XD Admin should be started explicitly ? Can YARN RM start XD-Admin without external intervention ?

Bhanu-Bigdata Developer
  • 131
  • 5
  • Which XD version you are using and what commands you used to start XD on YARN? – Janne Valkealahti Apr 02 '15 at 09:00
  • Hi, i am using spring-xd-1.1.0.RELEASE, and spring-xd-1.1.1.RELEASE-yarn libraries. I have used push command and submit commands to get my applications on Yarn. I just wanted to understand whether we need to start admin manually or YARN RM will start automatically ? Thanks for your help on this! – Bhanu-Bigdata Developer Apr 02 '15 at 10:55

1 Answers1

0

Mixing 1.1.0/1.1.1 should not matter because we didn't touch any YARN features(though I haven't tried it). You didn't mention what kind of setup this is in terms of a cluster size, etc.

XD on YARN did a huge change from 1.0 to 1.1 where whole YARN functionality was pretty re-written from scratch. With 1.0 you had to run separate YARN apps to start admin and containers, with 1.1 this was changed so that both admin and container could be controlled by a same application master.

Having said that, to give a little bit of background, with 1.1, on default(defined in servers.yml) YARN starts 3 containers and 1 admin. If you by any change are working with single node hadoop, you may hit resource limits. It's worth to read section "Configuring YARN memory reservations" from 1.1 reference docs. I've surely burned few time by trying to figure out why something doesn't start on YARN.

Problem might be 1) cluster doesn't have enough resources if those 3 default containers took it all, 2) admin is requestion too much memory and YARN is refusing to fulfil resource allocation request, etc.

What I can propose is to first drop container count to 1 and see if that helps. Secondly try to tweak or turn off memory settings mentioned in that ref doc section.

I've also seen cases with some OS's that ratio of physical mem vs. virtual mem of a process is just calculated wrong by OS(think some versions of RHEL do this) and YARN just kills your process.

It's also worth to try to see from logs if admin was actually started but killed by YARN(due to mem issues), if it failed to start(something went wrong with zookeeper, etc), of if it never got resources from YARN.

Let us know how it goes!

Janne Valkealahti
  • 2,552
  • 1
  • 14
  • 12
  • Hi Janne, Thanks for responding! we tried by reducing container instances to 1 and admin also to 1. However it didn't work for us. I have few observations which need some clarifications. 1. we are using Horton Works 2.6. so which portion of yarn classpath settings do we need to uncomment ? is it Hortonworks HDP 2.2 or Hadoop 2.5.x? 2. We have kept all YARN configurations as per the document, however irrespective of configuration settings each container is taking 4GB. Not sure where we went wrong ? 3. Also when we are clicking on the Appmaster we are getting "WHITELABEL ERROR PAGE" – Bhanu-Bigdata Developer Apr 08 '15 at 11:25
  • Application is getting failed status after setting the YARN Memory as per the document. earlier values are much bigger than what document suggested. Here is the error log when application gets failed. Diagnostics: Container [pid=23039,containerID=container_1428489982586_0004_02_000001] is running beyond virtual memory limits. Current usage: 247.6 MB of 1 GB physical memory used; 4.7 GB of 2.1 GB virtual memory used. Killing container. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 Failing this attempt. Failing the application. – Bhanu-Bigdata Developer Apr 08 '15 at 11:28
  • Hi, we have figured out the issue. Its issue with HSQLDB which is hammering admin instantiation. We fixed it.. Admin is up and running. Thanks! – Bhanu-Bigdata Developer Apr 10 '15 at 16:38
  • ok good, I was kinda wondering if this was a yarn or xd specific error. It's usually a little difficult to find out what goes wrong when running apps on yarn. – Janne Valkealahti Apr 11 '15 at 17:52
  • Yup..we figured out the issue after checking YARN application logs. We installed mysqldb and made it up to get admin instantiation. – Bhanu-Bigdata Developer Apr 14 '15 at 05:09