1

Launching using spark-ec2 script results in:

Setting up ganglia RSYNC'ing /etc/ganglia to slaves... <...>

Shutting down GANGLIA gmond: [FAILED]

Starting GANGLIA gmond: [ OK ]

Shutting down GANGLIA gmond: [FAILED]

Starting GANGLIA gmond: [ OK ]

Connection to <...> closed. <...> Stopping httpd:
[FAILED] Starting httpd: httpd: Syntax error on line 199 of /etc/httpd/conf/httpd.conf: Cannot load modules/libphp-5.5.so into server: /etc/httpd/modules/libphp-5.5.so: cannot open shared object file: No such file or directory [FAILED] [timing]

ganglia setup: 00h 00m 03s Connection to <...> closed.

Spark standalone cluster started at <...>:8080 Ganglia started at <...>:5080/ganglia

Done!

However, when I netstat, there is no 5080 port listened on.

Is this related to the above error with httpd or it's something else?

EDIT:

So the issue is found (see the answer below), and the fix can be applied locally on the instance, after which Ganglia works fine. However the question is how to fix this issue in the root, so that spark-ec2 script can start Ganglia normally without intervention.

Oleg Shirokikh
  • 3,447
  • 4
  • 33
  • 61

3 Answers3

3

The fact that ganglia is not available is related to these errors - ganglia is php application and it won't run without php module for apache.

Which version of spark you are using to start cluster?

It is wierd error - these file should be present in AMI image.

vvladymyrov
  • 5,715
  • 2
  • 32
  • 50
  • Here is my answer :) I'm testing this with spark master (spark 1.3.1 doesn't let to specify custom spark-ec2 repo. With spark master branch - you can use these options `--spark-ec2-git-repo=https://github.com/smartkiwi/spark-ec2 --spark-ec2-git-branch=branch-1.3` . But of course it is better to wait for official update and not rely on third party sources for obvious security reasons. – vvladymyrov Jun 11 '15 at 01:58
  • Gotcha.. It's not a problem, since I can temporary manually change httpd.conf. But good job finding and submitting the fix – Oleg Shirokikh Jun 11 '15 at 02:00
1

Just traced the error: /etc/httpd/conf/httpd.conf is trying to load libphp-5.5 library while modules/ contains libphp-5.6 version...

Changing httpd.conf fixes the issue, however I'd be good to know a permanent fix within spark-ec2 script

Oleg Shirokikh
  • 3,447
  • 4
  • 33
  • 61
  • 1
    I've submitted pull request with the fix. It is not spark-ec2 script in spark codebase. It is park of mesos/spark-ec2. Here is link to pull request - https://github.com/mesos/spark-ec2/pull/118 . Once they'll merge it - there will be no need to update spark. spark-ec2 script will pull new version from mesos/spark-ec2 repository – vvladymyrov Jun 11 '15 at 01:28
0

This is because httpd fails to launch. As you have noted httpd.conf is trying to load modules and failing. You can reproduce the problem via apachectl start and examine exactly what modules are failing to load.

In my case there was one involving "auth" and "core". The last four (maybe five) listed will also fail to load. I did not encounter anything related to PHP so maybe our cases our different. Anyway the hacky solution is to comment out the problems. I did so and am running Ganglia without issue.

Chris
  • 507
  • 6
  • 13