Cannot install Ganglia on EMR 4.0.0

Question

I'm following this guide to install Spark on the latest AMI / EMR cluster:

http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-spark-launch.html

I'd like to install Ganglia to monitor the cluster so I added Name=Ganglia to the list of applications to install:

aws emr create-cluster --name "Spark cluster" --release-label emr-4.0.0 --applications Name=Spark Name=Ganglia --ec2-attributes KeyName=myKey --instance-type m3.xlarge --instance-count 3 --use-default-roles

But I'm receiving the following error message:

A client error (ValidationException) occurred when calling the RunJobFlow operation: Specified application: Ganglia is invalid

Here are the versions that I am running:

aws --version
aws-cli/1.7.41 Python/2.7.7 Linux/2.6.32-431.29.2.el6.x86_64

Same goes for Hue. The cli is not aligned with the documentation. — Interfector, Jul 29 '15 at 14:22

score 2 · Answer 1 · answered Jul 29 '15 at 21:22

2

Ganglia is not part of the EMR 4.0 release.

The official API documentation shows valid values of: "Hadoop", "Hive", "Mahout", "Pig", and "Spark."

It appears that the AWS CLI documentation is incorrect.

answered Jul 29 '15 at 21:22

John Rotenstein

241,921
22
380
470

Is there another way to install Ganglia whilst still installing Spark as an application using `--release-label emr-4.0.0`? – d18s Jul 30 '15 at 08:31
Update: Ganglia is now available again. – John Rotenstein Jan 07 '16 at 05:24
Yes ganglia is now available, with EMR 4.2.0 release: http://docs.aws.amazon.com/ElasticMapReduce/latest/ReleaseGuide/emr-ganglia.html – and_apo Jan 07 '16 at 11:30

score 1 · Answer 2 · answered Aug 19 '15 at 20:14

1

The usual Ganglia bootstrap action also doesn't work under emr-4.0.0. See this issue.

answered Aug 19 '15 at 20:14

Russ

3,644
1
13
15

score 0 · Answer 3 · answered Nov 17 '15 at 07:45

0

Amazon is probably working on creating an official release of Ganglia for EMR 4.x. Until that comes out, you can use this bootstrap action:

s3://support.elasticmapreduce/release/4.x/ganglia/install_ganglia_emr-4.0.0.rb

answered Nov 17 '15 at 07:45

amey91

542
6
17

score -1 · Answer 4 · answered Aug 03 '15 at 16:54

If you continue by using a bootstrap action, you should be ok.

// AWSCLI Example

aws emr create-cluster                  \
  --bootstrap-actions file://bootstrap_actions.json \
  ...

// bootstrap_actions.json

{
    "Name": "Install Ganglia",
    "Path": "s3://elasticmapreduce/bootstrap-actions/install-ganglia"
  },

Or from DataPipeline (pipeline definition file example):

   {
      "id": "EmrCluster",
      "name": "My Cluster (staging)",
      "type": "EmrCluster",
      "bootstrapAction": [
        "s3://elasticmapreduce/bootstrap-actions/install-ganglia"
      ],
      etc..
    },

Cannot install Ganglia on EMR 4.0.0

4 Answers4