1

I have a hadoop cluster with 7 nodes, 1 master and 6 core nodes. Ganglia is setup on each machine, and the web front end correctly shows 7 hosts.

But it only shows metrics from the master node (with both gmetad and gmond). The other nodes have the same gmond.conf file as the master node, and the web front end clearly sees the nodes. I don't understand how ganglia can recognize 7 hosts but only show metrics from the box with gmetad.

Any help would be appreciated. Is there a quick way to see if those nodes are even sending data? Or is this a networking issue?

update#1: when I telnet into a gmond host machine that is not the master node, and look at port 8649, I see the XML but no data. When I telnet to 8649 on the master machine, I see XML and data. Any suggestions of where to go from here?

Eka
  • 63
  • 2
  • 10

2 Answers2

4

Set this to all gmond.conf files of every node you want to monitor:

send_metadata_interval = 15 // or something.

Now all the nodes and their metrics are showed in master (gmetad).

This extra configuration is necessary if you are running in a unicast mode, i.e., if you are specifying a host in udp_send_channel rather than mcast_join. In the multi-cast mode, the gmond deamons can query each other any time and proactive sending of monitoring data is not required.

Sebastian Kruse
  • 318
  • 2
  • 7
Giorgos Argyriou
  • 61
  • 8
0

In gmond configuration, ensure the following is all provided:-

cluster {
  name = "my cluster" #is this the same name as given in gmetad conf?
  ## Cluster name
  owner = "unspecified"
  latlong = "unspecified"
  url = "unspecified"
}

udp_send_channel {
  #mcast_join = 239.2.11.71 ## Comment this
  host = 192.168.1.10 ## IP address/hostname of gmetad node
  port = 8649
  ttl = 1
}

/* comment out this block itself
udp_recv_channel {
...
}
*/

tcp_accept_channel {
  port = 8649
}

save and quit. Restart your gmond daemon. Then execute "netcat 8649". Are you able to see XML with metrics now?

vivek_nk
  • 1,590
  • 16
  • 27