I have a Hadoop cluster set up, and I would like to view the network usage between all nodes in this cluster, i.e. if I have N nodes, I want to see NxN network connections so that I can view all traffic between all nodes.
I am running the cluster on a Linux system, how can I do this?
ntop-ng combined with an s-flow or netflow connector from your switches.
This is not a free solution if you're a commercial entity, and if you want extra analysis then it's even more not-free.
E.g. below is my top-talkers at the instant that the screenshot was taken:
There's plenty of other information you can get from ntop-ng. As far as I'm concerned it's pretty much a must-have for visualising network traffic.
Note that on 10GbE networks you're going to need some serious hardware to run it on, but for 1Gbps networks then you can run it on pretty much anything, depending on the number of switches and ports you're monitoring.
