2

I know ganglia can be used to monitor resource utilization in a cluster like Spark. But it will provide an overall report for my application.

But if I want to know how much resource is being utilized by a specific portion of my code, is there any way?

A
My code
B

For example, I want to know CPU/RAM utilization from A to B. I can calculate the runtime within the code, i.e. java application for spark, but I don't know how I can specifically know the resource utilization for that portion. I have an idea that if somehow I can generate a report (like call api for ganglia report) at B, it can basically show me resources utilized up to B. Although it will not exclude anything before A still it will work for me for now if such solution exists.

Thank you in advance.

user3086871
  • 671
  • 3
  • 7
  • 25

1 Answers1

1

Apparently new project sparkoscope seems to work on this, i.e. monitoring from source code level. However, their project is not well documented,so I am facing trouble to put their project in working position. Nevertheless, it is a start. Hope it helps someone like me.

user3086871
  • 671
  • 3
  • 7
  • 25
  • 1
    I think with a few adjustments it can work for you. Since the metrics are exported in HDFS together with their timestamp, it's easy to extract the part you are interested by using the start and stop timestamp of your code. Regarding the installation requirements I can help out. Is this https://github.com/ibm-research-ireland/sparkoscope/issues/8 your issue? – Yiannis Gkoufas Jun 18 '17 at 09:19
  • It was, and it was solved later I think it was something to do with machines I used – user3086871 Jul 01 '17 at 10:50