1

YourKit alleges to be able to open binary HPROF dumps, but it insists that binary dumps made from Hadoop jobs are not valid snapshots.

The dumps are made by specifying the mapred.task.profile.params option with a value of -agentlib:hprof=cpu=samples,heap=sites,depth=6,force=n,thread=y,verbose=n,file=%s,format=b (as specified in the Hadoop: The Definitive Guide.

Is there any way to open this file with YourKit? Are there other graphical profiling tools that will open the files created?

DeejUK
  • 12,891
  • 19
  • 89
  • 169
  • You can directly hook in yourkit via its yjpagent and take snapshots. There is no need to use hprof. – Thomas Jungblut Dec 20 '12 at 17:10
  • @Thomsas: Thanks - do you know of any examples of using this with Hadoop? I assumed that connecting to a process somewhere on the Hadoop cluster would be more hassle than using an HPROF dump. – DeejUK Jan 02 '13 at 09:50
  • it is basically just deploying the yjpagent that is shipped with yourkit and instead of `-agentlib:hprof` use `-agentlib:yjpagent`. – Thomas Jungblut Jan 02 '13 at 10:18
  • Thanks. Such is Hadoop it's quite hard to know where and when mappers will start, so does this approach create a dump or does it require connecting to the process as it's running? – DeejUK Jan 03 '13 at 15:17
  • Exactly, you need to take care that the agentlib is on every node. In yourkit you can create snapshots on specific events (high CPU, OOM error) or when the application exists- you should have a look at their documentation, what to configure for the agentlib parameter. Live monitoring is a cool feature as well if you can connect to a live server. – Thomas Jungblut Jan 03 '13 at 16:31

0 Answers0