0

I'm trying to profile reduce tasks on a cluster using

conf.setBoolean("mapred.task.profile", true);
conf.set("mapred.task.profile.params", "-agentlib:hprof=cpu=samples,heap=sites,depth=6,force=n,thread=y,verbose=n,file=%s");
conf.set("mapred.task.profile.maps", "0-1");
conf.set("mapred.task.profile.reduces", "0-1");

As expected, I've got 4 profile files generated:

     2849 Mar 25 15:30 attempt_201402231438_11879_m_000000_0.profile
  8786788 Mar 25 15:30 attempt_201402231438_11879_m_000001_0.profile
  4129237 Mar 25 15:31 attempt_201402231438_11879_r_000000_0.profile
 12644892 Mar 25 15:32 attempt_201402231438_11879_r_000001_0.profile

(files with _m_ are generated during mapper task profiling, and _r_ are generated during reduce task profiling)

attempt_201402231438_11879_m_000001_0.profile looks good:

THREAD START (obj=50000161, id = 200001, name="main", group="main")
THREAD START (obj=500021bd, id = 200004, name="TGT Renewer for yampolsy", group="main")
THREAD END (id = 200004)
...
TRACE 300000: (thread=200001)
        <empty>
TRACE 300001: (thread=200001)
        java.lang.Thread.<init>(Thread.java:493)
TRACE 300002: (thread=200001)
        java.lang.Thread.<init>(Thread.java:206)
...
SITES BEGIN (ordered by live bytes) Tue Mar 25 15:30:45 2014
          percent          live          alloc'ed  stack class
 rank   self  accum     bytes objs     bytes  objs trace name
    1 35.41% 35.41%  99614736    1  99614736     1 317704 byte[]
    2 16.79% 52.20%  47232000 2880 125952000  7680 318394 char[]
...
SITES END
CPU SAMPLES BEGIN (total = 6089) Tue Mar 25 15:30:45 2014
rank   self  accum   count trace method
...
 859  0.02% 99.57%       1 319489 java.lang.System.arraycopy
 860  0.02% 99.59%       1 319490 java.lang.Object.<init>
CPU SAMPLES END

After that miracle starts:

  1. attempt_201402231438_11879_m_000000_0.profile contains only standard header (no profile information). I'm not concerned about this file much as my primary interest is reducer, but this does not feel right
  2. attempt_201402231438_11879_r_000000_0.profile and attempt_201402231438_11879_r_000001_0.profile contain thread and trace information, but does not include CPU SAMPLES and SITES information.

Job completed successfully.

Any idea what could go wrong?

Why I'm not getting CPU SAMPLES for reducers?

Did anyone experience the same problem?

Yevgen Yampolskiy
  • 7,022
  • 3
  • 26
  • 23
  • Did you figure something out finally? I have a similar problem but in my case I debug just mappers and there is no CPU SAMPLES anywhere! For more info look http://stackoverflow.com/questions/25983999/hadoop-hprof-profiling-no-cpu-samples-written – nikicc Sep 22 '14 at 22:41
  • I do not think so, I would update the post – Yevgen Yampolskiy Sep 23 '14 at 19:57

0 Answers0