I'm trying to profile reduce tasks on a cluster using
conf.setBoolean("mapred.task.profile", true);
conf.set("mapred.task.profile.params", "-agentlib:hprof=cpu=samples,heap=sites,depth=6,force=n,thread=y,verbose=n,file=%s");
conf.set("mapred.task.profile.maps", "0-1");
conf.set("mapred.task.profile.reduces", "0-1");
As expected, I've got 4 profile files generated:
2849 Mar 25 15:30 attempt_201402231438_11879_m_000000_0.profile
8786788 Mar 25 15:30 attempt_201402231438_11879_m_000001_0.profile
4129237 Mar 25 15:31 attempt_201402231438_11879_r_000000_0.profile
12644892 Mar 25 15:32 attempt_201402231438_11879_r_000001_0.profile
(files with _m_
are generated during mapper task profiling, and _r_
are generated during reduce task profiling)
attempt_201402231438_11879_m_000001_0.profile
looks good:
THREAD START (obj=50000161, id = 200001, name="main", group="main")
THREAD START (obj=500021bd, id = 200004, name="TGT Renewer for yampolsy", group="main")
THREAD END (id = 200004)
...
TRACE 300000: (thread=200001)
<empty>
TRACE 300001: (thread=200001)
java.lang.Thread.<init>(Thread.java:493)
TRACE 300002: (thread=200001)
java.lang.Thread.<init>(Thread.java:206)
...
SITES BEGIN (ordered by live bytes) Tue Mar 25 15:30:45 2014
percent live alloc'ed stack class
rank self accum bytes objs bytes objs trace name
1 35.41% 35.41% 99614736 1 99614736 1 317704 byte[]
2 16.79% 52.20% 47232000 2880 125952000 7680 318394 char[]
...
SITES END
CPU SAMPLES BEGIN (total = 6089) Tue Mar 25 15:30:45 2014
rank self accum count trace method
...
859 0.02% 99.57% 1 319489 java.lang.System.arraycopy
860 0.02% 99.59% 1 319490 java.lang.Object.<init>
CPU SAMPLES END
After that miracle starts:
attempt_201402231438_11879_m_000000_0.profile
contains only standard header (no profile information). I'm not concerned about this file much as my primary interest is reducer, but this does not feel rightattempt_201402231438_11879_r_000000_0.profile
andattempt_201402231438_11879_r_000001_0.profile
contain thread and trace information, but does not include CPU SAMPLES and SITES information.
Job completed successfully.
Any idea what could go wrong?
Why I'm not getting CPU SAMPLES for reducers?
Did anyone experience the same problem?