Ubuntu 16.04 LTS - How to enable symbols for the perf tool

Question

I'm trying to gather some profiling data for my app and I run the perf tool and Flame Graphs for that.

I'm referring the instructions provided in this slideshare: https://www.slideshare.net/brendangregg/java-performance-analysis-on-linux-with-flame-graphs

Below are the commands that I'm running:

1. sudo perf record -F 997 -a -g
2. sudo perf script > out.stacks01

When I run the second command, it displays below messages:

Failed to open /tmp/perf-9931.map, continuing without symbols.
no symbols found in <some path>, maybe install a debug package?

I did some online browsing and tried installing the debug packages as as mentioned here: https://gist.github.com/NLKNguyen/2fd920e2a50fd4b9701f

However, when I run "sudo apt-get update", it eventually fails saying "Failed to fetch......"

Could anybody figure out what's going on here? What do I need to do in order to install the debug symbols package correctly?

EDIT: My key problem was that the flame graph I was generating didn't have Java symbols in it and hence I ended up focusing on above errors/messages. The below accepted answer provides a very good explanation for my original post. However, I was able to resolve my issue by running jmaps as shown below:

sudo perf record -F 997 -a -g -- sleep 300; jmaps 

This is documented in instructions in the slide share link shared above.

Answer 1

Failed to open /tmp/perf-9931.map message is not about incorrect debuginfo - it is about profiling code which was generated by JIT (and Java usually generate machine code from class files with JIT), when there was no compatible with perf profiling agent running.

In http://www.brendangregg.com/perf.html#JIT_Symbols there is recommendation "Java can do this with perf-map-agent" to use https://github.com/jvm-profiling-tools/perf-map-agent which will generate map files for perf:

Architecture

Linux perf tools will expect symbols for code executed from unknown memory regions at /tmp/perf-.map. This allows runtimes that generate code on the fly to supply dynamic symbol mappings to be used with the perf suite of tools.

perf-map-agent is an agent that will generate such a mapping file for Java applications. It consists of a Java agent written C and a small Java bootstrap application which attaches the agent to a running Java process.

When the agent is attached it instructs the JVM to report code blobs generated by the JVM at runtime for various purposes. Most importantly, this includes JIT-compiled methods but also various dynamically-generated infrastructure parts like the dynamically created interpreter, adaptors, and jump tables for virtual dispatch (see vtable and itable entries). The agent creates a /tmp/perf-.map file which it fills with one line per code blob that maps a memory location to a code blob name.

The Java application takes the PID of a Java process as an argument and an arbitrary number of additional arguments which it passes to the agent. It then attaches to the target process and instructs it to load the agent library.

And in https://www.slideshare.net/brendangregg/java-performance-analysis-on-linux-with-flame-graphs Gregg used special hacked build of OpenJDK - slide 36 - "-XX:+PreserveFramePointer •  I hacked OpenJDK x86_64 to support frame pointers".

And from slide 41 Gregg talks about /tmp/perf-*.map files:

Fixing Symbols

 • For JIT'd code, Linux perf already looks for an externally provided symbol file: /tmp/perf-PID.map, and warns if it doesn't exist •  This file can be created by a Java agent

# perf script 
Failed to open /tmp/perf-8131.map, continuing without symbols

(also "See lkml for "perf: add support for profiling jitted code"" - https://lwn.net/Articles/633846/ and other)