3

On loading and running a kernel module and then profiling through perf.

$ perf record -a -g --call-graph dwarf sleep 30
$ perf report

my kernel module's symbols are not present in the perf's report. Although the symbols are present in /proc/kallsyms. Also the module is not present in perf buildid-list As this answer says to make the module a kernel module, I tried but didn't help. What are the possible reasons that could lead to this?

Mateusz Piotrowski
  • 8,029
  • 10
  • 53
  • 79
Saty Anand
  • 480
  • 3
  • 11
  • Does you module compute something for long amount of time or call something which compute long? If there were no profiling sample into the module function or in some function which was resolved as called through your module functions, module will not show anything in perf report. – osgx Jun 02 '17 at 16:50
  • My function does run for long enough. How do I know so? Because I ran same function from a userspace program and saw it's symbols in the profiling report with some significant percentage of CPU time. I'm wondering is there some way wherein perf would not be having access to my external kernel module's symbols? – Saty Anand Jun 03 '17 at 08:09
  • Does the following warning when getting profiling report causing the problem? `no symbols found in /sbin/dhclient, maybe install a debug package? no symbols found in /bin/kmod, maybe install a debug package? Failed to open [thrUserCtrl], continuing without symbols no symbols found in /usr/sbin/dnsmasq, maybe install a debug package? ` – Saty Anand Jun 03 '17 at 09:04
  • Saty, So, there are long functions in the module itself? There are messages for every dso without debug symbols, most are user-space programs, but is 'thrUserCtrl' your module? Perf was unable to find it, is it installed into `/lib/modules/\`uname -r\`/extra` (https://wiki.centos.org/HowTos/BuildingKernelModules)? What is your kernel version (`uname -a`?) – osgx Jun 03 '17 at 11:43
  • To answer your questions: Yes, there are long functions in the module itself. Yes 'thrUserCtrl' is my module. No, I haven't installed I `in /lib/modules/`uname -r`/extra `. Kernel version is - 3.13.0-32-generic. – Saty Anand Jun 03 '17 at 12:26

1 Answers1

2

The message Failed to open [thrUserCtrl], continuing without symbols sounds like perf was unable to find your module. Try installing it into 

/lib/modules/`uname -r`/extra

directory as said in https://wiki.centos.org/HowTos/BuildingKernelModules:

6. In this example, the file cifs.ko has just been created. 
 As root, copy the .ko file to the /lib/modules/<kernel-version>/extra/
 directory.

   [root@host linux-2.6.18.i686]# cp fs/cifs/cifs.ko /lib/modules/`uname -r`/extra

(don't forget depmod -a command after changing files in /lib/modules)

This message is generated in map__load: http://elixir.free-electrons.com/linux/v4.11/source/tools/perf/util/map.c#L284

int map__load(struct map *map)
{
    const char *name = map->dso->long_name;
    int nr;
    ...
    nr = dso__load(map->dso, map);
    if (nr < 0) {
        if (map->dso->has_build_id) {
         ...
        } else
            pr_warning("Failed to open %s", name);

        pr_warning(", continuing without symbols\n");
        return -1;

when dso__load function returns error.

osgx
  • 90,338
  • 53
  • 357
  • 513
  • Try to specify kallsyms file to `perf report` command with `--kallsyms=` option. – osgx Jun 03 '17 at 11:54
  • As it says to install the module in '/lib/modules/`uname -r`/extra' isn't it making the module a part of the kernel? Isn't it possible for perf to profile a module loaded using insmod by specifying the path of my module's .ko file or something similar? – Saty Anand Jun 03 '17 at 12:33
  • I copied the module to `'/lib/modules/uname -r/extra' ` and the modprobe to load it. Now I could see my module and it's top level functions(the functions that are being passed to threads), but still the other function's(functions being called by the thread function) symbols are absent in perf report. – Saty Anand Jun 03 '17 at 13:03
  • 1
    Saty, inner functions of your module may be inlined into top function (recompile with disabled inlining `-fno-inline` or no optimizations `-O0` to see them all). Now you see the functions of your module and problem solved. No, copying into extra is not make module part of the kernel. Don't know yet how perf searches the module, but probably there is no full path recorded anywhere by `insmod`, or it may work in newer kernel (3.13 have less module support in dso__load, only `DSO_BINARY_TYPE__SYSTEM_PATH_KMODULE` in http://elixir.free-electrons.com/linux/v3.13/source/tools/perf/util/dso.h#L24). – osgx Jun 03 '17 at 13:14