Fixing 'no symbolic type information' from dtrace in Linux?

Question

Just documenting this: (self-answer to follow)

I'm aware that Sun's dtrace is not packaged for Ubuntu due to licensing issues; so I downloaded it and built it from source on Ubuntu - but I'm having an issue pretty much like the one in Simple dtraces not working · Issue #17 · dtrace4linux/linux · GitHub; namely loading of the driver seems fine:

dtrace-20130712$ sudo make load
tools/load.pl
23:20:31 Syncing...
23:20:31 Loading: build-2.6.38-16-generic/driver/dtracedrv.ko
23:20:34 Preparing symbols...
23:20:34 Probes available: 364377
23:20:44 Time: 13s

... however, if I try to run a simple script, it fails:

$ sudo ./build/dtrace -n 'BEGIN { printf("Hello, world"); exit(0); }'
dtrace: invalid probe specifier BEGIN { printf("Hello, world"); exit(0); }: "/path/to/src/dtrace-20130712/etc/sched.d", line 60: no symbolic type information is available for kernel`dtrace_cpu_id: Invalid argument

As per the issue link above:

(ctf requires a private and working libdwarf lib - most older releases have broken versions).

... I then built libdwarf from source, and then dtrace based on it (not trivial, requires manually finding the right placement of symlinks); and I still get the same failure.

Is it possible to fix this?

score 1 · Answer 1 · answered Jul 27 '13 at 21:49

Well, after a trip to gdb, I figured that the problem occurs in dtrace's function dt_module_getctf (called via dtrace_symbol_type and, I think, dt_module_lookup_by_name). In it, I noticed that most calls propagate the attribute/variable dm_name = "linux"; but when the failure occurs, I'd get dm_name = "kernel"!

Note that original line 60 from sched.d is:

    cpu_id = `dtrace_cpu_id; /* C->cpu_id; */

Then I found thr3ads.net - dtrace discuss - accessing symbols without type info [Nov 2006]; where this error message is mentioned:

dtrace: invalid probe specifier fbt::calcloadavg:entry { printf("CMS_USER: %d, CMS_SYSTEM: %d, cpu_waitrq: %d\n", `cpu0.cpu_acct[0], `cpu0.cpu_acct[1], `cpu0.cpu_waitrq);}: in action list: no symbolic type information is available for unix`cpu0: No type information available for symbol

So:

on that system, the request `cpu0.cpu_acct[0] got resolved to unix`cpu0;
and on my system, the request `dtrace_cpu_id got resolved to kernel`dtrace_cpu_id.

And since "The backtick operator is used to read the value of kernel variables, which will be specific to the running kernel." (howto measure CPU load - DTrace General Discussion - ArchiveOrange), I thought maybe explicitly "casting" this "backtick variable" to linux would help.

And indeed it does - only a small section of sched.d needs to be changed to this:

translator cpuinfo_t < dtrace_cpu_t *C > {
    cpu_id = linux`dtrace_cpu_id; /* C->cpu_id; */
    cpu_pset = -1;
    cpu_chip = linux`dtrace_cpu_id; /* C->cpu_id; */
    cpu_lgrp = 0; /* XXX */
/*  cpu_info = *((_processor_info_t *)`dtrace_zero); /* ` */ /* XXX */
};

inline cpuinfo_t *curcpu = xlate <cpuinfo_t *> (&linux`dtrace_curcpu);

... and suddenly - it starts working!:

dtrace-20130712$ sudo ./build/dtrace -n 'BEGIN { printf("Hello, world"); exit(0); }'
dtrace: description 'BEGIN ' matched 1 probe
CPU     ID                    FUNCTION:NAME
  1      1                           :BEGIN Hello, world

PS:

Protip 1: NEVER do dtrace -n '::: { printf("Hello"); }' - this means "do a printf on each and every kernel event", and it will completely freeze the kernel; not even CTRL-Alt-Del will work!

Protip 2: If you want to use DTRACE_DEBUG as in Debugging DTrace, use sudo -E:

dtrace-20130712$ DTRACE_DEBUG=1 sudo -E ./build/dtrace -n 'BEGIN { printf("Hello, world"); exit(0); }'
libdtrace DEBUG: reading kernel .ctf: /path/to/src/dtrace-20130712/build-2.6.38-16-generic/linux-2.6.38-16-generic.ctf
libdtrace DEBUG: opened 32-bit /proc/kallsyms (syms=75761)
...

Fixing 'no symbolic type information' from dtrace in Linux?

1 Answers1