4

My goal is to get an ability to use userspace probes on Ubuntu 12.04/precise host. This is possible starting from 3.5.0 kernel, so I installed the following packages:

linux-headers-3.5.0-34
linux-headers-3.5.0-34-generic
linux-image-3.5.0-34-generic
linux-image-3.5.0-34-generic-dbgsym
linux-image-generic-lts-quantal
linux-source
linux-source-3.5.0

and systemtap v1.7 from 12.10/quantal.

But the only thing I see after starting any stap script is:

ERROR: module version mismatch (#55-Ubuntu SMP Thu Jun 6 20:18:19 UTC 2013 vs #55~precise1-Ubuntu SMP Fri Jun 7 16:25:50 UTC 2013), release 3.5.0-34-generic
ERROR: Build-id mismatch [man error::buildid]: "/usr/lib/debug/boot/vmlinux-3.5.0-34-generic" byte 0 (0x84 vs 0xc2) address 0xffffffff8168e60c rc 0

uname -a:

Linux nginx-dev01g 3.5.0-34-generic #55~precise1-Ubuntu SMP Fri Jun 7 16:25:50 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

systemtap v2.1 from 13.04/raring shows exactly the same error.

systemtap v1.6 (default version in 12.04) is failing at compile time:

In file included from /usr/share/systemtap/runtime/transport/transport.c:53:0,
             from /usr/share/systemtap/runtime/print.c:18,
             from /usr/share/systemtap/runtime/runtime.h:128,
             from /tmp/stapYh6Wqv/stap_c22f4bc3b03cd6f9fed549d2973f11fd_361755.c:59:
/usr/share/systemtap/runtime/transport/relay_v2.c:241:2: error: initialization from incompatible pointer type [-Werror]
/usr/share/systemtap/runtime/transport/relay_v2.c:241:2: error: (near initialization for ‘__stp_relay_callbacks.create_buf_file’) [-Werror]
In file included from /usr/share/systemtap/runtime/print.c:18:0,
             from /usr/share/systemtap/runtime/runtime.h:128,
             from /tmp/stapYh6Wqv/stap_c22f4bc3b03cd6f9fed549d2973f11fd_361755.c:59:
/usr/share/systemtap/runtime/transport/transport.c: In function ‘_stp_get_root_dir’:
/usr/share/systemtap/runtime/transport/transport.c:386:8: error: ‘struct hlist_head’ has no member named ‘next’
In file included from /tmp/stapYh6Wqv/stap_c22f4bc3b03cd6f9fed549d2973f11fd_361755.c:61:0:
/usr/share/systemtap/runtime/stat.c: In function ‘_stp_stat_get’:
/usr/share/systemtap/runtime/stat.c:213:2: error: ‘cpu_possible_map’ undeclared (first use in this function)
/usr/share/systemtap/runtime/stat.c:213:2: note: each undeclared identifier is reported only once for each function it appears in
/usr/share/systemtap/runtime/stat.c: In function ‘_stp_stat_clear’:
/usr/share/systemtap/runtime/stat.c:248:2: error: ‘cpu_possible_map’ undeclared (first use in this function)
In file included from /tmp/stapYh6Wqv/stap_c22f4bc3b03cd6f9fed549d2973f11fd_361755.c:13594:0:
/usr/share/systemtap/runtime/pmap-gen.c: In function ‘_stp_pmap_new_sx’:
/usr/share/systemtap/runtime/pmap-gen.c:717:3: error: ‘cpu_possible_map’ undeclared (first use in this function)
/usr/share/systemtap/runtime/pmap-gen.c: In function ‘_stp_pmap_get_sx’:
/usr/share/systemtap/runtime/pmap-gen.c:943:2: error: ‘cpu_possible_map’ undeclared (first use in this function)
In file included from /usr/share/systemtap/runtime/map.c:20:0,
             from /tmp/stapYh6Wqv/stap_c22f4bc3b03cd6f9fed549d2973f11fd_361755.c:13609:
/usr/share/systemtap/runtime/map-stat.c: In function ‘_stp_pmap_new_hstat_linear’:
/usr/share/systemtap/runtime/map-stat.c:71:3: error: ‘cpu_possible_map’ undeclared (first use in this function)
/usr/share/systemtap/runtime/map-stat.c: In function ‘_stp_pmap_new_hstat_log’:
/usr/share/systemtap/runtime/map-stat.c:98:3: error: ‘cpu_possible_map’ undeclared (first use in this function)
In file included from /tmp/stapYh6Wqv/stap_c22f4bc3b03cd6f9fed549d2973f11fd_361755.c:13609:0:
/usr/share/systemtap/runtime/map.c: In function ‘_stp_pmap_new’:
/usr/share/systemtap/runtime/map.c:242:2: error: ‘cpu_possible_map’ undeclared (first use in this function)
/usr/share/systemtap/runtime/map.c: In function ‘_stp_pmap_clear’:
/usr/share/systemtap/runtime/map.c:350:2: error: ‘cpu_possible_map’ undeclared (first use in this function)
/usr/share/systemtap/runtime/map.c: In function ‘_stp_pmap_del’:
/usr/share/systemtap/runtime/map.c:402:2: error: ‘cpu_possible_map’ undeclared (first use in this function)
/usr/share/systemtap/runtime/map.c: In function ‘_stp_pmap_agg’:
/usr/share/systemtap/runtime/map.c:732:2: error: ‘cpu_possible_map’ undeclared (first use in this function)
/usr/share/systemtap/runtime/map.c: In function ‘_stp_pmap_size’:
/usr/share/systemtap/runtime/map.c:908:2: error: ‘cpu_possible_map’ undeclared (first use in this function)
cc1: all warnings being treated as errors

Unfortunately, updating to 12.10/13.04 is not an option.

So, is there any chance to get working systemtap with 3.5.0 or 3.8.0 kernels on Ubuntu 12.04/precise?

Andrei Belov
  • 1,131
  • 9
  • 8

2 Answers2

2

You may need to update your version of systemtap, as a very old one (e.g. 1.7 is from early 2012) will often not work with a very new kernel (but vice versa is fine).

OTOH, the original build-id error is a real version-mismatch problem within the ubuntu -dbgsym repositories. You need to ensure that your running-kernel version exactly matches the dbgsym one. With Ubuntu this can be tricky, because "uname -r" and the dpkg names do not contain a uniquely identify the build. See the difference between:

#55-Ubuntu SMP Thu Jun 6 20:18:19 UTC 2013

and

#55~precise1-Ubuntu SMP Fri Jun 7 16:25:50 UTC 2013

Why do you need kernel debug symbols for userspace probes though? Please include your stap script and stap-report output. Try also the stap-prep script.

Another complicating factor is the possible lack of CONFIG_UPROBES in the Ubuntu kernel. See also http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=691167

fche
  • 2,641
  • 20
  • 28
1

I compiled systemstap release-2.3. It works on Ubuntu 12.04 with linux-image-3.5.0-40-generic-dbgsym.

Install the kernel with debug info. Where to get debug symbols for kernel X?

Compile systemstap 2.3

sudo apt-get build-dep systemtap
git clone git://sourceware.org/git/systemtap.git
cd systemtap
git checkout release-2.3
./configure
make all # you can ignore the xmlto error

# enjoy it
sudo ./stap testsuite/systemtap.examples/network/tcpdumplike.stp
wenjianhn
  • 438
  • 3
  • 7