Running SystemTap user-space probes inside a container

Question

I am learning SystemTap and I have created a simple C program to grasp the basics. When I run the program and a probe in the hosting system, the probe works flawlessly, but when I copy the exact same process in a container I run into some problems (container is running in privileged mode).

When I leave both the program and probe running for a few seconds a WARNING message appears and this is what the stap output looks like after I stop the program:

[root@client ~]# stap -v tmp-probe.stp /root/tmp
Pass 1: parsed user script and 482 library scripts using 115544virt/94804res/16320shr/78424data kb, in 140usr/20sys/168real ms.
Pass 2: analyzed script: 2 probes, 1 function, 0 embeds, 1 global using 116996virt/97852res/17612shr/79876data kb, in 10usr/0sys/6real ms.
Pass 3: using cached /root/.systemtap/cache/73/stap_7357b5a96975af17d2210a04bada9b6a_1260.c
Pass 4: using cached /root/.systemtap/cache/73/stap_7357b5a96975af17d2210a04bada9b6a_1260.ko
Pass 5: starting run.
WARNING: probe process("/root/tmp").statement(0x40113a) at inode-offset 12761464:0000000060a7c1a2 registration error [man warning::pass5] (rc -5)
0
Pass 5: run completed in 0usr/70sys/3579real ms.

The binary has been compiled with: gcc -ggdb3 -O0 tmp.c -o tmp and contains these probe points:

process("/root/tmp").begin $syscall:long $arg1:long $arg2:long $arg3:long $arg4:long $arg5:long $arg6:long
process("/root/tmp").end $syscall:long $arg1:long $arg2:long $arg3:long $arg4:long $arg5:long $arg6:long
process("/root/tmp").plt("puts")
process("/root/tmp").plt("sleep")
process("/root/tmp").syscall $syscall:long $arg1:long $arg2:long $arg3:long $arg4:long $arg5:long $arg6:long
process("/root/tmp").mark("in_test")
process("/root/tmp").function("main@/root/tmp.c:11")
process("/root/tmp").function("test@/root/tmp.c:5")

tmp.c:

#include <stdio.h>
#include <sys/sdt.h>
#include <unistd.h>

void test() {
    STAP_PROBE(test, in_test);
    printf("here\n");
    sleep(1);
}

int main() {
    while (1) {
        test();
    }
}

tmp-probe.stp:

global cnt

probe process(@1).mark("in_test") {
    cnt++
}

probe process(@1).end {
    printf("%ld\n", cnt)
    exit()
}

A more verbose stap output:

[root@client ~]# stap -vv tmp-probe.stp /root/tmp
Systemtap translator/driver (version 4.6/0.186, rpm 4.6-4.fc35)
Copyright (C) 2005-2021 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
tested kernel versions: 2.6.32 ... 5.15.0-rc7
enabled features: AVAHI BOOST_STRING_REF DYNINST BPF JAVA PYTHON3 LIBRPM LIBSQLITE3 LIBVIRT LIBXML2 NLS NSS READLINE MONITOR_LIBS
Created temporary directory "/tmp/stapRsSKyO"
Session arch: x86_64 release: 5.16.20-200.fc35.x86_64
Build tree: "/lib/modules/5.16.20-200.fc35.x86_64/build"
Searched for library macro files: "/usr/share/systemtap/tapset/linux", found: 7, processed: 7
Searched for library macro files: "/usr/share/systemtap/tapset", found: 11, processed: 11
Searched: "/usr/share/systemtap/tapset/linux/x86_64", found: 20, processed: 20
Searched: "/usr/share/systemtap/tapset/linux", found: 407, processed: 407
Searched: "/usr/share/systemtap/tapset/x86_64", found: 1, processed: 1
Searched: "/usr/share/systemtap/tapset", found: 36, processed: 36
Pass 1: parsed user script and 482 library scripts using 115580virt/95008res/16520shr/78460data kb, in 130usr/30sys/163real ms.
derive-probes (location #0): process("/root/tmp").mark("in_test") of keyword at tmp-probe.stp:3:1
derive-probes (location #0): process("/root/tmp").end of keyword at tmp-probe.stp:7:1
Pass 2: analyzed script: 2 probes, 1 function, 0 embeds, 1 global using 117032virt/97860res/17624shr/79912data kb, in 10usr/0sys/6real ms.
Pass 3: using cached /root/.systemtap/cache/73/stap_7357b5a96975af17d2210a04bada9b6a_1260.c
Pass 4: using cached /root/.systemtap/cache/73/stap_7357b5a96975af17d2210a04bada9b6a_1260.ko
Pass 5: starting run.
Running /usr/bin/staprun -v -R /tmp/stapRsSKyO/stap_7357b5a96975af17d2210a04bada9b6a_1260.ko
staprun:insert_module:191 Module stap_7357b5a96975af17d2210a04bada9b_698092 inserted from file /tmp/stapRsSKyO/stap_7357b5a96975af17d2210a04bada9b6a_1260.ko
WARNING: probe process("/root/tmp").statement(0x40113a) at inode-offset 12761464:0000000060a7c1a2 registration error [man warning::pass5] (rc -5)
0
stapio:cleanup_and_exit:352 detach=0
stapio:cleanup_and_exit:369 closing control channel
staprun:remove_module:292 Module stap_7357b5a96975af17d2210a04bada9b_698092 removed.
Spawn waitpid result (0x0): 0
Pass 5: run completed in 10usr/70sys/2428real ms.
Running rm -rf /tmp/stapRsSKyO
Spawn waitpid result (0x0): 0
Removed temporary directory "/tmp/stapRsSKyO"

Running SystemTap user-space probes inside a container

0 Answers0