2

I'd like to trace functions of the particular PID and collect some stats (total calls, total times, etc.), and it's not completely clear for me how to create BPF_HASH with pairs of funcname+my_struct.

Is there any way to obtain names of called functions in BPF program?

I suppose I should read IP register using "PT_REGS_IP(ctx)" but I don't completely understand how translate the value to human-readable string.

At the moment BPF program looks in the following way:

#include <uapi/linux/ptrace.h>
#include <linux/sched.h>

struct data_t {
    u32 pid;
    u64 delta;
    u64 start;
} __attribute__((packed));

BPF_HASH(faddr, u64, struct data_t);
BPF_PERF_OUTPUT(events);

int do_entry(struct pt_regs *ctx) {
    struct data_t *data;
    data->start = bpf_ktime_get_ns();
    u64 ip = PT_REGS_IP(ctx);
    faddr.update(&ip, data);

    return 0;
}

int do_return(struct pt_regs *ctx) {
    struct data_t *data;
    u64 ip = PT_REGS_IP(ctx);
    data = faddr.lookup(&ip);

    if (data->start == 0)
        return 0;       // missed start

    data->delta = bpf_ktime_get_ns() - data->start;
    data->pid = bpf_get_current_pid_tgid();

    events.perf_submit(ctx, &data, sizeof(data));
    faddr.delete(&ip);

    return 0;
}

But at startup I got:

error: <unknown>:0:0: in function do_entry i32 (%struct.pt_regs*): A call to built-in function 'abort' is not supported.
pchaigno
  • 11,313
  • 2
  • 29
  • 54
lesovsky
  • 326
  • 2
  • 14
  • Could you please provide the commands you use to compile, then load and attach your program as well? – Qeole Nov 19 '18 at 17:26
  • Sorry for incompleteness. The full code examples can be found in my previous [question](https://stackoverflow.com/questions/53324158/golang-ebpf-and-functions-duration), there example of traced C program and example of main Go program. C program should be compiled with debug symbols (gcc -g), go program compiled with defaults (go build) – lesovsky Nov 20 '18 at 05:20
  • 1
    It's best if you can provide a minimal, reproducible example program in each question, even if that means having some duplicate content between questions. The userspace code is necessary to run your BPF program and someone who wants to reproduce may not have read your previous question. – pchaigno Nov 20 '18 at 16:11
  • ok, will do in next questions. – lesovsky Nov 21 '18 at 05:25

1 Answers1

3

You have an error in your do_entry function. You're trying to dereference a null pointer:

struct data_t *data;
data->start = bpf_ktime_get_ns();

The following should work better:

int do_entry(struct pt_regs *ctx) {
    struct data_t data = {}; // initializes data with zeros.
    data.start = bpf_ktime_get_ns();
    u64 ip = PT_REGS_IP(ctx);
    faddr.update(&ip, &data);
    return 0;
}

I don't understand why the error message mentions abort though. I'll ask around.


How to translate memory addresses to function names will depend on the userspace library you're using. If you're using bcc, there's a ksym method you can use. I don't know if there's an equivalent in gobpf.


You have at least one other error, in do_return:

data = faddr.lookup(&ip);
if (data->start == 0)
    return 0;       // missed start

You will need to check data is not null before dereferencing it. Otherwise, the verifier will reject your program.

data = faddr.lookup(&ip);
if (!data || data->start == 0)
    return 0;       // missed start
pchaigno
  • 11,313
  • 2
  • 29
  • 54
  • I've replaced pointer to variable and fix mistake in do_return(), but got something strange that I haven't seen before: `$ go build timings.go $ sudo ./timings bpf: Failed to load program: Permission denied 0: (bf) r6 = r1 1: (79) r1 = *(u64 *)(r6 +128) 2: (7b) *(u64 *)(r10 -16) = r1 3: (18) r1 = 0xffff8fcad7999800 ... a lot of similar messages here ... 96: (b7) r5 = 8 97: (85) call bpf_perf_event_output#25 invalid indirect read from stack off -8+0 size 8 Failed to load do_return: error loading BPF program: permission denied` – lesovsky Nov 20 '18 at 05:19
  • error affected by `ameba_events.perf_submit(ctx, &data, sizeof(data));` – lesovsky Nov 20 '18 at 05:31
  • changing it to `ameba_events.perf_submit(ctx, data, sizeof(data));` fixes it. – lesovsky Nov 20 '18 at 05:38
  • anyway, reading through PT_REGS_IP(ctx) in do_entry and do_return, returns different values and I can't use these values as key in BPF_HASH. Maybe is it possible to translate address into a function name inside the BPF program and use string as hash key? – lesovsky Nov 20 '18 at 06:06
  • BPF has little support for string manipulation, so I wouldn't recommend using function names as hash key, even if you found a way to translate addresses. However, since `do_entry` and `do_return` trace the same function, you could transmit the IP value from `do_entry` to `do_return` with a per-cpu hash map indexed on the current PID (it's common practice in BPF tracing programs). I'd rather not detail this here (if you need details) as the answer is already pretty bloated. – pchaigno Nov 20 '18 at 16:10
  • it's just my first steps with eBPF. originally I'd like to trace a set of several functions of single pid, hence i'm not sure that current PID is suitable (because its value always be the same). Moreover I'd like to store some stats between function calls, so in such case I suppose a map with funcnames as keys is preferred. But if you have more right ways, please let me know. – lesovsky Nov 21 '18 at 05:25
  • 1
    I'm not sure I see the issue with having several functions to trace. As long as you're sure the `do_return` is called right after the `do_entry`, you can retrieve the IP value from the map without errors. Maybe we can discuss this with examples in a chat (IRC?) or in another SO question? – pchaigno Nov 21 '18 at 09:17
  • It would be great, i've found your contacts and if you don't mind I will email you and will try to explain what I trying to "invent" ))) – lesovsky Nov 21 '18 at 19:31