2

my docker container base image is ubuntu, and I ran it with full privilege options that means in run command I use these switches:

--cap-add=SYS_ADMIN --security-opt apparmor:unconfined

I wanna use blktrce using below command:

sudo blktrace -d / -a issue -o - | blkparse -f "%p %T.%9t %D %S ^C %d\n" -i - >stream.out

but, first time when is use this command I get this error:

Debugfs is not mounted at /sys/kernel/debug

i searched and found this solution which led to use this command:

mount -t debugfs none /sys/kernel/debug

after that, when I use the blktrace command again, i get this error:

BLKTRACESETUP(2) / failed: 25/Inappropriate ioctl for device
Thread 1 failed open /sys/kernel/debug/block/(null)/trace1: 2/No such file or directory
Thread 0 failed open /sys/kernel/debug/block/(null)/trace0: 2/No such file or directory
Thread 2 failed open /sys/kernel/debug/block/(null)/trace2: 2/No such file or directory
Thread 4 failed open /sys/kernel/debug/block/(null)/trace4: 2/No such file or directory
Thread 3 failed open /sys/kernel/debug/block/(null)/trace3: 2/No such file or directory
Thread 5 failed open /sys/kernel/debug/block/(null)/trace5: 2/No such file or directory
Thread 7 failed open /sys/kernel/debug/block/(null)/trace7: 2/No such file or directory
Thread 8 failed open /sys/kernel/debug/block/(null)/trace8: 2/No such file or directory
Thread 6 failed open /sys/kernel/debug/block/(null)/trace6: 2/No such file or directory
Thread 12 failed open /sys/kernel/debug/block/(null)/trace12: 2/No such file or directory
Thread 10 failed open /sys/kernel/debug/block/(null)/trace10: 2/No such file or directory
Thread 13 failed open /sys/kernel/debug/block/(null)/trace13: 2/No such file or directory
Thread 15 failed open /sys/kernel/debug/block/(null)/trace15: 2/No such file or directory
Thread 14 failed open /sys/kernel/debug/block/(null)/trace14: 2/No such file or directory
Thread 17 failed open /sys/kernel/debug/block/(null)/trace17: 2/No such file or directory
Thread 16 failed open /sys/kernel/debug/block/(null)/trace16: 2/No such file or directory
Thread 18 failed open /sys/kernel/debug/block/(null)/trace18: 2/No such file or directory
Thread 11 failed open /sys/kernel/debug/block/(null)/trace11: 2/No such file or directory
Thread 19 failed open /sys/kernel/debug/block/(null)/trace19: 2/No such file or directory
Thread 20 failed open /sys/kernel/debug/block/(null)/trace20: 2/No such file or directory
Thread 9 failed open /sys/kernel/debug/block/(null)/trace9: 2/No such file or directory
Thread 21 failed open /sys/kernel/debug/block/(null)/trace21: 2/No such file or directory
Thread 22 failed open /sys/kernel/debug/block/(null)/trace22: 2/No such file or directory
Thread 23 failed open /sys/kernel/debug/block/(null)/trace23: 2/No such file or directory
FAILED to start thread on CPU 0: 1/Operation not permitted
FAILED to start thread on CPU 1: 1/Operation not permitted
FAILED to start thread on CPU 2: 1/Operation not permitted
FAILED to start thread on CPU 3: 1/Operation not permitted
FAILED to start thread on CPU 4: 1/Operation not permitted
FAILED to start thread on CPU 5: 1/Operation not permitted
FAILED to start thread on CPU 6: 1/Operation not permitted
FAILED to start thread on CPU 7: 1/Operation not permitted
FAILED to start thread on CPU 8: 1/Operation not permitted
FAILED to start thread on CPU 9: 1/Operation not permitted
FAILED to start thread on CPU 10: 1/Operation not permitted
FAILED to start thread on CPU 11: 1/Operation not permitted
FAILED to start thread on CPU 12: 1/Operation not permitted
FAILED to start thread on CPU 13: 1/Operation not permitted
FAILED to start thread on CPU 14: 1/Operation not permitted
FAILED to start thread on CPU 15: 1/Operation not permitted
FAILED to start thread on CPU 16: 1/Operation not permitted
FAILED to start thread on CPU 17: 1/Operation not permitted
FAILED to start thread on CPU 18: 1/Operation not permitted
FAILED to start thread on CPU 19: 1/Operation not permitted
FAILED to start thread on CPU 20: 1/Operation not permitted
FAILED to start thread on CPU 21: 1/Operation not permitted
FAILED to start thread on CPU 22: 1/Operation not permitted
FAILED to start thread on CPU 23: 1/Operation not permitted

how can i solve that?

Update1:

there is sda folder in /sys/kernel/debug/block/ and into this folder there are these files:

trace0 trace1 trace2 etc.

Update2:

@abligh thank for your answer, but did not help. The strace output about ioctl is:

ioctl(3, BLKTRACESETUP, {act_mask=64, buf_size=524288, buf_nr=4, start_lba=0, end_lba=0, pid=0}, 0x7ffe8a4ceac0) = -1 ENOTTY (Inappropriate ioctl for device)
write(2, "BLKTRACESETUP(2) / failed: 25/Inappropriate ioctl for device\n", 61) = 61
ioctl(3, BLKTRACESTOP, 0x7f6fd19789d0) = -1 ENOTTY (Inappropriate ioctl for device)
ioctl(3, BLKTRACESTOP, 0x7ffe8a4ce540) = -1 ENOTTY (Inappropriate ioctl for device)
ioctl(3, BLKTRACESTOP, 0x7f6fd19789e0) = -1 ENOTTY (Inappropriate ioctl for device)`
ioctl(3, BLKTRACETEARDOWN, 0x7f6fd19789e0) = -1 ENOTTY (Inappropriate ioctl for device)

to answer your question about why i run blktrace in container, i would say I'm using containers as cluster, so i need the trace of each node.

abligh
  • 24,573
  • 4
  • 47
  • 84
mR.aTA
  • 174
  • 2
  • 19
  • Does `blktrace` work from outside the container? I'm guessing the issue might be that the block devices themselves are not visible from the container. – abligh May 15 '16 at 19:14
  • @abligh yes, it's ok with host. – mR.aTA May 16 '16 at 06:46
  • then it isn't a kernel lock or similar. I suspect your issue is simply that `BLKTRACESETUP` is not working in a container. This may be a kernel limitation. An `strace` of `blktrace` would be helpful. – abligh May 16 '16 at 08:19
  • @abligh i think there is no kernel limitation, cuz i run docker container with `--cap-add SYS_ADMIN` and `--security-opt apparmor:unconfined`. sorry did not get your advise, what's an `state` of `blktrace`? – mR.aTA May 16 '16 at 09:23
  • @abligh does my new update on question help? – mR.aTA May 16 '16 at 09:27
  • did you mount sysfs in your container before you mounted debugfs? You should ensure that both `/sys` and `/proc` are mounted first. – abligh May 16 '16 at 09:40
  • @abligh No, I did not. how can i do that? – mR.aTA May 16 '16 at 09:57
  • something like `mount -t sysfs sysfs /sys ; mount -t proc none /proc` before the debugfs mount. – abligh May 16 '16 at 09:59
  • @abligh they were mounted already! problem did not solved yet. – mR.aTA May 16 '16 at 10:08
  • Are the block devices you are trying to trace present in `/sys/block/` in the container? You might add an `strace` of the `blktrace` to the question (i.e. `sudo strace -f -o/tmp/trace blktrace ...` then post `/tmp/trace` somewhere). – abligh May 16 '16 at 10:54
  • @abligh in `sys/block/` there are these: `loop0 loop1 loop2 loop3 loop4 loop5 loop6 loop7 ram0 ram1 ram10 ram11 ram12 ram13 ram14 ram15 ram2 ram3 ram4 ram5 ram6 ram7 ram8 ram9 sda sr0` would you please post a clear answer? – mR.aTA May 16 '16 at 12:02

1 Answers1

1

I don't have enough information to debug this, but you asked for an answer to be posted.

The root of the problem seems to be here:

BLKTRACESETUP(2) / failed: 25/Inappropriate ioctl for device

This is indicating that the ioctl call to set up block tracing is failing. It is failing inside the container, but according to the comments works outside the container. This would indicate that the problem is with the container setup or a limitation in the kernel that prevents that ioctl from being used in a container at all.

The error inappropriate ioctl for device is errno 25, i.e. ENOTTY. I can't immediately see what would be returning that unless it can't find the device node at all (given you've already demonstrated from the fact it works outside the container that the block trace code is compiled in). I can't remember whether this is in a module, but it would be worth trying block tracing outside the container first (then check it inside the container), just to check this isn't a module loading issue.

The first step in debugging this would be to use strace tool (as suggested above) so you know exactly which system call is being made with what parameters. EG run:

sudo strace -f -s2048 -o/tmp/trace blktrace -d / -a issue -o - | blkparse -f "%p %T.%9t %D %S ^C %d\n" -i - >stream.out

and look at /tmp/trace afterwards. strace will list all the system calls made. See if you can determine from that which ioctl is failing.

Secondly, I'd ensure that the block device that you are trying to trace actually appears within the correct place(s) in /proc/ and /sys/. Something wrong is happening here:

Thread 0 failed open /sys/kernel/debug/block/(null)/trace0: 2/No such file or directory

Note the (null) in the debug line, which clearly should not be there - that should be the name of the block device. This is possibly a consequence of the failed ioctl, or possibly indicative of a problem within the /sys/ hierarchy.

BLKTRACESETUP is handled in the kernel here. This eventually calls doblktracesetup here. I cannot immediately see any reason why this would not work from an appropriately permissioned container, which makes me suspect your /sys hierarchy might not be set up right. But the output of strace would be helpful.

Also, the inevitable comment: why are you doing this in a container? Why not run it outside the container?

EDIT: Looks like it might be a kernel thing. See https://github.com/scaleway/kernel-tools/issues/107 - this suggests you (a) need to modprobe the specific modules first, and (b) may need a specific kernel.

abligh
  • 24,573
  • 4
  • 47
  • 84