1

I run FreeBSD-based kernel as QEMU/KVM guest.

I'm working on a FreeBSD-based OS kernel SCSI driver and have an issue with read system call produces corrupted data.

To troubleshoot the problem I use the Kernel running in QEMU and would like to trace memory access performed by the DMA controller which is responsible for delivering data into the user-supplied buffer. In case of QEMU The controller is QEMU SCSI/ATA Disk device. So I tried to set a watchpoint on a user supplied buffer

Example:

Setting the breakpoint on int sys_read(struct thread *td, struct read_args *uap) I got some buffer arrived from the user:

(gdb) p uap->buf
 $5 = 0x7ffd4593f000 "User buffer initial data"
(gdb) watch *0x7ffd4593f000
 Hardware watchpoint 7: *0x7ffd4593f000
(gdb) c

The problem with that is the watchpoint is never hit. Why? I'd like to use it to understand examine data that are being transferred from the device into the memory.

Is it ever possible to watch access from the DMA controller?

UPD:

I managed to hit the watchpoint. It looks as follows:

Thread 3 hit Hardware watchpoint 7: *0x7ffd4593f000                                                                                                                                                                
                                                                                                                                                                                                                   
Old value = 1750343715                                                                                                                                                                                             
New value = 1819043144                                                                                                                                                                                             
0x00005586ec68f146 in ?? ()                                                                                                                                                                                        
(gdb) bt                                                                                                                                                                                                           
#0  0x00005586ec68f146 in ?? ()
#1  0x654b206f6c6c6548 in ?? ()
#2  0x6f7266206c656e72 in ?? ()
#3  0x707372657375206d in ?? ()
#4  0x6574617200656361 in ?? ()
#5  0x6d69562079622064 in ?? ()
#6  0x20230a2e312e3820 in ?? ()
#7  0x2079616d20756f59 in ?? ()
#8  0x2074692074696465 in ?? ()
#9  0x7227756f79206669 in ?? ()
...
#509 0x0a632e6e69616d2f in ?? ()
#510 0x32312c30352c347c in ?? ()
#511 0x323436312c37312c in ?? ()
#512 0x222c323037343132 in ?? ()
#513 0x0000000000000000 in ?? ()

It's highly likely to be correct, since the first 4 bytes shown as Old value and New value matches to what I expect to be read.

But the strange thing is that it is hit only once upon QEMU startup. Subsequent read system calls does not trigger the watchpoint. In order for it to be hit I restart QEMU and set it again.

What may this stacktrace mean?

Some Name
  • 8,555
  • 5
  • 27
  • 77
  • 1
    I think the hardware mechanism for watchpoints uses virtual addresses, and I'm pretty sure it's based on the debug registers (DR) *in a specific core*. For both reasons, it's only plausible for it to trigger on accesses done by that logical core, not by DMA by a device. – Peter Cordes Jan 28 '22 at 06:12
  • Or are you debugging the qemu process on the host? If you're debugging inside the guest, I think it's probably still unlikely, although the watchpoint mechanism is them emulated or virtualized depending on how you're running qemu... – Peter Cordes Jan 28 '22 at 06:14
  • @PeterCordes Yeah. I'm debugging QEMU process from the host attaching to the GDB server running inside. My expectation on why hardware watchpoint should work in case of DMA access was that CPU has to maintain caches in the coherent state and therefore knows about all DMA devices memory access via bus snooping (as specified in the Intel Manual). – Some Name Jan 28 '22 at 06:17
  • @PeterCordes The watchpoint is sometimes hit anyway, but I still hasn't managed to make it being hit consistently. – Some Name Jan 28 '22 at 06:28
  • Oh, well your question didn't say so, and wasn't even tagged [qemu]! Are you using qemu with hardware virtualization (KVM), or is it fully emulating the guest machine? (Speaking of emulating, Bochs has a built-in debugger that might be worth looking into; it aims at providing full-system debugging). – Peter Cordes Jan 28 '22 at 06:29
  • 1
    Re: coherency via snooping (or actually directory-based coherency, with L3 tags or a dedicated structure as a snoop filter): that only gives you physical addresses. Can you watch a physical address in qemu? Like I said, there might not be any mechanism to have the CPU take a debug exception based on anything other than virtual address load/store by the current core. I'd have to check the docs on how x86 DR0..7 work. – Peter Cordes Jan 28 '22 at 06:33
  • @PeterCordes Thanks for pointing the direction related to DRs. – Some Name Jan 28 '22 at 06:34

1 Answers1

4

The watchpoint handling provided for QEMU's gdbstub is really only intended to work with accesses done by the guest CPU, not for those done by DMA from emulated devices. I'm surprised it hits at all, and not surprised that the behaviour is a bit weird.

If you can repro this on a QEMU setup using purely emulation (ie not KVM, hvf or whpx), then my suggestion for debugging this would be to run QEMU itself in a host gdb, and use the host gdb to set a watchpoint on the host memory that corresponds to the relevant bit of guest memory. Unfortunately that requires some knowledge of QEMU internals to find the host memory, and generally to understand what's going on and relate what QEMU is doing to what the guest execution is.

Supplementary debugging tip: if you can take a 'snapshot' just before the bug is triggered, that gives you a shorter reproduce case which is "load from snapshot and trigger bug" rather than "boot entire guest OS and userspace then trigger bug". More detail in this blog post.

Supplementary debugging tip 2: if you take the "debug QEMU with host gdb" approach, you can use the reverse-debugger rr, which is very handy for memory-corruption bugs, because you can say "now execute backwards to whatever last touched this memory". More info in this post.

Peter Maydell
  • 9,707
  • 1
  • 19
  • 25