2

I am working on supporting posted interrupts in a VMM that I'm writing that uses Intel VT-x virtualization. One of the VM entry requirements specified in the documentation for enabling posted interrupts is that the "Acknowledge interrupt on exit" VM-exit control must be set to 1.

When I set this control to 1, my guest OS runs for a short while before it stops responding. Then, the host OS stops responding as well, and a message is printed to the host kernel log that says that the CPU core that the guest OS was running on has experienced a hard lockup (NMI watchdog: Watchdog detected hard LOCKUP on cpu 10).

I'm reading the Intel documentation and trying to think through this, but was wondering if anyone else knew what's going on. My general thoughts right now are that the host OS must be sending an interrupt to the core that the guest OS is running on at the moment (i.e. my guest OS is not involved in sending the interrupt), which causes a VM exit. Since I set the "Acknowledge interrupt on exit" control to 1, the processor acknowledges to the interrupt controller that the interrupt was received, and puts the vector in the VM-exit interruption-information field. Furthermore, since I don't do anything with the interruption-information field in my VMM at the moment, the interrupt then doesn't get handled by the host OS, which causes the problem. Am I heading in the right direction?

Jack Humphries
  • 13,056
  • 14
  • 84
  • 125

1 Answers1

4

Yes, not allowing the host OS to handle device interrupts would be a problem. Instead of setting the "acknowledge interrupt on exit" control to one you should probably set it to 0. Then when you get a VM exit due to a hardware interrupt you should set the processor interrupt enable flag to allow the interrupt to be acknowledged and serviced normally in the host.

From the Intel 64 and IA-32 Architectures Software Developer’s Manual:

33.2 INTERRUPT HANDLING IN VMX OPERATION

  • Acknowledge interrupt on exit. The “acknowledge interrupt on exit” VM-exit control in the controlling VMCS controls processor behavior for external interrupt acknowledgement. If the control is 1, the processor acknowledges the interrupt controller to acquire the interrupt vector upon VM exit, and stores the vector in the VM-exit interruption-information field. If the control is 0, the external interrupt is not acknowledged during VM exit. Since RFLAGS.IF is automatically cleared on VM exits due to external interrupts, VMM re-enabling of interrupts (setting RFLAGS.IF = 1) initiates the external interrupt acknowledgement and vectoring of the external interrupt through the monitor/host IDT

Alternatively, if you have a good reason to set this this bit, say if you need to route certain hardware interrupts to the guest, then you'll need to invoke the host OS's interrupt handler as if it had been invoked by the CPU.

Ross Ridge
  • 38,414
  • 7
  • 81
  • 112
  • Yeah, I need to enable this in order to get posted interrupt processing to work. I'll try this and see how it goes. Thanks. – Jack Humphries Dec 30 '17 at 04:15
  • 2
    I just added in code to call the correct Linux interrupt handling function on a VM exit caused by an external interrupt, and everything works now. For anyone interested, there is some code that does this in the `vmx_handle_external_intr` function in `arch/x86/kvm/vmx.c` in the KVM project. – Jack Humphries Dec 30 '17 at 22:25
  • @abcd, I am also studying X86 VT-X, and I read the code you mentioned in KVM, it is complicated to understand the full context. Do you have a repo or code sample on this for my reference? Thanks, – wangt13 Aug 26 '19 at 00:25
  • @wangt13 Yes, I added support for posted interrupt processing to Dune (https://github.com/project-dune/dune). Take a look at the three most recent commits in the repository. Let me know if I can help with anything specific. – Jack Humphries Sep 01 '19 at 00:15
  • @abcd, let me first check out dune's code to go through it. I will post question to SO if I have on it. – wangt13 Sep 02 '19 at 07:39