0

I'm working on a kernel for i386 and want to do the following:

  1. Write data to a virtual 4MB page in the current page directory ("current" as in it's loaded to cr3).
  2. Make a separate page directory and set one of its entries to map to the physical address mapped to the 4MB page mentioned in step (1).
  3. Unmap the virtual page from step in the current page directory (that is, set that PD's entry to zero).
  4. Load the separate page directory from step 2 (via cr3) and read that data.

My intention with this is that I can write data to a page in one address space, then "transfer" that data to another address space by mapping to a page in that AS, then umapping the page in the current AS. However, this doesn't seem to work as intended. Once I load the second page directory, I'm noticing the data I wrote in step 1 isn't actually at the new virtual page I mapped to. I assume this has something to do with the TLB not properly being flushed, but I call invlpg on the virtual addresses in steps 2 and 3.

I'm able to fix this by flushing the entire TLB by reloading cr3. That is:

mov %cr3, %eax
mov %eax, %cr3

If I do this at the end of step 2 (after mapping the page in the separate page directory), then I can expect to see the data I wrote from step 1 in the new page dir after it's loaded.

Does this only work because invlpg will only invalidate TLB entries for pages in the current page directory? I always thought the TLB operated completely separately from the page table. One thing I expected was also for the TLB to be flushed at the start of step 4 when I load the new page directory via cr3, but I'm not seeing expected behavior.

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847
BrockLee
  • 931
  • 2
  • 9
  • 24
  • 1
    Unless I'm misunderstanding you, step 4 (loading cr3 with the new page directory) should flush the entire TLB; an additional save and reload of cr3 should not do anything more. – Chris Dodd May 28 '22 at 02:10
  • Are any of your PTEs / PDEs marked as "global"? i.e. allowing the TLB entry to persist across writes of CR3 (designed for parts of virtual address-space a kernel wants to keep the same in every process.) https://wiki.osdev.org/Paging#Page_Directory. If that's not it, how are you testing this? This obviously isn't a [mcve], and you haven't even linked to a whole project in case anyone's feeling keen or bored and wants to look over the code or debug it for you. That's a problem because your symptoms don't make sense given what you say you're doing, as Chris points out. – Peter Cordes May 28 '22 at 02:18
  • TLB invalidation wouldn't explain what you're seeing. As long as you've executed a store to a given physical page (via some virtual address), it doesn't matter whether the TLB keeps caching that old translation or not. All that matters is that the *new* CR3 points to a PD that maps the new virtual address to the same physical address as the store. (And that there wasn't already a stale global TLB entry for that virtual address). – Peter Cordes May 28 '22 at 02:28
  • `invlpg` does work on virtual addresses, not physical, flushing any TLB entry for it, including global ones: https://www.felixcloutier.com/x86/invlpg. The only exception being entries from other PCID (process context IDs) are allowed to not be flushed, but if you haven't set CR4.PCIDE = 1 then they're not enabled and everything is using the same context-ID of 0. – Peter Cordes May 28 '22 at 02:31
  • (Followup for completeness.) None of the PTDs have the global bit set. Since the kernel is still very early in development, I've mainly been debuging via a combination of print statements and dgb. Yeah sorry I couldn't come up with a minimal reproducer. I couldn't think of a way to reproduce this that wasn't "build my entire project from scratch", so any suggestions are really a shot in the dark. I found a workaround that doesn't involve "resetting cr3". For now, I'm leaning this might be something wrong with QEMU TCG since at least QEMU with KVM doesn't seem to reproduce this. – BrockLee Jun 07 '22 at 01:19

0 Answers0