0

I am developing a basic hypervisor on ARM (using the board Arndale Exynos 5250). I want to load Linux(ubuntu or smth else)/Android as the guest. Currently I'm using a Linaro distribution.

I'm almost there, most of the big problems have already been dealt with, except for the last one: reserving memory for my hypervisor such that the kernel does not try to OVERWRITE it BEFORE parsing the FDT or the kernel command line.

The problem is that my Linaro distribution's U-Boot passes a FDT in R2 to the linux kernel, BUT the kernel tries to overwrite my hypervisor's memory before seeing that I reserved that memory region in the FDT (by decompiling the DTB, modifying the DTS and recompiling it). I've tried to change the kernel command-line parameters, but they are also parsed AFTER the kernel tries to overwrite my reserved portion of memory.

Thus, what I need is a safe memory location in the physical RAM where to put my hypervisor's code at such that the Linux kernel won't try to access (r/w) it BEFORE parsing the FDT or it's kernel command line.

Context details:

  • The system RAM layout on Exynos 5250 is: physical RAM starts at 0x4000_0000 (=1GB) and has the length 0x8000_0000 (=2GB).
  • The linux kernel is loaded (by U-Boot) at 0x4000_7000, it's size (uncompressed uImage) is less than 5MB and it's entry point is set to be at 0x4000_8000;
  • uInitrd is loaded at 0x4200_0000 and has the size less than 2MB
  • The FDT (board.dtb) is loaded at 0x41f0_0000 (passed in R2) and has the size less than 35KB
  • I currently load my hypervisor at 0x40C0_0000 and I want to reserve 200MB (0x0C80_0000) starting from that address, but the kernel tries to write there (a stage 2 HYP trap tells me that) before looking in the FDT or in the command line to see that the region is actually reserved. If instead I load my hypervisor at 0x5000_0000 (without even modifying the original DTB or the command line), it does not try to overwrite me!
  • The FDT is passed directly, not through ATAGs

Since when loading my hypervisor at 0x5000_0000 the kernel does not try to overwrite it whatsoever, I assume there are memory regions that Linux does not touch before parsing the FDT/command-line. I need to know whether this is true or not, and if true, some details regarding these memory regions.

Thanks!

RELATED QUESTION:

Does anyone happen to know what is the priority between the following: ATAGs / kernel-command line / FDT? For instance, if I reserve memory through the kernel command-line, but not in the FDT (.dtb) should it work or is the command-line overriden by the FDT? Is there somekind of concatenation between these three?

Zuzu Corneliu
  • 1,594
  • 2
  • 15
  • 27
  • 1
    If it is possible for U-Boot to overwrite your hypervisor code then you are not quite there yet. You should first focus on that: it should not be possible for a guest runtime to alter the hypervisor. Reconsider the design of memory isolation. – Christophe Augier Jan 20 '14 at 14:10
  • I have not said that U-Boot is the one that overwrites my hypervisor :) It is the Linux kernel. My hypervisor code executes BETWEEN u-boot (hypervisor is loaded by u-boot) and the kernel (hypervisor loads linux). And it's not that it actually overwrites it, it's just that it tries to (obviously resulting in a HYP trap) – Zuzu Corneliu Jan 20 '14 at 14:22
  • Ok my misunderstanding so it seems memory isolation is ok. But then didn't you configure the guest address space so that 0x40000000 maps to 0x0c800000? – Christophe Augier Jan 20 '14 at 14:32
  • why would I do that? this would cause serious inconsistencies for the entire FDT information given to Linux.. – Zuzu Corneliu Jan 20 '14 at 14:40
  • 0x0C800000 = 200MB in hex, it is not an address :) – Zuzu Corneliu Jan 20 '14 at 14:43
  • Why would it cause inconsistencies? the FDT doesn't need to be changed at all if you are using hardware virtualization. The stage-2 translation should hide the fact that for the guest the physical address is not 0x40000000 but instead 0x4c800000. In the end the guest is unchanged and the memory where the hypervisor is located is safe. – Christophe Augier Jan 20 '14 at 14:51
  • first you said to map 0x4000_0000 to 0x0C80_0000, I understand now that you meant 0x4C80_0000, but if you think about it there's still a problem; if you look at my post I mentioned there that the system RAM starts at 1GB and has the length 2GB; this means that the maximum addressable byte of the system RAM is the one with the address (3GB-1) – Zuzu Corneliu Jan 20 '14 at 15:06
  • AFTER this address (3GB-1) there are still memory-mapped devices and this poses a problem, since I would still be forced to leave a gap between these and the system RAM with my stage 2 mappings; and even if there weren't memory mapped devices there, a gap would still be unavoidable; thank you very much for trying to help me out, but I think it would be better if you thought more about it ( maybe with a piece of paper in your hands, please don't take any offense cause I don't mean any :) ) – Zuzu Corneliu Jan 20 '14 at 15:08
  • I never talked about translating the whole 2GB. You need to manage the memory pool in your hypervisor. You can either split it in banks or do dynamic memory allocation. Of course it is easiest to split the memory statically and for instance define a 1GB memory bank for your first guest OS (which ok would need a one line change to the FDT). In this example with Linux MMU set you would get the following memory translation 0xc0000000 -stage-1-> 0x40000000 -stage-2-> 0x4c800000. – Christophe Augier Jan 20 '14 at 15:25
  • Oh and by the way I opened the exynos 5250 datasheets and it seems all the devices are mapped before 0x20000000. – Christophe Augier Jan 20 '14 at 15:26

2 Answers2

0

As per https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/Documentation/arm/Booting, safe locations start 128MB from start of RAM (assuming the kernel is loaded in that region, which is should be). If a zImage was loaded lower in memory than what is likely to be the end address of the decompressed image, it might relocate itself higher up before it starts decompressing. But in addition to this, the kernel has a .bss region beyond the end of the decompressed image in memory.

(Do also note that your FDT and initrd locations already violate this specification, and that the memory block you are wanting to reserve covers the locations of both of these.)

Effectively, your reserved area should go after the FDT and initrd in memory - which 0x50000000 is. But anything > 0x08000000 from start of RAM should work, portably, so long as that doesn't overwrite the FDT, initrd or U-Boot in memory.

unixsmurf
  • 5,852
  • 1
  • 33
  • 40
  • damn! I don't know how I didn't realize this before but I remember calculating the address of my HYP NOT TO overlap initrd and the FDT; I now realize my calculations were very wrong; I read that document too before posting this and that's actually the reason I've positioned it at 0x5000_0000 in the first place (after initrd and the FDT), but it seemed peculiar to me that the kernel didn't try to write there even if I didn't modify the original FDT and initrd; I am still confused by this, I'll explain why in the comment below – Zuzu Corneliu Jan 21 '14 at 06:39
  • so..you say that the FDT and initrd locations violate the recommended mem-layout specification, but I have not chosen the locations in memory of either of them (uImage, initrd and the FDT are positioned at the locations they were positioned by the u-boot in the UNTOUCHED prebuilt linaro image); also, there's one more thing that bothers me; I also tried positioning my HYP just below the top of RAM (at address (3GB-200MB); the RAM starts at 1GB and it's 2GB in size); but Linux tries to write there too...thus, it didn't seem to me that the "ABOVE ram base + 128MB" thing still holds.. – Zuzu Corneliu Jan 21 '14 at 06:46
  • in the first comment I meant "even if I didn't modify the original FDT and/or kernel command-line" – Zuzu Corneliu Jan 21 '14 at 06:51
  • I guess I'll try to relocate everything how that document recommends (regardless of the positioning the u-boot in the original linaro image chose) and test if reserving memory for HYP using the FDT/kernel command-line works after this, but it still seems peculiar to me that the kernel tries to do write operations at the top of RAM before even parsing the FDT and/or the command-line; this actually makes me think that it uses hardcodings, which would really be messing up my intentions.. for now, thanks a lot for pointing out how stupidly I chose to position the HYP at 0x40C0_0000! – Zuzu Corneliu Jan 21 '14 at 06:59
  • @Zuzel: well, in defence of the U-boot port, that document used to be pretty vague, and was updated long after U-Boot for the Arndale was published. If you are seeing your image being overwritten near top of RAM, I would suggest that either that is caused by U-Boot (which usually relocates itself to near top of RAM) or that you have somehow failed to communicate to the kernel that it should ignore that region. – unixsmurf Jan 21 '14 at 08:38
  • U-Boot has finished execution when my hypervisor starts executing, thus it couldn't be the first option; as for communication with the kernel regarding reservation of that memory region, I tried to use 1) a kernel command-line parameter, more specifically memmap=nn[KMG]$ss[KMG], see here: https://www.kernel.org/doc/Documentation/kernel-parameters.txt and 2) modification of the FDT; none seem to work for now, since when cat-ing /proc/iomem, the region is still shown as system RAM (thus it's still used on allocations, etc..); but I'm still digging, I'll let you know of any progress – Zuzu Corneliu Jan 21 '14 at 08:53
  • thanks, I ended up positioning it at ram base + 128MB; on two ARM boards with the Linux versions I had installed on them, it seems to work; I needed confirmation that I was doing the right thing – Zuzu Corneliu Feb 03 '14 at 16:15
0

The priority of kernel/FDT/bootloader command line depends on the kernel configuration - do a menuconfig and check under "Boot options". You can combine ATAGS with the built-in command lines, but not FDT - after all, the FDT chosen node is supposed to be generated by the bootloader - U-boot's FDT support is OK so you should let it do this rather than baking it into the .dts if you want an FDT command line.

The kernel is pretty conservative before it's got its memory map since it has to blindly trust the bootloader has laid things out as specified. U-boot on the other hand is copying bits of itself all over the place and is certainly the culprit for the top end of RAM - if you #define DEBUG in (I think) common/board_f.c you'll get a dump of what it hits during relocation (not including the Exynos iRAM SPL/boot code stuff, but that won't make a difference here anyway).

Notlikethat
  • 20,095
  • 3
  • 40
  • 77