2

When writing kernels and OSes in general, whether it be in assembly or something higher level, you need to assemble or compile your code in flat binary, right?

You can't assemble it or compile it to anything like ELF format or anything like that, right?

If you did, the processor would misinterpret the formatting as code and start executing unintended instructions.

After all, you format executable binaries so that the OS knows where code and date segments start and stop and then can load them into the GDT and add them to the paging structures.

But if the program you're writing actually is an OS, then it won't be running over an OS like a user app will as it is the OS, right?

That is, the OS runs on the metal, not over any other software.

Am I correct here?

matrix
  • 21
  • 4
  • 2
    But you usually have a boot loader which may understand e.g. ELF format. See also [multiboot](https://www.gnu.org/software/grub/manual/multiboot/multiboot.html). – Jester Sep 19 '17 at 16:10
  • wow. ok. i never knew that. the bootloader knows the formatting and can decipher it. – matrix Sep 19 '17 at 16:13
  • You have a chicken and egg problem yes. At some point, determined by system design and/or a specific implementation, the very first code run on a processor can be built as an elf or whatever, but has to land in the rom as a memory image, after that that first program can be designed to interpret other formats and then you build on that... – old_timer Sep 19 '17 at 16:52
  • 1
    Also separate the toolchain and the target, in that if the toolchain natively uses elf for objects and the binary, that is fine because that is just a file format you can convert to some other file format either using the tools as in objcopy or can create your own tools separately to prepare the data for the target hardware. – old_timer Sep 19 '17 at 17:05
  • 1
    Side note: Most x86 OSes don't modify the GDT for different processes. Every 32-bit process on Linux uses the same CS descriptor, and every 64-bit process uses another. A context switch only requires saving/restoring registers and pointing CR3 at a new set of page tables. It uses a flat memory model so base=0 and limit=-1UL covers the entire virtual memory space. Memory protection is done through paging, not segmentation. – Peter Cordes Sep 19 '17 at 17:58
  • @PeterCordes It's silly to modify the GDT for different processes (after all, it's supposed to be *global*). But the LDT is actually modified because you can load custom LDT entries for your process (cf. `modify_ldt(2)`). – fuz Sep 19 '17 at 19:14
  • @peter limit=-1UL ??? Do you mean limit = UL-1? – matrix Sep 19 '17 at 21:23
  • @peter Also, I already knew those things about segmentation and paging in 64-bit mode. but segmentation is more than just bases and limits. descriptors have more info in them like privilege levels and types. are you saying that in 64-bit mode, the hardware completely ignores all of those things? – matrix Sep 19 '17 at 22:45
  • @matrix: No, I mean that all user-space processes can use the *same* segment descriptor for their code segments. Linux (the kernel) only needs to create a few descriptors, for code32, code64, data (and maybe stack IIRC). Every process can use the same descriptors. Kernel mode of course has a separate descriptor, which makes it run in ring0. And `-1UL` is C syntax for `0xFFFFFFFFFFFFFFFF` (or `0xFFFFFFFF` in 32-bit mode where `long` is 32-bit). – Peter Cordes Sep 20 '17 at 01:35

1 Answers1

5

Sort of, each platform has it's own boot sequence, which also defines how the user can gain the control over the machine.

For example in x86 world the first code run is from BIOS ROM (hardwired firmware from motherboard vendor), which will then load initial bootloader (sector 0 on storage block-device, so the BIOS firmware must already contain some simplistic/full drivers for all kind of different devices = it's not some kind of trivial couple instructions warming up the PC, but an small OS itself). Then bootloader will handle further loading of remaining code (either extended bootloader code if it doesn't fit into single sector or kernel or whatever user provided).

So if your bootloader is complex enough, it may understand even ELF binary for kernel, and load it from it.

But as long as you track down the very initial code to be run (BIOS firmware on x86 PC), then that one must be flat binary, starting code at some specified address (defined by CPU/board vendor, depending on the state of CPU after RST signal).

Also the OS may be already too late into the process, so it has no guarantee it runs directly on the bare metal, it may land into already some virtualised environment. Usually it's possible to detect it, but with perfect emulation/virtualisation it may be undetectable (then again, achieving something perfect tends to be very elusive in computer world). But that doesn't change anything (*) from the OS point of view, it may still proceed as if it runs on bare metal, it's up to the emulation/virtualisation to catch up and live up to OS expectations.

*) actually it may be very important for security sensitive installations, to detect any tampering over the environment, and handle such situations to mitigate security risks (self destruction or wiping sensitive information).

Update: also with the modern boards supporting UEFI (modern BIOS), TPM (Trusted Platform Module) chip, IME (Intel Management Engine) that initial flat binary can be encrypted, and digitally signed, so the CPU will not execute it unless the decryption and validation of signature are successful.

With IME the situation is even more complicated, it's like computer within computer, so there may be something going on in the background, even when the wrapping x86 machine is not awaken (just on power in stand-by).

If you are just starting to look into OS development, don't worry about this. If you are planning to create some x86 based security/medical device, then maybe pay additional attention to these.

Ped7g
  • 16,236
  • 3
  • 26
  • 63
  • Thanks. I know that VM stuff exists. But I was asking about non virtual situations. So, if your bootloader is simplistic and can't understand any formatting, then you're stuck assembling your OS code into flat binary, right? – matrix Sep 19 '17 at 16:25
  • Yes, you must prepare something what your bootloader (\*) is capable to load. If it's simple flat binary, then you have to produce kernel like that. The VM stuff happens today probably more often than not, for example most of the OS dev tutorials target BOCHS (I'm sort of mixing up VM and emulator stuff, but neither is "running on bare metal"). (\*) and the bootloader must be prepared in a way as that machine defines, either as sector 0 (or sector 1 when partition table is in 0) content on x86, or as some EEPROM content baked into the board directly. – Ped7g Sep 19 '17 at 16:27
  • I don't know from my head, but I'm pretty sure it will be specified either here https://www.gnu.org/software/grub/grub-documentation.html or in the code itself (which is public). – Ped7g Sep 19 '17 at 16:31
  • i assume it understands ELF as ELF is the standard binary format for the Linux world, yes? – matrix Sep 19 '17 at 16:37
  • *"assume = to make an 'ass' out of 'u' and 'me'."* / https://www.gnu.org/software/grub/manual/multiboot/multiboot.txt / *"If bit 16 in the `flags' word is set, then the fields at offsets 12-28 in the Multiboot header are valid ... Compliant boot loaders must be able to load images that either are in ELF format or contain the load address information embedded in the Multiboot header; they may also directly support other executable formats"* – Ped7g Sep 19 '17 at 16:41
  • ***With respect to your update***: What if you remove the UEFI ROM from the motherboard, then use a chip reader/writer to read the binary flashed on it, then use a disassembler to get it into assembly, then find the part of the UEFI code that says "check for digital signature", then remove that code from the assembly, then reassemble the new code, and then use the chip reader/writer to reflash it, then resolder it back onto the board, and then boot whatever you want. Would that work? – matrix Sep 19 '17 at 17:24
  • I don't recall any details, but IIRC the TPM chip is capable both to validate signature and to decrypt the machine code on the fly on it's own (without CPU), so I guess already first instruction of firmware is executed from encrypted image. And I guess the CPU is stalled at the beginning, until the signature is verified, if the board is configured to run only trusted code. I don't know how the board can be reconfigured legitimately (it can, as you can boot unsigned bootloaders and custom OS), but you certainly can't jump back to trusted code after some untrusted did already run. – Ped7g Sep 19 '17 at 17:48
  • I mean, it's not a huge problem if you manage to run your own code on the target machine, as long as you don't manage to decrypt/tamper-with the original one. It may be a bit tricky, if your code would manage to behave exactly as the trusted one and provide all the tasks, like for example simulating voting machine GUI, but then again the final sending of fake results should fail to be signed by the machine key from TPM, which would refuse to sign the data, because it didn't boot in trusted way. *If* the SW is properly done and uses TPM to full extent on all loose ends... ;) – Ped7g Sep 19 '17 at 17:55
  • But I'm getting more into the details which are beyond my expertise, so take my info with grain of salt, and if you need exact information, rather turn to the official documentation and professional help of people working with these things, I'm neither doing this lowest level stuff at the moment (I'm on more upper layers), nor on x86 platforms. – Ped7g Sep 19 '17 at 17:58