Changing segments to Ring3(user) from Ring0(kernel) results in segmentation fault

Question

I am writing a toy operating system. I am trying to implement memory protection for an operating system. All I want to do is create protection for the kernel from user space programs. I want to do this purely with segmentation and not paging.

Here is the GDT:

gdt_start:
    
    dd 0x0 ; 4 byte
    dd 0x0 ; 4 byte


gdt_code: 
    dw 0xfff7    ; segment length, bits 0-15
    dw 0x0       ; segment base, bits 0-15
    db 0x0       ; seg2ment base, bits 16-23
    db 10011010b ; flags (8 bits)
    db 11001111b ; flags (4 bits) + segment length, bits 16-19
    db 0x0       ; segment base, bits 24-31


gdt_data:
    dw 0xfff7
    dw 0x0
    db 0x0
    db 10010010b
    db 11001111b
    db 0x0

U_code: 
    dw 0xfff7    ; segment length, bits 0-15
    dw 0x0008       ; segment base, bits 0-15
    db 0x0       ; seg2ment base, bits 16-23
    db 11111010b ; flags (8 bits)
    db 11001111b ; flags (4 bits) + segment length, bits 16-19
    db 0x0       ; segment base, bits 24-31


U_data:
    dw 0xfff7
    dw 0x0008
    db 0x0
    db 11110010b
    db 11001111b
    db 0x0
gdt_end:

U_data and U_code are going to be the user space (ring 3). When I am in kernel space (ring 0) and try switching data segments by executing:

mov ax, 0x20 
mov ds, ax
mov ss, ax
mov es, ax
mov fs, ax
mov gs, ax

I get a segmentation error (13).

What am I doing wrong? Any guidance would be greatly appreciated.

I'm not sure if it's related, but I think you have an endian problem in your descriptors: did you really mean for the U_code and U_data segments to have base address 0x00000008? — Nate Eldredge, Mar 28 '23 at 03:54
If I'm reading the manual correctly, you're not allowed to load SS with the selector for a segment with a different DPL than your CPL. Your stack must always have the same privilege level as your code. If you comment out the `mov ss, ax` line, does the code still crash? — Nate Eldredge, Mar 28 '23 at 04:13
@NateEldredge, it stopped crashing after commenting out mov ss, ax. I think I did enter the base address in wrong as well thanks. I was trying to split ram in half — Jacob Knuth, Mar 28 '23 at 04:38
Yeah I was going to say that you have the endianness wrong for the base AND limit. I figured you were trying to get kernel memory running from the first half of memory and the user memory running the second half. Without paging and splitting memory in half make sure you have enough RAM. — Michael Petch, Mar 28 '23 at 04:46
@MichaelPetch: Are you sure? I haven't tried, but from the manual, I don't think you can use a far JMP to a code segment to change privilege levels. (For one thing, you'd have the same issue with the stack segment being the wrong privilege.) AFAIK the ways to switch to a lower-privilege ring are `retf`, `iret` or a task switch, all of which will load a new `ss:esp` simultaneously with the branch. — Nate Eldredge, Mar 28 '23 at 05:05
You'd have to jump into user mode (I assume user mode is ring 3) from kernel mode by using a far RET or an IRET with the code segment set to 0x18 | 0x03 = 0x1b and set the segment registers to 0x20 | 0x03 = 0x23. IRET would be preferable since you can set SS:ESP and flags at the same time you jump to an address in user mode. — Michael Petch, Mar 28 '23 at 05:11
The other day I created some test code to demonstrate using IRETD to get into user mode. This code example has its own bootloader. The important part is in stage2.asm which gets loaded by the boottoader to 0x7e00. To simplify the example I split the first 64KiB into kernel mode and user mode but the process is the same if you split 4GiB of memory into 2 halves although you'll have to somehow get your user mode code into the area at 2GiB in your case. https://github.com/mpetch/EnterUserModeSegmentation — Michael Petch, Mar 31 '23 at 16:57

Nate Eldredge · Accepted Answer · 2023-03-28T13:13:22.290

3

mov ss, ax

You are loading ss with a data segment whose DPL is 3, while your CPL is 0. That's not allowed. Your stack always has to be at the same privilege level as you. (Also, it wouldn't make any sense to load ss without loading esp at the same time.)

The switch to the user's stack needs to happen simultaneously with the transition to CPL 3. The main ways to do this are:

RETF or IRET. When the destination segment is less privileged, it will pop ss:esp from the (old) stack as well as cs:eip.
A hardware task switch, in which case ss:esp is loaded from the TSS along with all the other registers.

Call gates also perform a stack switch, but you can only use them to call more privileged code, not the other way around.

Also, as noted in comments, your base and limit fields seem to be mixed up. As it stands your user segments have a base of 0x00000008 and all your segments have a limit of 0xffff7000. Note that "bits 0-15" are the least significant bits. So if it U_data is supposed to be the upper half of memory, then you would want

    dw 0xffff
    dw 0x0000
    db 0x0
    db 11110010b
    db 11000111b  ; note the low nibble is now 7
    db 0x80

Note that this is probably still not what you want, as there may or may not be any memory there (e.g. if your machine has less than 2 GB), and there may also be I/O mapped into that region which the user should not have access to. So you'd really need to inspect the memory map to decide where to put the user's segments.

edited Mar 28 '23 at 13:13

answered Mar 28 '23 at 05:29

Nate Eldredge

48,811
6
54
82

iret is used when returning from an interrupt right? I guess that makes sense since your interrupt executes code in the kernel then returns to the lower privileged user task. What does REIF do? – Jacob Knuth Mar 28 '23 at 14:21
Thanks for the help. I am very new to any sort of OS dev. If my OS boots in to the kernel, then wants to run a user program that is at 0x80000000 on physical memory. 0x80000000 is the start of the user space segment as well. U guys are saying I can not do a far jump to that segment as it is lower privileged? What I'm confused is RETF and IRET are both returns, Can I jump in to the user space program for the first time with a return? – Jacob Knuth Mar 28 '23 at 14:51
@JacobKnuth yes you place values on the stack yourself and then use RETF or IRET like a jump. The processor doesn't care how the values got on the stack (whether you put them there yourself or they were there because of CALL or interrupt etc). It just takes the values from the stack and transfers control (after validation checks). – Michael Petch Mar 28 '23 at 14:57
ok that makes sense, could you show me a code example using NASM? what order do I push that data (code segment, offset, stack segment)? – Jacob Knuth Mar 28 '23 at 15:50
@JacobKnuth: For the full description of what `ret` (or any other instruction) does, consult [the instruction set reference](https://www.felixcloutier.com/x86/ret). You will find complete pseudocode for its operation. (`retf` is the mnemonic in NASM to specify the far return, opcode 0xcb, instead of the near return, opcode 0xc3; that's assembler-specific so the reference doesn't mention it.) You'll end up in the block marked "RETURN-TO-OUTER-PRIVILEGE-LEVEL". – Nate Eldredge Mar 29 '23 at 03:07
@JacobKnuth: A large part of OS development is careful reading of little-used sections of official documentation. Especially when, as in this case, you're using particularly obscure and obsolete CPU features. Yes, it looks long and intimidating up front, but you can do it! – Nate Eldredge Mar 29 '23 at 03:09

Changing segments to Ring3(user) from Ring0(kernel) results in segmentation fault

1 Answers1