0

Recently, I tried to switch execution level from EL2 to EL1 and using qemu (aarch64) to simulate rpi3 behavior.

Following is my switching code:

.globl from_el2_to_el1  
from_el2_to_el1:
    mov     x1, (1 << 31)       // EL1 uses aarch64
    msr     hcr_el2, x1
    mov     x1, 0x3c5           // EL1h (SPSel = 1) with interrupt disabled
    msr     spsr_el2, x1
    ldr     x1, =set_el1_sp
    msr     elr_el2, x1
    eret                        // return to EL1

set_el1_sp:
    mov     sp, x0
    ret

it will be triggered from another .c file like:

void * ptr = kmalloc(0x10000);
ptr = (void*)((unsigned long)ptr + (unsigned long)(0x10000));
from_el2_to_el1(ptr);

Now, by using aarch64 gdb, I'm sure current exception level and stack pointer has been changed to expected value.

But, the weird part is, my print function (by uart) go wrong after calling from_el2_to_el1. Before it, if I called my uart_printf, the assembly is like:

IN: 
// uart_printf("A\n");
0x00080a38:  f9000fe0  str      x0, [sp, #0x18]
0x00080a3c:  f9400fe0  ldr      x0, [sp, #0x18]
0x00080a40:  91404000  add      x0, x0, #0x10, lsl #12
0x00080a44:  f9000fe0  str      x0, [sp, #0x18]
0x00080a48:  90000020  adrp     x0, #0x84000
0x00080a4c:  91320000  add      x0, x0, #0xc80
0x00080a50:  940000ef  bl       #0x80e0c

after calling from_el2_to_el1, `uart_printf("A\n") becomes:

IN: 
// uart_printf("A\n") after from_el2_to_el1;
0x00080a5c:  90000020  adrp     x0, #0x84000
0x00080a60:  91320000  add      x0, x0, #0xc80
0x00080a64:  940000ea  bl       #0x80e0c

----------------
IN: 
0x00080e0c:  a9a17bfd  stp      x29, x30, [sp, #-0x1f0]!
0x00080e10:  910003fd  mov      x29, sp
0x00080e14:  f9001fe0  str      x0, [sp, #0x38]
0x00080e18:  f900dfe1  str      x1, [sp, #0x1b8]
0x00080e1c:  f900e3e2  str      x2, [sp, #0x1c0]
0x00080e20:  f900e7e3  str      x3, [sp, #0x1c8]
0x00080e24:  f900ebe4  str      x4, [sp, #0x1d0]
0x00080e28:  f900efe5  str      x5, [sp, #0x1d8]
0x00080e2c:  f900f3e6  str      x6, [sp, #0x1e0]
0x00080e30:  f900f7e7  str      x7, [sp, #0x1e8]
0x00080e34:  3d804fe0  str      q0, [sp, #0x130]

----------------
IN: 
0x00000200:  00000000  .byte    0x00, 0x00, 0x00, 0x00

I can't figure out what cause this, also, if I enable IRQ, I found it will trigger exeception 4 (SYNC_INVALID_EL1h)

Last, if you suspected my uart_printf if problem occurred, following is my implementation:

int uart_printf(char *str, ...) {
    int ret;
    char buff[MAX_BUFF_SIZE];
    va_list va;
    va_start(va, str);
    ret = mini_vsnprintf(buff, MAX_BUFF_SIZE, str, va);
    va_end(va);
    uart_print(buff);
    return ret;
}

mini_vsnprintf is taken from this site.

From my testing, after calling from_el2_to_el1, it will block before executing the first line of uart_printf. So, maybe something go wrong in parameter or something?

My question are:

  1. Are there something wrong in my switching function?
  2. Any possibility about why it go wrong from my limited observation?

Any suggestion is appreciated, thanks in advance!

Steven
  • 811
  • 4
  • 23
  • what do you mean by wrong ? `adrp x0, #0x84000; add x0, x0, #0xc80; bl #0x80e0c` looks same in both situations – user3124812 Jul 30 '21 at 00:05
  • Scenario 2 lack first four instructions in scenario 1. Also, after `bl #0x80e0c`, in scenario 1, it didn't do `stp x29, x30, [sp, #-0x1f0]! 0x00080e10: 910003fd mov x29, sp.....`. In short, two same statement (`uart_printf("A\n")`) do the different thing in EL2 and EL1. (I know this might be right, but it isn't normal that exception only occurred in EL1) – Steven Jul 30 '21 at 00:18
  • what make you think that first 4 instructions are anyhow related to `uart_printf("A\n"))` ? and anything after `bl #0x80e0c` are not related to `uart_printf()` at all, that's some other piece of code – user3124812 Jul 30 '21 at 02:00
  • you are right, first four instructions is nothing to do with `uart_printf` and code after `bl #0x80e0c` might be part of `uart_printf()`. But I suspect there must be something wrong in the part after `bl #0x80e0c`, and it cause exception. – Steven Jul 30 '21 at 02:26
  • Best way to know what is causing that error is to check exception syndrome register (ESR), so you would get a clue what sort of event triggers it. My bet is on that MMU is not set properly. Each `EL` has it's own MMU table and when EL is switched memory mapping changed as well. So either MMU tables are not properly set for EL1, or MMU for EL1 is disabled all along. As a result address in `PC` register could not be resolved to fetch instruction. Also make sure that 'security' is set properly. For example MMU tables are configured for secure state but you switch into EL1_non_secure. – user3124812 Jul 30 '21 at 10:22
  • Thanks for your advice, I solved the problem. Instead change any assembly, I just add gcc flag: `-mcpu=cortex-a53+nofp`. I think this is relative to this [question](https://stackoverflow.com/questions/66444111/gcc-for-arm-force-no-floating-point) – Steven Jul 30 '21 at 15:01

0 Answers0