QEMU/vExpress issue regarding SMP and U-boot

Question

Since quite a long time, there is still an issue with QEMU/vExpress and U-boot using more than one CPU core (vExpress refers to a ARM Cortex-A15 CPU). Before digging into Qemu/U-boot code, has anybody investigated further about that ? (I'm ready to go further into the bootstrap code of vExpress trying to stall other CPUs than the first one). Currently, it sounds like Qemu starts as many U-boot instances as there are declared CPUs (using -smp options).

Answer 1

This sounds like the problem is in the guest code not expecting to be started the way QEMU starts it. For vexpress, QEMU has two ways of booting:

1. you boot a Linux kernel, which we boot as the vexpress bootrom boots Linux kernels, which is by starting the primary CPU and putting the secondary CPUs into a WFI loop waiting for the kernel to get to a point where it wants them to start
2. you boot something "bare metal", which we boot by starting all of the CPUs at once, and expecting the bare metal image to sort them out (i.e. to have a bit of code in its early boot up that looks at the CPU ID registers and diverts the secondary CPUs into a holding loop).

If you do a bare-metal boot but your guest image doesn't have the necessary code to handle the secondary CPUs, the effect will be that all of them try to follow the primary-CPU codepath, which doesn't work very well (multiple copies of console output, weird crashes, other confusion).

I'm not entirely sure how the real hardware does this, but part of the issue here is that QEMU doesn't emulate the "configuration controller" microcontrollers on the real h/w and we don't run the "Boot Monitor" software/firmware that the h/w usually does, so a bare-metal image for QEMU has to do some work that would on the h/w be done by one or the other of those.

Answer 2

I finally succeeded in starting U-boot on Qemu/vExpress with more than one CPU. The method for stalling secondary CPUs is similar to how Qemu does when starting a Linux kernel image.

Just apply the following patch in the U-boot root directory (tested on versions > 2018)

From bf0fb125a3d85a5f0a6d7da744ebf110fc7e0f47 Mon Sep 17 00:00:00 2001
From: Daniel Rossier <daniel.rossier@heig-vd.ch>
Date: Thu, 4 Jul 2019 13:06:39 +0200
Subject: [PATCH] ARM vExpress: stalling secondary CPUs when Qemu starts U-boot

Signed-off-by: Daniel Rossier <daniel.rossier@heig-vd.ch>
---

 board/armltd/vexpress/Makefile        |  3 +-
 board/armltd/vexpress/lowlevel_init.S | 91 +++++++++++++++++++++++++++
 2 files changed, 93 insertions(+), 1 deletion(-)
 create mode 100644 board/armltd/vexpress/lowlevel_init.S

diff --git a/board/armltd/vexpress/Makefile b/board/armltd/vexpress/Makefile
index 2a659de012..9048f9b39e 100644
--- a/board/armltd/vexpress/Makefile
+++ b/board/armltd/vexpress/Makefile
@@ -3,5 +3,6 @@
 # (C) Copyright 2000-2004
 # Wolfgang Denk, DENX Software Engineering, wd@denx.de.

-obj-y  := vexpress_common.o
+obj-y  := vexpress_common.o 
+obj-y  += lowlevel_init.o
 obj-$(CONFIG_TARGET_VEXPRESS_CA15_TC2) += vexpress_tc2.o
diff --git a/board/armltd/vexpress/lowlevel_init.S b/board/armltd/vexpress/lowlevel_init.S
new file mode 100644
index 0000000000..dca03d9606
--- /dev/null
+++ b/board/armltd/vexpress/lowlevel_init.S
@@ -0,0 +1,91 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * (C) Copyright 2019 Daniel Rossier <daniel.rossier@heig-vd.ch>
+ */
+
+#include <config.h>
+
+/*
+ * Routine: save_boot_params (called after reset from start.S)
+ * Description: check the running CPU and stall it if not running on
+ * the primary CPU.
+ * This allows U-boot to run on Qemu/vExpress with a number of CPU
+ * greater than 1 (-smp > 1)
+ */
+
+.global save_boot_params
+
+/*
+ * The stalling code is mainly based on the bootcode portion (smpboot)
+ * from Qemu.
+ */
+__start:
+stall_secondary_cpu:
+   ldr r2, GIC_CPU_IF
+   ldr r0, BOOT_REG
+   mov r1, #1
+   str r1, [r2]
+   mov r1, #0xff
+   str r1, [r2, #4]
+   dsb sy
+
+__wfi:
+   wfi
+   ldr r1, [r0]
+   tst r1, r1
+   beq __wfi
+   bx  r1
+   @ Never reach this point
+
+   @ GIC CPI IF address
+GIC_CPU_IF:
+   .word 0x2c002000
+
+@ Store the entry point used at CPU wake-up
+BOOT_REG:
+   .word 0x1c010030
+
+@ vExpress SRAM
+DEST_ADDR:
+   .word 0x14000000
+
+__end:
+
+save_boot_params:
+
+   @ Check if we are on the primary (CPU #0)
+   @ Read Multiprocessor ID register
+    mrc    p15, 0, r0, c0, c0, 5
+   ands    r0, r0, #0x3
+   beq     out_primary
+
+   @ Relocate the code handling the secondary CPUs to SRAM
+   @ thus allowing U-boot to be relocated itself in the DRAM
+   @ without interfering with this code.
+
+   ldr r0, =__start
+   ldr r1, DEST_ADDR
+
+   ldr r2, __code_size
+
+relocate:
+
+   ldr r3, [r0], #1
+   str r3, [r1], #1
+
+   subs r2, r2, #1
+   bne relocate
+
+   @ Now, jump to the stalling code
+   ldr pc, DEST_ADDR
+
+out_primary:
+
+   /* Returns */
+   b   save_boot_params_ret
+
+__code_size:
+   .word   __end - __start
+
+
+
-- 
2.17.1