1

I compile a macOS driverkit system extension as a Universal library so that it contains both x86_64 and arm64. One Apple Silicon computer A the driver starts when I attach the USB device. On Apple Silicon computer B I can see kernel: exec_mach_imgact: disallowing arm64 platform driverkit binary "com.example.driver", should be arm64e being printed in the Console.app when the USB device is attached. I've looked at the source code of where this is happening but I cannot figure out what the problem is. If I compile it for arm64e then it get exec_mach_imgact: not running binary "com.example.driver" built against preview arm64e on computer A, bit then it starts on computer B.

None of the computers have -arm64e_preview_abi set in the boot-args.

If I create a new Xcode (12.4) project on each machine and build Release then computer A and otool -fvv com.example.driver gives

Fat headers
fat_magic FAT_MAGIC
nfat_arch 2
architecture x86_64
    cputype CPU_TYPE_X86_64
    cpusubtype CPU_SUBTYPE_X86_64_ALL
    capabilities 0x0
    offset 16384
    size 73856
    align 2^14 (16384)
architecture arm64
    cputype CPU_TYPE_ARM64
    cpusubtype CPU_SUBTYPE_ARM64_ALL
    capabilities 0x0
    offset 98304
    size 73856
    align 2^14 (16384)

On computer B the same command gives

Fat headers
fat_magic FAT_MAGIC
nfat_arch 2
architecture x86_64
    cputype CPU_TYPE_X86_64
    cpusubtype CPU_SUBTYPE_X86_64_ALL
    capabilities 0x0
    offset 16384
    size 73280
    align 2^14 (16384)
architecture arm64
    cputype CPU_TYPE_ARM64
    cpusubtype CPU_SUBTYPE_ARM64_ALL
    capabilities 0x0
    offset 98304
    size 73296
    align 2^14 (16384)

How can I make the driver start on both machines?

tuple_cat
  • 1,165
  • 2
  • 7
  • 22
  • On the one that starts, which slice starts? Could it be that it's actually running in Rosetta? (Activity Monitor will show you) – pmdj Jan 31 '21 at 10:30
  • Architecture says "Apple" in Activity Monitor on the machine where the driver starts. – tuple_cat Jan 31 '21 at 13:39

3 Answers3

2

Dexts should indeed be arm64 and x86_64 (but as pmdj explains, system binaries are still arm64e.)

As hinted by the name of (and need for) the -arm64e_preview_abi, arm64e is currently only exposed as a developer preview, to allow for testing.

However, you shouldn't get the disallowing arm64 error: did you set other interesting boot-args on computer B? (in particular, amfi= may be relevant)

blraaz
  • 36
  • 1
  • 1
    At some point I had `boot-args="amfi_get_out_of_my_way=0x1"`, but I am unsure if I missed restarting after removing that. Now I have updated to Big Sur 11.2 and after updating the OS the driver starts when it is built as universal binary for `x86_64 arm64` – tuple_cat Feb 03 '21 at 09:55
0

Updated answer

We're now quite a few years down the road of DriverKit and Apple Silicon and I've shipped a few dexts since this question first showed up. Looks like I got my answer spectacularly wrong and didn't fix it when I had new information. Oops!

arm64 is definitely the correct architecture for developing and deploying dexts for macOS 11-13 and iPadOS 16.

Apple uses arm64e for its own "platform" dexts, but that doesn't have any bearing on third-party dexts.

There seems to be a bug in macOS which adds to the confusion: in certain setups, the system appears to expect dexts to be built as arm64e binaries. I've not tracked down the precise trigger, but my main suspect is turning off System Integrity Protection (SIP). It may also be related to the amfi_get_out_of_my_way=0x1 flag. In any case, turning on SIP (the default state) and not using any custom kernel boot-args, makes correctly signed and entitled arm64 ABI dexts work just fine.

Original answer

(I've left this because the first 4 comments below refer to it.)

My experience so far indicates that arm64e is the correct and only correct Apple Silicon architecture to use for dexts.

For one, there's the "disallowing arm64 platform" error, and also Apple's own DriverKit based drivers are built for arm64e:

% otool -fvv /System/Library/DriverExtensions/com.apple.AppleUserHIDDrivers.dext/com.apple.AppleUserHIDDrivers
Fat headers
fat_magic FAT_MAGIC
nfat_arch 2
architecture x86_64
    cputype CPU_TYPE_X86_64
    cpusubtype CPU_SUBTYPE_X86_64_ALL
    capabilities 0x0
    offset 16384
    size 96208
    align 2^14 (16384)
architecture arm64e
    cputype CPU_TYPE_ARM64
    cpusubtype CPU_SUBTYPE_ARM64E
    capabilities CPU_SUBTYPE_ARM64E_PTRAUTH_VERSION 0
    offset 114688
    size 95312
    align 2^14 (16384)

That leaves the question of why your arm64e build isn't working. The "built against preview arm64e" error suggests the problem isn't with the computer but the binary. Are you using identical binaries on the 2 systems? Perhaps one has SIP disabled, so it's more permissive of badly built binaries?

Have you tried a "hello world" style dext, in a freshly created project on the latest version of Xcode? Check that runs natively on both machines. Once that's working, compare Xcode's compiler and linker command lines with those from your build script - or if you're also using Xcode, compare your target's build settings with the "clean" one.

pmdj
  • 22,018
  • 3
  • 52
  • 103
  • I updated the question with some more info. I will try to make the example projects start, but I doubt that it will start on computer B since the binary is still arm64 even if I created a new Xcode project. – tuple_cat Jan 31 '21 at 13:18
  • The binaries that I tested are from the same installer that I have, so they are the same. SIP is disabled on both machines. – tuple_cat Jan 31 '21 at 13:34
  • Interesting is this [read](https://developer.apple.com/documentation/driverkit/debugging_and_testing_system_extensions) where Apple says that you can test your driverkit driver with `arm64e`. There are instructions on how to set the `-arm64e_preview_abi` `boot-args`. That and since Xcode builds `arm64` as default makes me think that `arm64` is the one that should be used for now. Maybe it is different depending on if the dext is one that ships with the OS or if it is one that the user installs. – tuple_cat Jan 31 '21 at 21:14
  • If I explicitly build for `x86_64` and `arm64e` then the `capabilities` says `PTR_AUTH_VERSION USERSPACE 0`, which is different from the `capabilities` line for `AppleUserHIDDrivers` (`CPU_SUBTYPE_ARM64E_PTRAUTH_VERSION 0`) – tuple_cat Feb 02 '21 at 16:53
0

I think the problem is "amfi_get_out_of_my_way=0x1",

My DEXT was once executed on MacOS12.4 Mac mini M1 arm64, and its boot-args was as follows: boot-args="-arm64e_preview_abi"

But now my boot-args are set to: boot-args="-arm64e_preview_abi amfi_get_out_of_my_way=0x1"

I try to compile my DEXT with Xcode1430 on MacOS1331 Mac mini M1 arm64, Then I get the same result as tuple_cat.

kernel: exec_mach_imgact: disallowing arm64 platform driverkit binary "com.example.driver"

I tried to compile my DEXT with Xcode1430 on MacOS1331, Mac mini x86_64 works fine. But Mac mini M1 arm64 doesn't work.

erich
  • 1
  • I've had a consulting client run into this problem on macOS 13 when they had SIP disabled: for some reason, with SIP off, it seems to expect dexts to use `arm64e`, but with SIP on, it wants `arm64`. I haven't investigated it more deeply, as their problems went away when they turned SIP back on and used the self-serve development signing entitlements. But perhaps what they actually did was turn on SIP *and* remove `amfi_get_out_of_my_way=0x1`, so I'm now not sure what exactly the trigger is. SIP seems the more likely culprit though. – pmdj Apr 14 '23 at 11:15