-1

Many questions ask how to get M1/M2 apple silicon macs to install pure Intel or pure arm conda environments.

There are good answers on how to achieve both configurations.

The question I have not seen answered anywhere after extensive search is why it is not possible to mix architectures in one environment.

Is this an example of dynamically loaded library (DLL) hell?

If I just want to mix binaries, there should be no problem.

I should be able to use whichever architecture is more performant. Rosetta 2 should do emulation based on the binary architecture.

Where in the dependency chain do the problems arise? Can I ignore the warnings and still install mixed packages in the hope that they behave like separate binaries?

Cornelius Roemer
  • 3,772
  • 1
  • 24
  • 55
  • Dear close voter, I would like to know why a "why" question is opinion based. Why is the sky blue is not opinion based. – Cornelius Roemer Aug 05 '22 at 01:23
  • 2
    Are you asking about an ARM64 function calling an x86 function in the same binary process, or vice versa? There's no calling convention for that, no way of figuring out which x86 registers should get what value after a jump or call from ARM64 code. The emulation is at a whole-process level, for all of virtual address space, not on a per-function level to port individual functions to the standard ARM64 calling convention. Different ABIs (sizes and alignments for types, and struct layout) is also a problem. – Peter Cordes Aug 05 '22 at 01:23
  • @PeterCordes your comment helps a lot. I seem to have a fuzzy intuition about process and function level emulation. Is it correct that if ARM Python were to call x86 binary through a subprocess this would work, but numpy function level mixing wouldn't? This is a pretty satisfactory answer for me already. So some things could work but others couldn't depending on process vs function level dependencies? – Cornelius Roemer Aug 05 '22 at 01:40
  • 2
    Yeah, you can start a separate process of the other architecture, and talk to it through pipes or maybe even shared memory. But not calling into a library. – Peter Cordes Aug 05 '22 at 01:52
  • Excellent, I would love to see your comments as a brief answer for general limitations of emulation. Someone else maybe me could add china specifics later. – Cornelius Roemer Aug 05 '22 at 07:03

1 Answers1

3

Yeah, you can start a separate process of the other architecture, and talk to it through pipes or maybe even shared memory. But not calling into a library.

There's no calling convention for that, no way of figuring out which x86 registers should get what value after a jump or call from ARM64 code, or vice versa. (And in the general case, that could be a non-solvable problem for variadic functions, with x86-64 System V I think passing more FP args in registers than ARM64, and you wouldn't know which stack args to load into FP registers in a trampoline/shim/wrapper that would try to adapt between them, if that even makes sense because the callee would also be native ARM64 code that's just compiled from x86-64 machine code. You don't actually have x86-64 registers, although ARM64 does have more registers of the same width as the integer and vector regs Rosetta-2 supports (not AVX))

The emulation is at a whole-process level, for all of virtual address space of a whole process. Not on a per-function level to port individual functions to the standard ARM64 calling convention. Different ABIs (sizes and alignments for types, and struct layout) is also a problem.

If Rosetta-2 isn't fully ahead-of-time recompilation, i.e. if there's any run-time component like how a JVM JIT compiles but still runs the code inside a VM, then that would also have to start up when calling from an ARM64 process into an x86-64 function.

Peter Cordes
  • 328,167
  • 45
  • 605
  • 847