The x86-64 architecture is pretty complex; it doesn't just have a 64-bit mode and a 32-bit mode, it has two main modes (long and legacy), each with a number of sub-modes (see the Wikipedia article).
In Legacy mode, the CPU essentially emulates a 32-bit CPU. It has various sub-modes (real, protected, etc), but no ability to switch use 64-bit instructions or addressing. Generally, a non-64-bit-capable OS will run in this mode.
In Long mode, the CPU has 64-bit capability, but can also run in 32- and 16-bit "compatibility" modes. The mode switching is controlled by the L and D flags in the code segment descriptor (see "Extending x86 for the 64-bit World" in this PDF) -- essentially, different sections of memory can be marked as containing 64-, 32-, or 16-bit code, and the CPU switches into the appropriate mode for the code segment it's currently running. The kernel's code segments can be flagged as 64- or 32-bit independently of the code segments for running applications.
So in principle, it's simple. In practice, I'm sure there are loads of complications I'm not aware of (I don't really know that much about the context switching process), but as long as the OS "knows" it's running on a 64-bit CPU and configures the code segment descriptors appropriately, there's no fundamental problem with running 64-bit processes under a 32-bit kernel.
BTW, OS X could also run 64-bit processes under a 32-bit kernel on PowerPC G5 CPUs, as far back as version 10.3. PPC CPUs have a completely different architecture, and I have no idea how the mode-switching worked there.