It has been a major obstacle for decade. It was reported as impossible. Forum talks referred to issues related to setting and restoring GS. Wine HQ FAQ still refers to ABI incompatibility page which is not a live wiki page, but news archive link.
Wine 2.0 announced macOS 64-bit support. But… how? Isn't it something that all macOS hackers should know? Maybe some elegant (or dirty) trick interesting on its own for any x86-64 hacker.
The primary obstacle is a conflict over the GS segment base address (GS.base) maintained by the CPU under the control of the OS.
On 64-bit Windows, GS.base is used to hold the address of the Thread Environment Block (TEB) structure for each thread. Windows apps expect to access the TEB using %gs-relative addresses. This is hard-coded into the app code rather than being behind an API function.
On macOS, GS.base is used to hold the base of the thread-local storage area of the thread's struct _pthread, an internal implementation detail of the Pthreads implementation. It's less common for Mac apps to have hard-coded %gs-relative accesses baked into them, but some do and so do the system libraries.
On Linux, GS.base is available for 64-bit apps to use for their own purpose. So, there, Wine simply sets it using the OS-provided mechanism. Wine can't do that on macOS. Not only does the OS not provide any mechanism to do it but, if Wine could, it would break the system libaries. (It would also pose potential problems for the kernel on context switches and/or the kernel might fail to restore any value Wine might have set.)
The solution we figured out is only a partial solution. The most commonly accessed fields of the TEB structure are the "self" field (%gs:0x30) and a field for the thread-local storage implementation (%gs:0x58). Often, if apps need to access other fields, they first read the self field and then reference off of that.
On macOS, %gs:0x30 and %gs:0x58 correspond to particular slots of the thread-local storage area. They are in a part that's reserved to Apple (rather than, say, application uses). We found that one of those slots was unused. The other was used for the ttyname() function in the C library. As it happens, Wine never calls that function and there's little reason to expect any of the system libraries that it uses to do so, either.
So, Wine simply pokes the appropriate values at those %gs-relative locations. Therefore, when 64-bit Windows app code reads them, it gets what it needs. The actual TEB that Wine has allocated is located elsewhere (in heap-allocated memory), but apps find the address of the TEB in the place they expect to be the TEB self field, so they find it that way.
Apple has since graciously permanently reserved both of those slots for uses like Wine's. ttyname() now uses a different slot.
That said, as mentioned above, this solution is only partial. Some apps access other fields of the TEB directly using %gs-relative addresses at offsets other than 0x30 or 0x58. When they do so, they get junk values and/or overwrite values used by other parts of the system. So, Wine's support for 64-bit Windows apps is not complete on macOS. Some such apps will crash or otherwise misbehave. Luckily, it happens infrequently enough that it's not much of a problem in practice.
For reference, here are the commits that implement this solution:
http://source.winehq.org/git/wine.git/?a=commit;h=7501942008f91a9a137fe598ce5ce7cb47de5522 http://source.winehq.org/git/wine.git/?a=commit;h=3d8efb238808a519902e047d8673237debb0f0a2
