1

My problem:

I need to encode additional information about an object in a pointer to the object. What I thought I could do is use part of the pointer to do so. That is, use a few bits encode bool flags. As far as I know, the same thing is done with certain types of handles in the windows kernel.

Background:

I'm writing a small memory management system that can garbage-collect unused objects. To reduce memory consumption of object references and speed up copying, I want to use pointers with additional encoded data e.g. state of the object(alive or ready to be collected), lock bit and similar things that can be represented by a single bit.

My question:

How can I encode such information into a 64-bit pointer without actually overwriting the important bits of the pointer?

Since x64 windows has limited address space, I believe, not all 64 bits of the pointer are used, so I believe it should be possible. However, I wasn't able to find which bits windows actually uses for the pointer and which not. To clarify, this question is about usermode on 64-bit windows.

Thanks in advance.

thebear8
  • 194
  • 2
  • 11
  • If you make sure that your pointers always point to even addresses then the bottom bit will always be zero and therefore available for other purposes. This is a standard technique. Heap memory allocation typically returns pointers aligned to eight byte multiples giving three bits to play with. – john May 25 '20 at 18:21
  • All bits of a pointer are "important". With processes running in a virtual memory space, a valid pointer can be anything. – Sam Varshavchik May 25 '20 at 18:22
  • @SamVarshavchik So there are no unsused bits? – thebear8 May 25 '20 at 18:25
  • @SamVarshavchik You'll need perform bit wise operations to mask your use of the "non-significant" bits such that only the original pointer is every dereferenced. This seems awfully brittle and non-portable. – François Andrieux May 25 '20 at 18:26
  • There are no bits that are guaranteed to be unused, in perpetuity. – Sam Varshavchik May 25 '20 at 18:27
  • @FrançoisAndrieux The way I intend to use the encoded pointers is in a sort of "handle table" for allocated objects, so these pointers wouldn't be used directly. It's true that it won't be very portable though. – thebear8 May 25 '20 at 18:38
  • 2
    In practice, your object will probably require 8-byte alignment, so the bottom 3 bits will always be zero, and you can hide information there. For the upper bits, you can call `Get­System­Info` and look at the `lp­Maximum­Application­Address` to see the number of bits that are being used by the OS. Of course, you have to be prepared for the case that there aren't enough bits for what you need and use a fallback mechanism in that case. – Raymond Chen May 25 '20 at 20:17

1 Answers1

2

This is heavily dependent on the architecture, OS, and compiler used, but if you know those things, you can do some things with it.

x86_64 defines a 48-bit1 byte-oriented virtual address space in the hardware, which means essentially all OSes and compilers will use that. What that means is:

  • the top 17 bits of all valid addresses must be all the same (all 0s or all 1s)
  • the bottom k bits of any 2k-byte aligned address must be all 0s
  • in addition, pretty much all OSes (Windows, Linux, and OSX at least) reserve the addresses with the upper bits set as kernel addresses -- all user addresses must have the upper 17 bits all 0s

So this gives you a variety of ways of packing a valid pointer into less than 64 bits, and then later reconstructing the original pointer with shift and/or mask instructions.

If you only need 3 bits and always use 8-byte aligned pointers, you can use the bottom 3 bits to encode extra info, and mask them off before using the pointer.

If you need more bits, you can shift the pointer up (left) by 16 bits, and use those lower 16 bits for information. To reconstruct the pointer, just right shift by 16.

To do shifting and masking operations on pointers, you need to cast them to intptr_t or int64_t (those will be the same type on any 64-bit implementation of C or C++)

1There's some hints that there may soon be hardware that extends this to 56 bits, so only the top 9 bits would need to be 0s or 1s, but it will be awhile before any OS supports this

Chris Dodd
  • 119,907
  • 13
  • 134
  • 226