How I determine the number of references to a dynamic array?

Question

Following on from this question (Dynamic arrays and memory management in Delphi), if I create a dynamic array in Delphi, how do I access the reference count?

SetLength(a1, 100);
a2 := a1;
// The reference count for the array pointed to by both
// a1 and a2 should be 2. How do I retrieve this?

Additionally, if the reference count can be accessed, can it also be modified manually? This latter question is mainly theoretical rather than for use practically (unlike the first question above).

I don't particularly want to *modify* the reference count (there's no need for that), but to access it for debugging purposes (related issue, but now resolved). — magnus, Mar 05 '14 at 00:30

score 3 · Answer 1 · edited Mar 04 '14 at 08:33

3

After some googling, I found an excellent article by Rudy Velthuis. I highly recommend to read it. Quoting dynamic arrays part from http://rvelthuis.de/articles/articles-pointers.html#dynarrays

At the memory location below the address to which the pointer points, there are two more fields, the number of elements allocated, and the reference count.

enter image description here

If, as in the diagram above, N is the address in the dynamic array variable, then the reference count is at address N-8, and the number of allocated elements (the length indicator) at N-4. The first element is at address N.

How to access these:

SetLength(a1, 100);
a2 := a1;

// Reference Count = 2
refCount := PInteger(NativeUInt(@a1[0]) - SizeOf(NativeInt) - SizeOf(Integer))^;
// Array Length = 100
arrLength := PNativeInt(NativeUInt(@a1[0]) - SizeOf(NativeInt))^;

The trick in computing proper offsets is to account for differences between 32bit and 64bit platforms code. Fields size in bytes is as follows:

          32bit  64bit
RefCount  4      4
Length    4      8

edited Mar 04 '14 at 08:33

Stefan Glienke

20,860
2
48
102

answered Mar 04 '14 at 05:04

Kromster

7,181
7
63
111

But you must first test if dynamic array variable isn't nil, so that is allocated in memory! – GJ. Mar 04 '14 at 09:49
@GJ: This is an example code and of course it has many checks skipped to preserve the readability of the concept. Same way it is assumed variables are properly declared, types match, no functions overloaded, required uses included, etc. – Kromster Mar 04 '14 at 09:55
You should always test the memory pointer if exist in memory before you change the memory. – GJ. Mar 04 '14 at 11:04
2

@GJ. I honestly don't think Krom or anybody else is disputing that. The code in this answer is not meant to be production ready. Sometimes it helps the exposition to simplify code by removing all the boiler-plate error handling you need in real production code. – David Heffernan Mar 04 '14 at 11:09
@GJ. I agree with David. This clearly shows how to access the refcount. One could also define a PDynArrayRec and cast the var to that directly, subtracting one to get at the refcount. – Rudy Velthuis Mar 04 '14 at 13:34
@RudyVelthuis Shame the graphic does not support 64 bit! – David Heffernan Mar 04 '14 at 15:18
1

Yeah, I know. I noticed that too. I will add a similar graphic in a slightly different colour for 64 bit. – Rudy Velthuis Mar 04 '14 at 17:29

David Heffernan · Accepted Answer · 2014-03-04T10:39:36.437

3

You can see how the reference count is managed by inspecting the code in the System unit. Here are the pertinent parts from the XE3 source:

type
  PDynArrayRec = ^TDynArrayRec;
  TDynArrayRec = packed record
  {$IFDEF CPUX64}
    _Padding: LongInt; // Make 16 byte align for payload..
  {$ENDIF}
    RefCnt: LongInt;
    Length: NativeInt;
  end;
....
procedure _DynArrayAddRef(P: Pointer);
begin
  if P <> nil then
    AtomicIncrement(PDynArrayRec(PByte(P) - SizeOf(TDynArrayRec))^.RefCnt);
end;

function _DynArrayRelease(P: Pointer): LongInt;
begin
  Result := AtomicDecrement(PDynArrayRec(PByte(P) - SizeOf(TDynArrayRec))^.RefCnt);
end;

A dynamic array variable holds a pointer. If the array is empty, then the pointer is nil. Otherwise the pointer contains the address of the first element of the array. Immediately before the first element of the array is stored the metadata for the array. The TDynArrayRec type describes that metadata.

So, if you wish to read the reference count you can use the exact same technique as does the RTL. For instance:

function DynArrayRefCount(P: Pointer): LongInt;
begin
  if P <> nil then
    Result := PDynArrayRec(PByte(P) - SizeOf(TDynArrayRec))^.RefCnt
  else
    Result := 0;
end;

If you want to modify the reference count then you can do so by exposing the functions in System:

procedure DynArrayAddRef(P: Pointer);
asm
  JMP    System.@DynArrayAddRef
end;

function DynArrayRelease(P: Pointer): LongInt;
asm
  JMP    System.@DynArrayRelease
end;

Note that the name DynArrayRelease as chosen by the RTL designers is a little mis-leading because it merely reduces the reference count. It does not release memory when the count reaches zero.

I'm not sure why you would want to do this mind you. Bear in mind that once you start modifying the reference count, you have to take full responsibility for getting it right. For instance, this program leaks:

{$APPTYPE CONSOLE}

var
  a, b: array of Integer;

type
  PDynArrayRec = ^TDynArrayRec;
  TDynArrayRec = packed record
  {$IFDEF CPUX64}
    _Padding: LongInt; // Make 16 byte align for payload..
  {$ENDIF}
    RefCnt: LongInt;
    Length: NativeInt;
  end;

function DynArrayRefCount(P: Pointer): LongInt;
begin
  if P <> nil then
    Result := PDynArrayRec(PByte(P) - SizeOf(TDynArrayRec))^.RefCnt
  else
    Result := 0;
end;

procedure DynArrayAddRef(P: Pointer);
asm
  JMP    System.@DynArrayAddRef
end;

function DynArrayRelease(P: Pointer): LongInt;
asm
  JMP    System.@DynArrayRelease
end;

begin
  ReportMemoryLeaksOnShutdown := True;
  SetLength(a, 1);
  Writeln(DynArrayRefCount(a));
  b := a;
  Writeln(DynArrayRefCount(a));
  DynArrayAddRef(a);
  Writeln(DynArrayRefCount(a));
  a := nil;
  Writeln(DynArrayRefCount(b));
  b := nil;
  Writeln(DynArrayRefCount(b));
end.

And if you make a call to DynArrayRelease that takes the reference count to zero then you would also need to dispose of the array, for reasons discussed above. I've never encountered a problem that would require manipulation of the reference count, and strongly suggest that you avoid doing so.

One final point. The RTL does not offer this functionality through its public interface. Which means that all of the above is private implementation detail. And so is subject to change in a future release. If you do attempt to read or modify the reference count then you must recognise that doing so relies on such implementation detail.

edited Mar 04 '14 at 10:39

answered Mar 04 '14 at 09:25

David Heffernan

601,492
42
1,072
1,490

-1 Calling procedure DynArrayRelease directly is wrong in case that reference counter is 1 because memory of variable must be delocated. – GJ. Mar 04 '14 at 10:05
@GJ I don't think that I advocate doing that. In fact quite the opposite. I warn of the dangers of interfering with the reference count. Perhaps you'd like to point out the specific part of my answer where I suggest doing that? Then I can correct the mis-information. – David Heffernan Mar 04 '14 at 10:07
I think that in case if decrement reference counter to 0, than we must also call `systrm.@DynArrayClear` procedure. – GJ. Mar 04 '14 at 10:13
@GJ Please can you point out where I say otherwise, or even recommend doing this. At least my answer actually answered the question! – David Heffernan Mar 04 '14 at 10:14
However the `DynArrayRelease` procedure should never relase the memory of variable and that is wrong! – GJ. Mar 04 '14 at 10:23
@GJ. Please can you answer the question that I asked in my two comments here. It seems to me that your complaint should be addressed at the asker who is the person that wants to manipulate ref count. I strongly discourage doing so. Please can you read my answer again. – David Heffernan Mar 04 '14 at 10:24
A have read yor answer again and I disagree that your procedure `DynArrayRelease` will ever relase the variable memory, it only decrement the reference counter and nothig more. – GJ. Mar 04 '14 at 10:28
@GJ. Where in my answer do I claim that it will release memory? I have never said that! From the answer, "And if you make a call to DynArrayRelease that takes the reference count to zero then you would also need to dispose of the array." – David Heffernan Mar 04 '14 at 10:30
In that case change the procedure name `DynArrayRelease` to `DynArrayDecRef`. – GJ. Mar 04 '14 at 10:35
@GJ. Again, I suggest that you refer that comment to the author of the code, namely Embarcadero. Can't you see that this code all comes from the `System` unit? – David Heffernan Mar 04 '14 at 10:36
@GJ. Anyway, I've added a note to say that the name `DynArrayRelease` as chosen by the RTL designers is a little mis-leading. Is that enough to satisfy you? – David Heffernan Mar 04 '14 at 10:40
Yes it comes from sytem unit but it isn't public because you should never call directly this procedure. – GJ. Mar 04 '14 at 10:42
@GJ. A point that I make in my answer. I honestly cannot understand your problem. Let me try again. I recommend in my answer that the asker does not manipulate reference count. I explain that it is dangerous and give reasons why. It seems to me that my answer already agrees with everything that you said. I repeat, I do not suggest manipulating the ref count. The asker asked how to do it, I answered. If you would like to suggest to the asker that it is a bad idea, you should address your comments to the asker. What more do you want from me? – David Heffernan Mar 04 '14 at 10:47
I miss also warning that if reference counter is 1 we should not to decrement instead that we must to nil the variable so the memory shoud be delocated correctly. – GJ. Mar 04 '14 at 10:48
@GJ. It's way more complicated than that. That advice is not really any good. If the reference count is 1, that usually means that there is a single variable with a reference. However, if the programmer has meddled with the reference count, there could be any number of such variables, 0, 1, 2 or more. Can't you understand the parts of my answer that say not do to this? – David Heffernan Mar 04 '14 at 10:50
I mentioned that my latter question was purely theoretical, while the former was for a specific purpose (debugging). *"This latter question is mainly theoretical rather than for use practically (unlike the first question above)."* Thank you for the detailed answer and helping me understand the Delphi internals a little better. – magnus Mar 05 '14 at 00:33

How I determine the number of references to a dynamic array?

2 Answers2