Some metadata tokens for local variable types invalid after rewrite with dnlib

Question

What I am doing (partly for fun and learning, partly hopefully as a serious virtualization effort someday) is merging my VM dll with a target assembly via ILMerge.

Only afterwards I modify the newly created file with dnlib to replace the method bodies of selected methods with a call to my VM function. I pass the required metadata that is now absent in the method itself via a base64 encoded binary string and obviously the arguments and the old method body too (in the future I want to implement my own bytecode instruction set for that, but so far its just the original code base64-encoded).

Since .initlocals is from my experience always set in a .NET method, what I want to do is save the type of every local as data in such a way that I can initialize my local array in the Virtualizer runtime with it.

My current approach is just to save the MDToken writer.Write(local.Type.ToTypeDefOrRef().GetNonNestedTypeRefScope().MDToken.ToInt32());

I write my changes to the assembly with the PreserveAll flag opts.MetadataOptions.Flags = dnlib.DotNet.Writer.MetadataFlags.PreserveAll;

and in the runtime resolve the MDToken via

    for (int i = 0; i < numLocals; i++)
    {
        int token = ReadInt32(info, ref pos);
        Type t = Module.ResolveType(token);
    }

Now, this only works for types defined in the modified module itself, both value (struct s {...}) and reference types(for example Form1) and also reference types defined in other modules (like System.Windows.Forms.Form)

ResolveType fails with ArgumentOutOfRangeException (token not found) for all Core CLR types (object, int32, uint64 etc.) and all value types from outside the module (like System.Drawing.Point) also from what I can see for all array types, regardless of where the underlying type is defined or referenced.

Now, why is that so? If the spec at I.9.2.1

However, a metadata token is not a persistent identifier. Rather it is scoped to a specific metadata binary.

is to be interpreted that metadata tokens become invalid when the binary is modified, why does it work for some types very consistently? And shouldn't dnlib fix this with the PreserveAll flag? And why does this problem not occur at all in the method body instructions? Many instructions encode an InlineType and Module.ResolveType has never failed there.

And, more importantly, how to fix? How do I save a reliable type identifier in a binary form for the locals of a method?

score 0 · Accepted Answer · answered May 16 '21 at 10:18

However, a metadata token is not a persistent identifier. Rather it is scoped to a specific metadata binary.

What it's saying is that the metadata token is only meaningful within the scope of the module, you can't take the metadata token from one module and use it in another or even within a modified version of the same module (or at least not reliably).

The reason for this makes more sense when you consider what the metadata token really is. The metadata token is a reference to a record in a metadata table within the module, and that record contains further details; the high-order byte of the metadata token indicates the type of token (and thus the table containing the record), while the remaining 3 bytes indicate the row number.

If you take a metadata token from one module and try to use it in another module, you are assuming that the same record in each module represents the same thing. If you compile the same code with the same compiler, then this assumption may hold; but if you change the source or use a different compiler (or different version of the same compiler) then the row numbers may change for any number of reasons.

And why does this problem not occur at all in the method body instructions? Many instructions encode an InlineType and Module.ResolveType has never failed there.

Because the compiler emitting the IL with the metadata token is also emitting the tables into the same file. The compiler is able to keep these things in sync.

And, more importantly, how to fix? How do I save a reliable type identifier in a binary form for the locals of a method?

The only reliable way to reference a type between modules, is with the full type name and scope (the containing assembly, module or type in the case of a nested type).

Some metadata tokens for local variable types invalid after rewrite with dnlib

1 Answers1