Passing a struct pointer in C# interop results in NULL

Question

I am building a managed DLL for use in unmanaged environment (C/C++ app - FreeRDP). Interop works fine in most cases, but in one particular I am not able to pass a pointer to struct. In the API I have a struct:

typedef struct _IWTSListenerCallback IWTSListenerCallback;
struct _IWTSListenerCallback
{
    UINT(*OnNewChannelConnection)(IWTSListenerCallback* pListenerCallback,
                              IWTSVirtualChannel* pChannel,
                              BYTE* Data,
                              BOOL* pbAccept,
                              IWTSVirtualChannelCallback** ppCallback);
};

As well as a function I am calling:

UINT(*CreateListener)(IWTSVirtualChannelManager* pChannelMgr,
                      const char* pszChannelName,
                      ULONG ulFlags,
                      IWTSListenerCallback* pListenerCallback);

Both translated to C#:

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate uint ListenerCallbackNewConnectionDelegate(IntPtr listenerCallback, IntPtr channel, [MarshalAs(UnmanagedType.LPArray)] byte[] data, IntPtr accept, ref IntPtr channelCallback);

[StructLayout(LayoutKind.Sequential)]
public struct IWTSListenerCallback
{
    [MarshalAs(UnmanagedType.FunctionPtr)]
    public ListenerCallbackNewConnectionDelegate OnNewChannelConnection;
}

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate uint ChannelManagerCreateListenerDelegate(IntPtr channelManager, [MarshalAs(UnmanagedType.LPStr)] string channelName, ulong flags, IntPtr listenerCallback);

[MarshalAs(UnmanagedType.FunctionPtr)]
public ChannelManagerCreateListenerDelegate CreateListener;

And execution code:

var callback = new IWTSListenerCallback();
callback.OnNewChannelConnection = NewChannelConnection;
var pCallback = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(IWTSListenerCallback)));
Marshal.StructureToPtr(callback, pCallback, false);
var ret = channelManager.CreateListener(pChannelManager, "TestChannel", 0, pCallback);

And while pChannelManager (which is a pointer I obtain from unmanaged code calling my DLL) and the string are sent through without any problems, the pointer I create here (pCallback) is assigned successfuly in C#, but it results in a NULL in unmanaged code.

I assume the problem is either with how I defined the struct, or how I defined the function (although the function is being called successfuly in unmanaged code). I use the method to create the pointer in exact same way as in another part of the DLL, and it works perfectly fine there when passed to unmanaged function.

EDIT: By @jdweng suggestion:

[StructLayout(LayoutKind.Sequential)]
public struct TestCall
{
    public IntPtr channelManager;
    [MarshalAs(UnmanagedType.LPStr)]
    public string channelName;
    public ulong flags;
    public IntPtr listenerCallback;
}

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate uint ChannelManagerCreateListenerDelegate(IntPtr testStructure);

var test = new TestCall();
test.channelManager = pChannelManager;
test.channelName = "TestChannel";
test.flags = 0;
test.listenerCallback = pCallback;
var pTest = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(FreeRDPTypes.TestCall)));
Marshal.StructureToPtr(test, pTest, false);
var ret = channelManager.CreateListener(pTest);

Didn't work.

EDIT2: Workaround! Only if you have access to original unmanaged code. I rearranged the function arguments so the structure pointers are first, like this:

UINT(*CreateListener)(IWTSVirtualChannelManager* pChannelMgr,
                      IWTSListenerCallback* pListenerCallback,
                      const char* pszChannelName,
                      ULONG ulFlags);

And it works! Probably a problem with offset.

You need a c# class for OnNewChannelConnection. The callback is going to return a pointer to the structure. You should specify c language parameter calls the default is windows calling convention. In c language the parameter list is pushed in reverse order onto the stack. Which would be backwards from the structure order. For you code to work I think you need to change the order of the parameter list. — jdweng, Oct 18 '17 at 10:27
@jdweng I'm not sure I understood correctly, but the methods are specified with the __cdecl calling convention (through the delegates). Also all other parameters are going through without any distortion, only this one pointer. — Grzegorz Puławski, Oct 18 '17 at 10:39
I missed the cdecl. Sorry. The ListenerCallbackNewConnectionDelegate method is the only one passing a parameter list. A structure has the elements in order of the declarations. A parameter list when calling a function also creates a structure in order of the parameters.. The stack starts at top of memory and moves down so the parameters appear in the same order as the structure (I was wrong earlier since I forgot the stack starts at top of memory). The problem is when the function returns those values are on the stack which get lost. I will work when calling, but not work when returning. — jdweng, Oct 18 '17 at 12:27
The answer is to create a c# structure for CreateListener and pass the pointer to this structure. Then get rid of the parameters list for ListenerCallbackNewConnectionDelegate and just pass the strucuure pointer. — jdweng, Oct 18 '17 at 12:28
@jdweng please see edit in the Question if that's what you suggested. If yes, this unfortunately results in garbage values in all parameters in unmanaged code. — Grzegorz Puławski, Oct 18 '17 at 13:10
Where is the memory allocation for pChannelMgr, pszChannelName, and ppCallback? Right now you just have null pointers. — jdweng, Oct 18 '17 at 13:32
@jdweng in third part of the edit. I create a new TestCall then add the necessary objects, then allocate memory for the pointer using `Marshal.AllocHGlobal()` and then copy the struct into allocated memory via `Marshal.StructureToPtr()` — Grzegorz Puławski, Oct 18 '17 at 13:44
Make sure the pointers for pChannelMgr, pszChannelName, and ppCallback are in unmanaged memory. What you are doing is putting the pointers in managed memory. — jdweng, Oct 18 '17 at 13:53

score 1 · Answer 1 · answered Oct 19 '17 at 21:47

It was a matter of offset. C/C++ ULONG was typedef unsigned long which I wrongly assumed corresponded to C# ulong, but in fact the first one is 4 bytes in Visual, while the other is 8 bytes, which resulted in 4 bytes offset. Fixed by changing ulong to uint and adding [MarshalAs(UnmanagedType.U4)] for good measure. Final look of the function I was calling inside C#:

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate uint ChannelManagerCreateListenerDelegate(IntPtr channelManager, [MarshalAs(UnmanagedType.LPStr)] string channelName, [MarshalAs(UnmanagedType.U4)] uint flags, IntPtr listenerCallback);

Passing a struct pointer in C# interop results in NULL

1 Answers1