C# "A heap has been corrupted" using sam-ba.dll

Question

I'm writing a C# program that makes a call to the AT91Boot_Scan function in sam-ba.dll. In the documentation for this DLL, the signature for this function is void AT91Boot_Scan(char *pDevList). The purpose of this function is to scan and return a list of connected devices.

Problem: My current problem is that every time I call this function from C#, the code in the DLL throws an a heap has been corrupted exception.

Aside: From what I understand from reading the documentation, the char *pDevList parameter is a pointer to an array of buffers that the function can use to store the device names. However, when calling the method from C#, IntelliSense reports that the signature for this function is actually void AT91Boot_Scan(ref byte pDevList)

I was sort of confused by why this is. A single byte isn't long enough to be a pointer. We would need 4 bytes for 32-bit and 8 bytes for 64-bit... If it's the ref keyword that is making this parameter a pointer, then what byte should I be passing in? The first byte in my array of buffers or the first byte of the first buffer?

Code: The C# method I've written that calls this function is as follows.

/// <summary>
    /// Returns a string array containing the names of connected devices
    /// </summary>
    /// <returns></returns>
    private string[] LoadDeviceList()
    {
        const int MAX_NUM_DEVICES = 10;
        const int BYTES_PER_DEVICE_NAME = 100;
        SAMBADLL samba = new SAMBADLL();
        string[] deviceNames = new string[MAX_NUM_DEVICES];
        try
        {
            unsafe
            {
                // Allocate an array (of size MAX_NUM_DEVICES) of pointers 
                byte** deviceList = stackalloc byte*[MAX_NUM_DEVICES];

                for (int n = 0; n < MAX_NUM_DEVICES; n++)
                {
                    // Allocate a buffer of size 100 for the device name
                    byte* deviceNameBuffer = stackalloc byte[BYTES_PER_DEVICE_NAME];

                    // Assign the buffer to a pointer in the deviceList
                    deviceList[n] = deviceNameBuffer;
                }

                // Create a pointer to the deviceList
                byte* pointerToStartOfList = *deviceList;

                // Call the function. A heap has been corrupted error is thrown here.
                samba.AT91Boot_Scan(ref* pointerToStartOfList);

                // Read back out the names by converting the bytes to strings
                for (int n = 0; n < MAX_NUM_DEVICES; n++)
                {
                    byte[] nameAsBytes = new byte[BYTES_PER_DEVICE_NAME];
                    Marshal.Copy((IntPtr)deviceList[n], nameAsBytes, 0, BYTES_PER_DEVICE_NAME);
                    string nameAsString = System.Text.Encoding.UTF8.GetString(nameAsBytes);
                    deviceNames[n] = nameAsString;
                }
            }
        }
        catch (Exception e)
        {
            Console.WriteLine(e.Message);
        }
        return deviceNames;
    }

My attempt at a solution: I noticed that the line byte* pointerToStartOfList = *deviceList; wasn't correctly assigning the pointer for deviceList to pointerToStartOfList. The address was always off by 0x64.

I thought if I hard-coded in a 0x64 offset then the two addresses would match and all would be fine. pointerToStartOfList += 0x64;

However despite forcing the addresses to match I was still getting a a heap has been corrupted error.

My thoughts: I think in my code I'm either not creating the array of buffers correctly, or I'm not passing the pointer for said array correctly .

Is MAX_NUM_DEVICES definitely large enough? What you've done so far looks ok as far as I can tell from the documentation... Only thing, perhaps it shouldn't be `ref*` and you should just pass the pointer. — Matthew Watson, Mar 02 '17 at 16:33
Yes, definitely. The documentation for sam-ba.dll doesn't explicitly say, but they provided a code example where their number of devices was 5 (search for `CHAR *strConnectedDevices[5];`) I tried to copy their code example but it seems to be written in `C++` and not `C#` — Calseon, Mar 02 '17 at 16:36
This library was not designed very well and is really only suitable to be called from a C++ program. The actual argument type is `string`, not `ref byte`. Technically you can fix it by decompiling the interop library with ildasm.exe, modify the .il file to fix the argument type and putting it back together with ilasm.exe — Hans Passant, Mar 02 '17 at 16:47
@ Matthew Watson: Yeah... I thought the `ref*` part was weird too, but If I don't do it you'll notice that `pointerToStartOfList` is actually a `byte*` and not `byte` type. So I end up with a `cannot convert from 'ref byte*' to 'ref byte'` error. If I try casting it to byte then I get `A ref or out value must be an assignable variable` error. — Calseon, Mar 02 '17 at 16:48
@Hans Passant: Aww man, really? That's a real shame >.< Thank you though I'll try that. If I don't have any luck with that I'll try writing my own wrapper class in C++ that I can then call from C#...Ultimately, I really do need to put this in a C# program. — Calseon, Mar 02 '17 at 16:56
@Calseon We have often had to wrap "troublesome" C/C++ libraries in a more sensible 'C' wrapper to make it possible to call it from C#. — Matthew Watson, Mar 02 '17 at 17:32

Calseon · Accepted Answer · 2017-03-09T21:31:48.697

In the end I could not get sam-ba.dll to work. I had tried writing a C++ wrapper around the DLL but even then it was still throwing the a heap has been corrupted error. My final solution was to just embed the SAM-BA executable, sam-ba.exe, along with all of its dependencies inside my C# program.

Then, whenever I needed to use it I would run sam-ba.exe in command line mode and pass it the relevant arguments. Section 5.1 in the SAM-BA documentation provides instructions on how to run sam-ba.exe in command line mode.

i.e.

SAM-BA.exe \usb\ARM0 AT91SAM9G25-EK myCommand.tcl

C# "A heap has been corrupted" using sam-ba.dll

1 Answers1