Having trouble serializing these values in C# to be properly read in C++ as a known struct

Question

We've been given a program from another organization that collects data from multicast sources and collates and saves that data. It expects a C++ struct formatted as such:

#define SP_PACKET_SIZE 200
#define NAME_SIZE 64
struct spPacketStruct
{
    int Size;
    char Name[SP_PACKET_SIZE][NAME_SIZE];
    double Value[SP_PACKET_SIZE];
};

obviously I can't use this struct in C# because a struct can't have preinitialized arrays, so I figured create individual bits and just serialize them. So now I have this in C#:

int SpPacketSize;
char[,] SpNames = new char[SP_PACKET_SIZE, NAME_SIZE];
double[] SpValues = new double[SP_PACKET_SIZE];

My previous experience is with a BinaryWriter...I don't need to deserialize in C#, I just need to get it to the C++ program. My test serialization code is as follows:

System.IO.MemoryStream outputstream = new System.IO.MemoryStream();

BinaryFormatter serializer = new BinaryFormatter();
serializer.TypeFormat = System.Runtime.Serialization.Formatters.FormatterTypeStyle.TypesWhenNeeded;

serializer.Serialize(outputstream, SpPacketSize);
serializer.Serialize(outputstream, SpNames);
serializer.Serialize(outputstream, SpValues);

byte[] buffer = outputstream.GetBuffer();

udpclient.Send(buffer, buffer.Length, remoteep);

And I get a binary packet but the length isn't right because it's still including the type formats. When I look at this packet in Wireshark I see a System.Int32 notation in the beginning of it. This is making the packet larger than expected and thus not deserialized properly on the C++ side.

I added the TypesWhenNeeded TypeFormat thinking I could minimize it, but it didn't change...and I noticed there was no option to not TypeFormat, unless I missed it somewhere.

Does anyone have any hints on how to properly serialize this data without the extra info?

https://stackoverflow.com/questions/13853129/how-can-i-write-and-read-using-a-binarywriter is what you are looking for... — Alexei Levenkov, Jul 18 '19 at 18:36
That doesn't really seem to help, no...the problem isn't that I need to prepend something to the binary output, the problem is I need the TypeFormat to NOT be there. The first dat apoint of int, is the packet size count and tells the C++ program how many name/double pairs to attempt reading from the packet. — Robbie P., Jul 18 '19 at 18:47
Note: using strictures as a data protocol can be troublesome. `int` is not always the same size or byte order, for example. Padding can inject different amounts of deadspace. Read data, parse and and place into a structure, that's cool, but read directly into structure is a great way to get nasty surprises. — user4581301, Jul 18 '19 at 18:48
Establish and implement your own protocol on both sides. Use helper classes only if they do exactly what you want. Have you considered using a tool like Protocol Buffers to do the grunt work for you? — user4581301, Jul 18 '19 at 18:50
I don't have any control over the receive side. I don't have that code. They did send me a c++ class for sending to it in c++ which works fine in C++ but I need to be able to send to it from C#. I'm just now fooling around with sharpserializer but it actually made the packet way larger and blew out my socket buffer. I feel like this shouldn't be this difficult...I just need to get the 1's and zero's in the right order...I can share the code used to fill the struct in the C++ version if that would be helpful? — Robbie P., Jul 18 '19 at 18:56
@RobbieP. that's unfortunate that you can't use `BinaryWriter`... Wish you luck finding library that works for you (also I'd caution to be careful about most serialization solutions as they tend to write at least some metadata) — Alexei Levenkov, Jul 18 '19 at 19:08
yeah it's the metadata I' dlike to get rid of...is there a method to write binary without metadata? — Robbie P., Jul 18 '19 at 19:10

Sohaib Jundi · Accepted Answer · 2019-07-19T07:06:12.557

While you cannot pre-initialize struct fields, nor directly put the size in MarshalAs attribute for ByValue 2D array, there is a little work-around you can do. You can define two structs like this:

const short SP_PACKET_SIZE = 200;
const short NAME_SIZE = 64;

struct spPacketStruct
{
  public int Size;
  [MarshalAs(UnmanagedType.ByValArray, SizeConst = SP_PACKET_SIZE)]
  private fixedString[] names;
  public fixedString[] Names { get { return names ?? (names = new fixedString[SP_PACKET_SIZE]); } }
  [MarshalAs(UnmanagedType.ByValArray, SizeConst = SP_PACKET_SIZE)]
  private double[] value;
  public double[] Value { get { return value ?? (value = new double[SP_PACKET_SIZE]); } }
}

struct fixedString
{
  [MarshalAs(UnmanagedType.ByValTStr, SizeConst = NAME_SIZE)]
  public string Name;
}

By having additional struct to be a member of the original struct, you are able to specify the length of both dimensions by setting SizeConst in the original struct to the first dimension and setting it to the second dimension in the new struct. Making the field private and creating properties for them is merely for convenience, so you don't have to assign the array yourself when creating the struct.
Then you can serialize/deserialize the struct like this (code from this answer: https://stackoverflow.com/a/35717498/9748260):

public static byte[] GetBytes<T>(T str)
{
  int size = Marshal.SizeOf(str);
  byte[] arr = new byte[size];
  GCHandle h = default;

  try
  {
    h = GCHandle.Alloc(arr, GCHandleType.Pinned);
    Marshal.StructureToPtr(str, h.AddrOfPinnedObject(), false);
  }
  finally
  {
    if (h.IsAllocated)
    {
      h.Free();
    }
  }

  return arr;
}

public static T FromBytes<T>(byte[] arr) where T : struct
{
  T str = default;
  GCHandle h = default;

  try
  {
    h = GCHandle.Alloc(arr, GCHandleType.Pinned);
    str = Marshal.PtrToStructure<T>(h.AddrOfPinnedObject());
  }
  finally
  {
    if (h.IsAllocated)
    {
      h.Free();
    }
  }

  return str;
}

And one last thing when trying to serialize/deserialize structs like this, be aware of the struct alignment as it can mess with the struct size

I'll give this a shot in the morning when I get in to the office...Very promising, thank you! Upvoted and I'll mark as accepted once I test. — Robbie P., Jul 19 '19 at 02:30
this may be a dumb question, but how do I properly populate the values of the struct? Do I keep populating my non-struct variables and copy them into the struct somehow? Also, will there be a difference in serialization of string versus char array? — Robbie P., Jul 19 '19 at 14:17
m_spPacket.Name[0].Name = "Hello"; m_spPacket.Value[0] = 12345; — Robbie P., Jul 19 '19 at 14:22
@RobbieP. Yes like this, you missed an 's' here `m_spPacket.Name's'[0]`. I assumed that the char array is representing a string, since it contains names, so I went ahead and did it that way. If you really need it to be char array then change the second struct contents to: `[MarshalAs(UnmanagedType.ByValArray, SizeConst = NAME_SIZE)] public char[] Name;` — Sohaib Jundi, Jul 19 '19 at 15:19
you'll be pleased to know I got it working with your suggestions! I had to put a (GCHandle) after the default initializers, but other than that, it works beautifully and I'm going to go home and enjoy my weekend not stressing out about this problem anymore! — Robbie P., Jul 19 '19 at 18:47

Having trouble serializing these values in C# to be properly read in C++ as a known struct

1 Answers1