The problem is probably in your first struct. The C++ definition includes a pointer to a data buffer, but the Marshaller can't directly convert this to a .NET byte[] (or, going the other direction it probably converts the byte[] to an invalid pointer, thus your Illegal Params error). Instead, you can do this manually in two steps:
[StructLayout(LayoutKind.Sequential)]
public struct DATA_BUFFER
{
public IntPtr buffer;
public uint length;
}
And then read the buffer manually using Marshal (this is just a quick sample from my memory of the Marshal API):
var txBufBytes = new byte[pComm.TxBuf.length];
Marshal.Copy(pComm.TxBuff.buffer, 0, pComm.TxBuf.length);
In addition to that, as @svick mentioned in the comments, you will probably need to set the CharSet to CharSet.Ansi if your native code is assuming non-unicode/wide characters.
Finally, the C++ definition of the second struct seems to define single-element arrays which might in turn actually be pointers, rather than having the structs in memory. If that is the case you will probably have to replace your definition to use IntPtrs for the interop and then use Marshal.PtrToStructure to get the actual structures.
You should at least compare that the struct sizes are the same by comparing the results of sizeof(...) in C++ with Marshal.SizeOf(...) in C# for the struct definitions.
Given what you've said in the comments, you should be able to use the DATA_BUFFER modification I described above, and the original struct definition you used for JVM_COMM_BUFFER
[StructLayout(LayoutKind.Sequential)]
struct JVM_COMM_BUFFER
{
public DATA_BUFFER TxBuf;
public DATA_BUFFER RxBuf;
}
Combine this with a slight modification to your DllImport
[DllImport("jhi.dll", CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Ansi)]
public static extern UInt32 SendAndRecv(IntPtr handle, string AppId, [In, Out] ref JVM_COMM_BUFFER pComm);
The CharSet.Ansi is important for ensuring your .NET string gets marshalled correctly to an ansi string (assuming your native C function doesn't expect a wchar_t string type). The [In, Out] attributes might not be required, but might hint to the marshaller how to control that parameter properly.
If the JVM_COMM_BUFFER is truly INOUT, and you're pre-populating it with data before calling the function, you may need to make sure the data is all valid. The function you're calling may have documentation on what values it expects its parameters to have. However, the definitions here should marshal correctly based on the C++ definitions you provided.
