c++ to VB.Net IntPtr Strings

Question

Alright so I have this code, and I pass it to an unmanaged dll, to which I only know the exports, and have some sample code. I'm getting back the correct string, but it's followed by garbage bytes.

I'm basically translating code verbatim from a c++ example program that doesn't have this issue. I'm assume there is something fundamental I am missing here, so if anyone could tell me what that is, I'd appreciate it.

Example C++ Code

void CDUKPT_TESTDlg::OnButton4() 
{
    // TODO: Add your control notification handler code here
    unsigned char dataout[1024],tmp[1024],ksn[20],keyval[20];
    int nRet,len;
    memset(dataout,0,sizeof(dataout));
    memset(ksn,0,sizeof(ksn));
    memset(keyval,0,sizeof(keyval));
    memset(tmp,0,sizeof(tmp));
    UpdateData(TRUE);

    two_one((unsigned char *)m_strCURKSN.GetBuffer(m_strCURKSN.GetLength()),m_strCURKSN.GetLength(),ksn);
    two_one((unsigned char *)m_strMACK.GetBuffer(m_strMACK.GetLength()),m_strMACK.GetLength(),keyval);
    two_one((unsigned char *)m_EncryptDat.GetBuffer(m_EncryptDat.GetLength()),m_EncryptDat.GetLength(),dataout);

    len=m_EncryptDat.GetLength()/2;
    //extern int __stdcall ExtractDat(unsigned char *input,
    //unsigned short len,unsigned char *output,unsigned char *key,
    //unsigned char *ksn);
    nRet=ExtractDat(dataout,len,tmp,keyval,ksn); //External Call
    //Good string+bad trailing data comes back in tmp
    m_Result=tmp;
    UpdateData(FALSE);
}

This code spits out this Ü‰Òdÿo

Here is my VB.Net Code

Public Function Encrypt(ByVal inp As String) As String
    Dim tmpSB As New StringBuilder
    Dim i As Integer
    Dim tKsn As Char() = TwoOne(StrCurKsn)
    For i = tKsn.Length To 19
        tKsn = tKsn + Chr(0)
    Next
    Dim tMack As Char() = TwoOne(StrMack)
    For i = tMack.Length To 19
        tMack = tMack + Chr(0)
    Next
    Dim tEnc As Char() = TwoOne(inp)
    For i = tEnc.Length To 1023
        tEnc = tEnc + Chr(0)
    Next
    Dim len As Integer = tEnc.Length / 2

    Dim tmpStr(1023) As Char
    Array.Clear(tmpStr, 0, 1024)
    Dim tmpPtr = Marshal.StringToHGlobalAnsi(tmpStr)

    Dim nRet = ExtractDat(tEnc, len, tmpPtr, tMack, tKsn)

    tmpStr = Marshal.PtrToStringAnsi(tmpPtr)
    Dim tsl = tmpStr.Length
    Encrypt = tmpStr
End Function

This code spits this out

Ü‰Òdÿo ålUäÙålUäÙålUäÙålUäÙålUäÙålUäÙålUäÙ

So I get the right string, but it's followed by a repeating string of garbage characters. I'm hoping I've done something blatantly wrong here, but I've tried pulling the data as bytes, and chars, and converting in many different methods, and I can't seem to get rid of those characters...Also, ExtractDat doesn't return the length of the string(not a problem, as it's not supposed to, which is really annoying).

check that `m_EncryptDat.GetLength()` and `tEnc.Length / 2` are returning the same thing, cause I have feeling that they aren't though its just a hunch, Also those printf work the same in VB as it does in C where it stops at the '\0' or does it display display the whole thing since its an array, trying printing more chars in the C code and see if there any diff from the VB code. — Samy Vilar, Jun 15 '12 at 05:34
The array is filled with null bytes in the c++ string, after the data I wanted back. — Kelly Elton, Jun 15 '12 at 05:38
Both end up being two(m_EncryptDat.GetLength() and tEnc.Length/2) — Kelly Elton, Jun 15 '12 at 05:45
well like I thought VB strings aren't like C strings where there '\0' terminated http://www.lansa.com/support/tips/t0002.htm and http://msdn.microsoft.com/en-us/library/aa263531(v=vs.60).aspx so something is affecting the length of string, though I have no idea what. — Samy Vilar, Jun 15 '12 at 06:00
If you use a byte buffer instead of a string ptr, can you confirm that you are seeing the null byte where you expect it? Worst case scenario, use the byte buffer, seek the first null byte, and then use System.Text.Encoding.ASCII or .Default to convert the bytes to a string. — tcarvin, Jun 15 '12 at 14:03
@tcarvin I've passed a char() and a byte() into it with the exact same results, and if I seek the first null byte, it's at the offset 512 after all the garbage — Kelly Elton, Jun 15 '12 at 17:18
Interesting...Your C++ code assigns the 1024 byte tmp to m_Result. What are you using output m_Result to get `Ü‰Òdÿo `? You are sure that m_Result contains a null at position 6? — tcarvin, Jun 15 '12 at 18:08
Yeah. That's the last step it goes through, from there it displays in a winforms textbox, but I've debugged and looked at the data in it as an array. — Kelly Elton, Jun 15 '12 at 18:37

score 0 · Accepted Answer · answered Sep 12 '12 at 02:24

0

Turns out the dll was bad, so after I got a fresh compile from the vendor it seemed to work.

answered Sep 12 '12 at 02:24

Kelly Elton

4,373
10
53
97

c++ to VB.Net IntPtr Strings

1 Answers1