Call to MultiByteToWideChar() in 64-bit office gives wrong result

Question

I have an Excel VBA project I'm in the process of adapting for 64-bit Office. In one part, I make calls to MultiByteToWideChar() using any of 20 or so different code pages. (So StrConv is not an alternative.)

This has been working for me for years in 32-bit Office using the following declare:

Declare Function MultiByteToWideChar Lib "kernel32" ( _
  ByVal codepage As Long, _
  ByVal dwFlags As Long, _
  lpMultiByteStr As Any, _
  ByVal cbMultiByte As Long, _
  ByVal lpWideCharStr As Long, _
  ByVal cchWideChar As Long _
  ) As Long
    'params: UINT, DWORD, LPCSTR, int, LPWSTR, int
    'return: int

But my adaptation to 64-bit is not: I get wrong results (e.g. an empty string where a non-empty string is expected), and frequent crashes. I'm using a declare that I got from the Microsoft-provided Win32API_PtrSafe.TXT file. (Of course, it could have bugs.)

So, I'm guessing something is not right in the declare statement or in how I'm making the call.

Here's a minimal sample that repro's:

'Windows API declarations

Public Const MB_PRECOMPOSED = &H1       'use precomposed chars

Declare PtrSafe Function MultiByteToWideChar Lib "kernel32" ( _
  ByVal CodePage As Long, _
  ByVal dwFlags As Long, _
  ByVal lpMultiByteStr As String, _
  ByVal cchMultiByte As Long, _
  ByVal lpWideCharStr As String, _
  ByVal cchWideChar As Long _
  ) As Long
    'params: UINT, DWORD, LPCSTR, int, LPWSTR, int
    'return: int


' My function that calls MultiByteToWideChar

Private Function EncodedStringByteArrayToString(abStringData() As Byte, lngArrLen As Long, CodePage As Long) As String
    Dim lngStrLen As Long, str As String

    lngStrLen = MultiByteToWideChar(CodePage, MB_PRECOMPOSED, ByVal VarPtr(abStringData(1)), lngArrLen, 0&, 0)
    str = String(lngStrLen, " ")
    lngStrLen = MultiByteToWideChar(CodePage, MB_PRECOMPOSED, ByVal VarPtr(abStringData(1)), lngArrLen, StrPtr(str), lngStrLen)
    EncodedStringByteArrayToString = str
End Function


' Sample routine to produce repro

Private Sub TestMB2WCBug()
    Dim abStringData(1 To 9) As Byte
    Dim resultString As String

    abStringData(1) = 67
    abStringData(2) = 111
    abStringData(3) = 112
    abStringData(4) = 121
    abStringData(5) = 114
    abStringData(6) = 105
    abStringData(7) = 103
    abStringData(8) = 104
    abStringData(9) = 116

    resultString = EncodedStringByteArrayToString(abStringData(), 9, 10000)
End Sub

The comment doesn't match the formal parameters. A `String` cannot be an `LPCSTR` and `LPWSTR` at the same time. The function expects raw data buffers. The answer to [your previous question](https://stackoverflow.com/q/62107542/1889329) applies here as well. — IInspectable, Jun 01 '20 at 05:43
The comments after the declare were added a long time ago based on MSDN doc'n at the time. I edited the question to show the ```declare``` I had been using for 32-bit O. The new declare was simply taken from Win32API_PtrSafe.TXT, but the comments weren't changed. So it was the new, buggy declare that got out of sync. — Peter Constable, Jun 01 '20 at 15:13
@PeterConstable Yes, but the old declare wouldn't cut it either. `lpWideCharStr` must be `LongPtr`. — GSerg, Jun 01 '20 at 15:16
@GSerg Of course: the old declare was written for and only used on 32-bit Office. — Peter Constable, Jun 01 '20 at 15:30
64-bit vs. 32-bit is of no importance here. This issue is about character encoding, not pointer sizes. It was just as wrong for 32-bit Office as it is for 64-bit Office. — IInspectable, Jun 01 '20 at 16:05
@IInspectable The one found in the first revision of the question was wrong, but the currently displayed one would work for 32-bit Office because the pointer size equals to that of `Long`. — GSerg, Jun 01 '20 at 17:20

score 4 · Accepted Answer · answered Jun 01 '20 at 07:29

This has been working for me for years in 32-bit Office

It could not possibly work with the Declare that you have shown.

MultiByteToWideChar expects an LPWSTR as the output buffer. VB performs automatic conversion from Unicode to ANSI when passing strings into Declared functions, so there is no way that the function would receive a pointer to a wide string buffer when lpWideCharStr is declared As String. At best, it would receive a buffer that is large enough so no buffer overflow would occur, and then VB would perform conversion back to Unicode when returning from the function, so you will end up with a double-unicode string.

lpMultiByteStr is not a string either, it's an array of bytes in some encoding.

The code inside EncodedStringByteArrayToString seems to know all that, because it correctly passes a byte array for lpMultiByteStr and an StrPtr for lpWideCharStr. This could have not happened with the current declaration of MultiByteToWideChar.

The declaration that is assumed by the code in EncodedStringByteArrayToString is:

Declare PtrSafe Function MultiByteToWideChar Lib "kernel32" ( _
  ByVal CodePage As Long, _
  ByVal dwFlags As Long, _
  ByVal lpMultiByteStr As LongPtr, _
  ByVal cchMultiByte As Long, _
  ByVal lpWideCharStr As LongPtr, _
  ByVal cchWideChar As Long _
  ) As Long

Apparently you had that before, so just put it back.

So, for the ```lpWideCharStr``` param, declaring ```As LongPtr``` _and_ passing ```StrPtr(str)``` as the argument works. Would it also have worked if the param was declared ```As String``` and the arg passed was simply ```str``` (without the ```StrPtr()``` function)? — Peter Constable, Jun 01 '20 at 15:33
@PeterConstable No, VB automatically converts strings like I mentioned above. You would end up with a double unicode string if the buffer was big enough, or with a buffer overflow otherwise. — GSerg, Jun 01 '20 at 15:37
So, ```As String``` (and passing a String arg) could only be used in A APIs that expect an multibyte (not Unicode) string? — Peter Constable, Jun 01 '20 at 15:42
@PeterConstable If the api expects a type which ultimately resolves to `char*`. Often it is `LPSTR` or `LPCSTR`. — GSerg, Jun 01 '20 at 15:45

Call to MultiByteToWideChar() in 64-bit office gives wrong result

1 Answers1

Linked