How to import unicode data from WinHttpReq

Question

WinHttpRequest is used to import bank statement from Nordiget.com containing characters from Windows-1257 code page. For an unknown reason, FoxPro does replace accented characters with? signs. To fix this, COMPROP(WinHttpReq ,"UTF8",1) is added:

  WinHttpReq = CREATEOBJECT("WinHttp.WinHttpRequest.5.1")
  WinHttpReq.Open("GET", "https://ob.nordigen.com/api/v2/accounts/11111-22222-333333-4444-/transactions/", .f.)
  WinHttpReq.setrequestheader('accept', 'application/json')
  WinHttpReq.setrequestheader('Authorization', "Bearer afde554a5sd45a45as4")
  WinHttpReq.Send()
  * If this is removed, accented characters appear as single question characters:
  COMPROP(WinHttpReq ,"UTF8",1)

Response WinHttpReq.ResponseText contains accented characters represented by 4 bytes. The first 3 are the same but the fourth is different. Those characters are converted by code using

Function NordigetConvert(cStr)
cstr = strt( cStr, CHR(0xc3)+CHR(0x83)+ CHR(0xc2)+CHR(0x9c) , 'Ü' ) 
cstr = strt( cStr, CHR(0xc3)+CHR(0x83)+ CHR(0xc2)+CHR(0xBC) , 'ü' ) 
cstr = strt( cStr, CHR(0xc3)+CHR(0x83)+ CHR(0xc2)+CHR(0x84) , 'Ä' ) 
cstr = strt( cStr, CHR(0xc3)+CHR(0x83)+ CHR(0xc2)+CHR(0xB6) , 'ö' ) 
return cStr

This converts only ÜüÄö. There are much more characters to convert.

Tried

STRCONV(strconv(CHR(0xc3)+CHR(0x83)+ CHR(0xc2)+CHR(0x9c) ,10) ,11, 1257, 1)

this should return letter Ü but returns two question marks. Tried also code below from https://github.com/VFPX/Win32API/blob/master/libraries/kernel32/WideCharToMultiByte.md but it still returns two question marks. How to convert it?

SET ECHO OFF
SET TALK OFF
CLEAR


#DEFINE CF_UNICODETEXT  13
DO decl
 
LOCAL hData, lcUnicode

MESSAGEBOX( uconv (CHR(0xc3)+CHR(0x83)+ CHR(0xc2)+CHR(0x9c), 1257)  )

* end of main
 
FUNCTION uconv (lcSrc, lnCodePage)
    LOCAL lcDst, lnUsedDefault, lnResult
    lcDst = Repli(Chr(0), Len(lcSrc) * 2)
    lnUsedDefault = 0
 
    lnResult = WideCharToMultiByte (lnCodePage, 0,;
        lcSrc, Len(lcSrc),;
        @lcDst, Len(lcDst), "?", 0)
 
    IF lnResult = 0
    *   87 - ERROR_INVALID_PARAMETER
    *  122 - ERROR_INSUFFICIENT_BUFFER
    * 1004 - ERROR_INVALID_FLAGS
        ? "Error code:", GetLastError()
        lcDst = ""
    ELSE
        lcDst = SUBSTR(lcDst, 1, lnResult)
    ENDIF
RETURN SUBSTR(lcDst, 1, AT(Chr(0),lcDst)-1)
 
FUNCTION  memwchar2str (lnMemBlock)
* copies Unicode characters (two-byte) from a memory address to a VFP string
RETURN  mem2str(lnMemBlock, Chr(0)+Chr(0))
 
FUNCTION  mem2str(lnMemBlock, end_sequence)
#DEFINE BUFFER_SIZE   16
#DEFINE EMPTY_BUFFER  Repli(Chr(0), BUFFER_SIZE)
 
    DECLARE RtlMoveMemory IN kernel32 As Heap2Str;
        STRING @, INTEGER, INTEGER
 
    LOCAL lnPtr, lcResult, lcBuffer, lnPos
    lnPtr = lnMemBlock
    lcResult = ""
 
    DO WHILE .T.
        lcBuffer = EMPTY_BUFFER
        = Heap2Str (@lcBuffer, lnPtr, BUFFER_SIZE)
        lnPos = AT(end_sequence, lcBuffer)
        IF lnPos > 0
            lcResult = lcResult + SUBSTR(lcBuffer, 1, lnPos-1)
            RETURN  lcResult
        ELSE
            lcResult = lcResult + lcBuffer
            lnPtr = lnPtr + BUFFER_SIZE
        ENDIF
    ENDDO
 
PROCEDURE decl
    DECLARE INTEGER GetLastError IN kernel32
    DECLARE INTEGER OpenClipboard IN user32 INTEGER hwnd
    DECLARE INTEGER CloseClipboard IN user32
    DECLARE INTEGER GetClipboardData IN user32 INTEGER uFormat
    DECLARE INTEGER IsClipboardFormatAvailable IN user32 INTEGER wFormat
 
    DECLARE INTEGER WideCharToMultiByte IN kernel32;
        INTEGER CodePage, INTEGER dwFlags, STRING lpWideCharStr,;
        INTEGER cchWideChar, STRING @lpMultiByteStr, INTEGER cbMultiByte,;
        STRING lpDefaultChar, INTEGER lpUsedDefaultChar

WinHttpReq.ResponseText is a BSTR, so it stores Unicode data, not UTF8. Doesn't FoxPro knows how to handle it? If you need to convert to Ansi or UTF8 you can use WideCharToMultiByte winapi https://learn.microsoft.com/en-us/windows/win32/api/stringapiset/nf-stringapiset-widechartomultibyte (or a possible equivalent in FoxPro) — Simon Mourier, Feb 09 '23 at 10:40
How to call this API from FoxPro? Or can some strconv() function used? Documentation is at https://hackfox.github.io/section4/s4g663.html — Andrus, Feb 09 '23 at 10:44
That doesn't really look like unicode. If COMPROP() is fixing it then what is the problem. — Cetin Basoz, Feb 09 '23 at 11:52
It returns 4 bytes for every character. To show it in foxpro, it should be converted into single byte character in Windows-1257 or CPCURRENT() encoding used by Foxpro. Some STRCONV() calls or direct WinAPI call should do it. — Andrus, Feb 09 '23 at 14:27
@SimonMourier WideCharToMultiByte returns two question marks. I updated question. How tu use it? — Andrus, Feb 09 '23 at 21:20
Depends on what you want as output and how you can use this output with Foxpro. I guess 1257 (CP_BALTIC) should work as output if FoxPro knows how to handle that. But why to you pass 0xC383 (https://unicodemap.org/details/0xC383/index.html) and 0xC29C characters as input? — Simon Mourier, Feb 10 '23 at 07:49
Result should be single byte, 0xDC. Those 4 bytes per character are returned in WinHttpRequest.ResponseText — Andrus, Feb 10 '23 at 08:40
ResponseText is built from ResponseBody (which is a raw array of bytes) with an althorithm that depends on other context things like response Content-Type headers (charset). You might want to check that, or use ResponseBody directly. — Simon Mourier, Feb 10 '23 at 10:49
For `application/json` WinHttpRequest automatically assumes request/response body is UTF-8 and converts it to Unicode automagically so that on `Send` you use a BSTR which gets converted to UTF-8 and on receive the `ResponseText` is a BSTR which is converted from the UTF-8 body of the response so no manual conversion is needed (for `application/json` Content-Type only) and no byte-array fiddling on Send/ResponseBody is necessary in VBA (not sure for FoxPro). If you host is responding with any encoding but UTF-8 to an `application/json` c.type then something is very wrong with this web API impl. — wqw, Feb 17 '23 at 12:22

How to import unicode data from WinHttpReq

0 Answers0