restore runtime unicode strings

Question

I'm building an application that receives runtime strings with encoded unicode via tcp, an example string would be "\u7cfb\u8eca\u4e21\uff1a\u6771\u5317 ...". I have the following but unfortunately I can only benefit from it at compile time due to: incomplete universal character name \u since its expecting 4 hexadecimal characters at compile time.

QString restoreUnicode(QString strText)
   {
      QRegExp rx("\\\\u([0-9a-z]){4}");
      return strText.replace(rx, QString::fromUtf8("\u\\1"));
   }

I'm seeking a solution at runtime, I could I foreseen break up these strings and do some manipulation to convert those hexadecimals after the "\u" delimiters into base 10 and then pass them into the constructor of a QChar but I'm looking for a better way if one exists as I am very concerned about the time complexity incurred by such a method and am not an expert.

Does anyone have any solutions or tips.

Does `fromUtf8("\\u\\1")` work? Your thinking has the same problem as this attempt: `const char razy[] = "lass"; crazy Foo { int a; bool b; };` — Kerrek SB, Nov 18 '11 at 14:23
Why not use `QDataStream` to encode/decode data going over the sockets? — Silas Parker, Nov 18 '11 at 15:34
I do not have control over the server, just working with a third party data stream as it is a mixture of ascii and in small occasions embedded unicode. I have made a solution which works well, will post it in 6 hours when the answer your own question timer expires on this site. — Will, Nov 18 '11 at 15:57

score 1 · Answer 1 · answered Nov 18 '11 at 14:45

1

You should decode the string by yourself. Just take the Unicode entry (rx.indexIn(strText)), parse it (int result; std::istringstream iss(s); if (!(iss>>std::hex>>result).fail()) ... and replace the original string \\uXXXX with (wchar_t)result.

answered Nov 18 '11 at 14:45

Vlad

35,022
6
77
199

I have done something similar to which it works well, presence of Unicode is low so I'm hoping my solution will not bring in troubling CPU usage. I will post the solution in 6 hours when the site lets me. – Will Nov 18 '11 at 16:00

score 1 · Answer 2 · edited Mar 24 '15 at 11:40

#include <assert.h>
#include <iostream>
#include <string>
#include <sstream>
#include <locale>
#include <codecvt>          // C++11
using namespace std;

int main()
{
    char const  data[]  = "\\u7cfb\\u8eca\\u4e21\\uff1a\\u6771\\u5317";

    istringstream   stream( data );

    wstring     ws;
    int         code;
    char        slashCh, uCh;
    while( stream >> slashCh >> uCh >> hex >> code )
    {
        assert( slashCh == '\\' && uCh == 'u' );
        ws += wchar_t( code );
    }

    cout << "Unicode code points:" << endl;
    for( auto it = ws.begin();  it != ws.end();  ++it )
    {
        cout << hex << 0 + *it << endl;
    }
    cout << endl;

    // The following is C++11 specific.
    cout << "UTF-8 encoding:" << endl;
    wstring_convert< codecvt_utf8< wchar_t > >  converter;
    string const bytes = converter.to_bytes( ws );
    for( auto it = bytes.begin();  it != bytes.end();  ++it )
    {
        cout << hex << 0 + (unsigned char)*it << ' ';
    }
    cout << endl;
}

the stream is not exclusively unicode as there may be non unicode entries within the strings within the ongoing stream but thanks. — Will, Nov 18 '11 at 16:02

score 1 · Accepted Answer · answered Nov 18 '11 at 23:08

For closure and anyone who comes across this thread in future, here is my initial solution before optimising the scope of these variables. Not a fan of it but it works given the unpredictable nature of unicode and/or ascii in the stream of which I have no control over (client only), whilst Unicode presence is low, it is good to handle it instead of ugly \u1234 etc.

QString restoreUnicode(QString strText)
{
    QRegExp rxUnicode("\\\\u([0-9a-z]){4}");

    bool bSuccessFlag;
    int iSafetyOffset = 0;
    int iNeedle = strText.indexOf(rxUnicode, iSafetyOffset);

    while (iNeedle != -1)
    {
        QChar cCodePoint(strText.mid(iNeedle + 2, 4).toInt(&bSuccessFlag, 16));

        if ( bSuccessFlag )
            strText = strText.replace(strText.mid(iNeedle, 6), QString(cCodePoint));
        else
            iSafetyOffset = iNeedle + 1; // hop over non code point to avoid lock

        iNeedle = strText.indexOf(rxUnicode, iSafetyOffset);
    }

    return strText;
}

restore runtime unicode strings

3 Answers3