Are there any dangers to working internally in UTF-8 and then converting to UTF-16 only when needed in Windows?

Question

Visual studio tries to insist on using tchars, which when compiled with the UNICODE option then basically ends up using the wide versions of the Windows and other API.

Is there then any danger to using UTF-8 internally in the application (which makes use of the C++ STL easier and also enables more readable cross platform code) and then only converting to UTF-16 when you need to use any of the OS APIs?

I'm specifically asking about developing for more than one OS - Windows that doesn't use UTF-8 and others like Mac, that do.

Answer 1

As others have said, there is no danger to using UTF-8 internally, and then converting when you need to call Windows functions.

However, be aware that the cost of converting every time so might become prohibitively expensive if you're displaying a lot of text. (Remember, you don't just have the conversion, but you may also have the cost of allocating and freeing buffers to hold the temporary, converted strings.)

I should also point out there is wide-character support built in to STL, so there's really no reason for doing this. (std::wstring, et al.)

Additionally, working exclusively with UTF-8 is fine for English, but if you plan on supporting Eastern European, Arabic, or Asian character sets your storage requirements for text might turn out to be larger than those for UTF-16 (due to more characters requiring three or four code points to be stored). Again this will probably only be an issue if you're dealing with large volumes of text, but it's something to consider - doubly so if you're going to be transferring this text over a network connection at any time.

Answer 2

No, there are no dangers if you follow the guidelines.^[1] In fact it's the sanest and simplest way to go,^[2] even if you write for Windows only.

And note that UTF-8 is never any longer than UTF-16 for European languages, nor for non-BMP characters. It takes more space only for codepoints encoded with 3 bytes in UTF-8 and 2 in UTF-16, which is precisely the U+0800 to U+FFFF range,^[3] which is mostly CJK characters.

Answer 3

Since UTF-8 and UTF-16 are merely two ways of encoding numbers (which are then interpreted as so called code-points or glyphs) there is nothing wrong with converting back and forth: no information is lost. So no, there's no danger in converting (as long as the conversion is correct, of course).

Answer 4

I'm assuming your project is not about text processing, manipulation or transformation: For text processing, it is far easier to chose one and only one encoding, the same on all platforms, and then do the conversion if needed when using the native API.

But if your project is not centered around text processing/manipulation/transformation, then the restriction to UTF-8 on all platforms is not the simpliest solution.

Avoid using char on Windows

If you work with the char type on Windows development, then all the WinAPI will use char.

The problem is that the char type on Windows is used for "historical" applications, meaning pre-unicode application.

Every char text is interpreted as a non-Unicode text whose encoding/charset is chosen by the Windows user, not you the developper.

Meaning: If you believe you're working with UTF-8, send that UTF-8 char text to the WinAPI to output on GUI (and TextBox, etc.), and then execute your code on a Windows set up on Arabic (for example), then you'll see your pretty UTF-8 char text won't be handled correctly by the WinAPI because the WinAPI on that Windows believes all the char are to be interpreted as Windows-1256 encoding.

If you're working with char on Windows, you're forsaking Unicode unless every call to the WinAPI goes through a translation (usually through a Framework like GTK+, QT, etc., but it could be your own wrapper functions).

Optimization is the Root of all Evil, but then, converting all your UTF-8 texts from and to UTF-16 each time you discuss with Windows does seems to me to be quite an useless pessimization.

Alternative: Why not using TCHAR on all platforms?

What you should do is work with TCHAR, provide a header similar to tchar.h for Linux/MacOS/Whatever (redeclaring the macros, etc. in the original tchar.h header), augmenting it with a tchar.h-like header for the Standard Library objects you want to use. For example, my own tstring.hpp goes like:

// tstring.hpp
#include <string>
#include <sstream>
#include <fstream>
#include <iostream>

#ifdef _MSC_VER
#include <tchar.h>
#include <windows.h>
#else
#ifdef __GNUC__
#include <MyProject/tchar_linux.h>
#endif // __GNUC__
#endif


namespace std
{

#ifdef _MSC_VER

   // On Windows, the exact type of TCHAR depends on the UNICODE and
   // _UNICODE macros. So the following is useful to complete the
   // tchar.h headers with the C++ Standard Library's symbols.

   #ifdef UNICODE

      typedef              wstring        tstring ;
      // etc.
      static wostream &    tcout          = wcout ;

   #else // #ifdef UNICODE

      typedef              string         tstring ;
      // etc.
      static ostream &     tcout          = cout ;

   #endif // #ifdef UNICODE

#else // #ifdef _MSC_VER

    #ifdef __GNUC__

    // On Linux, char is expected to be UTF-8 encoded, so the
    // following simply maps the txxxxx type into the xxxxx
    // type, forwaking the wxxxxx altogether.
    // Of course, your mileage will vary, but the basic idea is
    // there.

    typedef                string         tstring ;
    // etc.
    static ostream &       tcout          = cout ;

    #endif // __GNUC__

#endif // #ifdef _MSC_VER

} // namespace std

Discplaimer: I know, it's evil to declare things in std, but I had other things to do than be pedantic on that particular subject.

Using those headers, you can use the C++ Standard Library combined with the TCHAR facility, that is, use std::tstring, which will be compiled as std::wstring on Windows (provided you compile defining theUNICODE and _UNICODE defines) and as std::string on the other char-based OSes you want to support.

Thus, you'll be able to use the platform's native character type at no cost whatsoever.

As long as you are agnostic with your TCHAR character type, there won't be any problem.

And for the cases you really want to deal with the dirty side of UTF-8 vs. UTF-16, then you need to provide the code for conversion (if needed), etc..

This is usually done by providing overloads of the same function for different types, and for each OS. This way, the right function is selected at compile time.

Answer 5

If you have an OS that takes wid(er) characters in its API, and you're writing an application that requires internationalization support, it is completely silly to be using char and UTF-8 as an internal representation in your program. You're using UTF-8 backwards. UTF-8 is for smuggling Unicode through operating systems interfaces, and storage and data interchange formats which cannot handle wide characters directly.

Answer 6

The "danger" is that UTF-8 character count is not the same as ASCII character count. E.g., U+24B62 is a single Unicode character but expands to 4 UTF-8 bytes. (See here for other examples.)

If you don't use the two interchangeably, you will be fine.

Answer 7

UTF-8 is a wild and wacky way of representing characters. You should avoid using it wherever possible. The windows API avoids UTF-8. ( If you insist on a 'multibyte' build, rather than a 'unicode' build it will do all the conversions for you, under the covers, so it can continue to use UTF16 - and if you are not careful the inefficiency of all those hidden conversions will eat you up. ) The wxWidgets library avoids UTF-8 in the same way, and that is cross-platform with MACs.

You should take a hint from this, and avoid UTF-8 yourself.

When do you need to use UTF-8? The snag with UTF16 is that it depends on the byte order in the words implemented in the hardware. So when you transfer data between different computers, which might use a different byte order in their hardware, you have to use UTF8 which has the same byte order on any hardware. This is why browsers and WWW pages use UTF8.