char vs wchar_t

Question

I'm trying to print out a wchar_t* string. Code goes below:

#include <stdio.h>
#include <string.h>
#include <wchar.h>

char *ascii_ = "中日友好";  //line-1
wchar_t *wchar_ = L"中日友好";  //line-2

int main()
{
    printf("ascii_: %s\n", ascii_);  //line-3
    wprintf(L"wchar_: %s\n", wchar_);  //line-4
    return 0;
}

//Output
ascii_: 中日友好

Question:

Apparently I should not assign CJK characters to char* pointer in line-1, but I just did it, and the output of line-3 is correct, So why? How could printf() in line-3 give me the non-ascii characters? Does it know the encoding somehow?
I assume the code in line-2 and line-4 are correct, but why I didn't get any output of line-4?

What compiler are you using? GCC defaults to Utf-8 o most platforms. Seems like an encoding issue to me. — cyco130, Sep 21 '11 at 07:41

Mysticial · Accepted Answer · 2011-09-21T08:24:50.520

First of all, it's usually not a good idea to use non-ascii characters in source code. What's probably happening is that the chinese characters are being encoded as UTF-8 which works with ascii.

Now, as for why the wprintf() isn't working. This has to do with stream orientation. Each stream can only be set to either normal or wide. Once set, it cannot be changed. It is set the first time it is used. (which is ascii due to the printf). After that the wprintf will not work due the incorrect orientation.

In other words, once you use printf() you need to keep on using printf(). Similarly, if you start with wprintf(), you need to keep using wprintf().

You cannot intermix printf() and wprintf(). (except on Windows)

EDIT:

To answer the question about why the wprintf line doesn't work even by itself. It's probably because the code is being compiled so that the UTF-8 format of 中日友好 is stored into wchar_. However, wchar_t needs 4-byte unicode encoding. (2-bytes in Windows)

So there's two options that I can think of:

Don't bother with wchar_t, and just stick with multi-byte chars. This is the easy way, but may break if the user's system is not set to the Chinese locale.
Use wchar_t, but you will need to encode the Chinese characters using unicode escape sequences. This will obviously make it unreadable in the source code, but it will work on any machine that can print Chinese character fonts regardless of the locale.

if use unicode escape seq, I gotta find out each Chinese word's seq, right? That'll be quite a lot of work to do, :P — Alcott, Sep 21 '11 at 08:32
Correct. However, I'm sure this is common enough where you can find a tool online that you can just copy and paste the Chinese text and it will give you the unicode escape sequence. And to keep the code readable, you can keep the actual Chinese text next to the escape sequence as a comment. — Mysticial, Sep 21 '11 at 08:34
Here we go, I already found one: http://www.rishida.net/tools/conversion/: `中日友好 -> "\u4E2D\u65E5\u53CB\u597D"` — Mysticial, Sep 21 '11 at 08:39

score 6 · Answer 2 · answered Sep 21 '11 at 07:44

6

Line 1 is not ascii, it's whatever multibyte encoding is used by your compiler at compile-time. On modern systems that's probably UTF-8. printf does not know the encoding. It's just sending bytes to stdout, and as long as the encodings match, everything is fine.

One problem you should be aware of is that lines 3 and 4 together invoke undefined behavior. You cannot mix character-based and wide-character io on the same FILE (stdout). After the first operation, the FILE has an "orientation" (either byte or wide), and after that any attempt to perform operations of the opposite orientation results in UB.

answered Sep 21 '11 at 07:44

R.. GitHub STOP HELPING ICE

208,859
35
376
711

I comment out the printf() line, I got some output but not the Chinese characters. Why? – Alcott Sep 21 '11 at 08:04
locale's encoding? How to fix it? – Alcott Sep 21 '11 at 08:12
@Alcott: I just updated my answer to this. I meant to add it here as a comment, but it's too large. – Mysticial Sep 21 '11 at 08:25

score 1 · Answer 3 · answered Sep 21 '11 at 07:48

You are omitting one step and therefore think the wrong way.

You have a C file on disk, containing bytes. You have a "ASCII" string and a wide string.

The ASCII string takes the bytes exactly like they are in line 1 and outputs them. This works as long as the encoding of the user's side is the same as the one on the programmer's side.

The wide string first decodes the given bytes into unicode codepoints and stored in the program- maybe this goes wrong on your side. On output they are encoded again according to the encoding on the user's side. This ensures that these characters are emitted as they are intended to, not as they are entered.

Either your compiler assumes the wrong encoding, or your output terminal is set up the wrong way.

char vs wchar_t

3 Answers3

Linked