How can I print a Unicode CString (in Japanese) to a text file?

Question

This seems like a tough one for me. I have this code that prints out a CString to a text file, but the value happens to be in Unicode (Japanese). As soon as this line is hit, nothing else below it gets printed.

Any idea how can I print the Japanese text in the text file?

#define OFSTREAM std::wofstream

OFSTREAM *outfile;
outfile = new OFSTREAM;
outfile->open (filename, ios::out);

CString varName = _T(" ");
/*stuff*/
*outfile << _T("  Name: ") << (LPCTSTR)varName << _T("\n");

Are you building with the macro `UNICODE` defined? I.e., is `LPCTSTR` the same as `LPCWSTR`? — ildjarn, Aug 31 '12 at 00:38
Is your CString unicode aware? `CString varName(_T("Unicode aware"));` — Jesse Good, Aug 31 '12 at 00:42
@ildjarn: Thanks for the suggestion. I was not, but I added a '#define UNICODE' in the top of the file and replaced the 'LPCTSTR' with an 'LPCWSTR,' and it made no difference so far. — SoftwareGuy, Aug 31 '12 at 00:46
UNICODE is not an encoding. Is it in UTF8 or UTF16 or a codepage or what? — Mooing Duck, Aug 31 '12 at 00:48
@JesseGood: Sorry, I'm not sure what that means. I'm having it declared as this: `CString varName = _T(" ");` — SoftwareGuy, Aug 31 '12 at 00:48
And `outfile` is a pointer to a `std::wostream`? Your code is 100% fine as far as we can see. We need more code to find the issue. — Mooing Duck, Aug 31 '12 at 00:49
@Mooing Duck: I'm not sure, but I'm seeing this in some resx files: `` — SoftwareGuy, Aug 31 '12 at 00:50
And here's the declaration for outfile: `OFSTREAM *outBasfile;` `#define OFSTREAM std::wofstream` — SoftwareGuy, Aug 31 '12 at 00:51
@SoftwareGuy: It still looks 100% fine. The problem is elsewhere. — Mooing Duck, Aug 31 '12 at 00:53
I see. I have no idea where to look next. I'm searching the whole solution for outFile, and am finding these: `outfile = new OFSTREAM;` and `outfile->open (filename,ios::out);` -- I'll add them up to the original message like you did mine. — SoftwareGuy, Aug 31 '12 at 01:20

score 2 · Accepted Answer · answered Aug 31 '12 at 14:16

The reason the stream stops working is because the fail bit gets set. You're not handling the error so the stream stops working. The fail bit needs to be cleared when errors happen.

The locale on the wostream object must be set such that the codecvt facet handles converting the Japanese wide characters into bytes. By default the "C" locale is used which in VS will only support ASCII characters. If you only need writing the file to work on Japanese versions of Windows you can do:

std::wofstream outfile(filename, ios::out);
outfile.imbue(std::locale("")); // use the system locale

CString varName = _T(" ");
/*stuff*/
outfile << L"  Name: " << (LPCTSTR)varName << L"\n";

Or you can specify the Japanese locale on Windows:

outfile.imbue(std::locale("Japanese")); // use the japanese locale on any Windows system

Both of these methods use the legacy Japanese locale encoding, which should probably be avoided. You can instead use UTF-8:

// replace just the codecvt facet of the stream's current locale
outfile.imbue(std::locale(outfile.getloc(), new std::codecvt_utf8_utf16<wchar_t>()));

Thanks a lot. That's the one I ended up using. – SoftwareGuy Sep 25 '12 at 22:00 — SoftwareGuy, Sep 25 '12 at 22:00
Won't `new std::codecvt_utf8_utf16()` cause memory leak? – Zoli Feb 19 '18 at 10:46 — Zoli, Feb 19 '18 at 10:46

Daniel S · Answer 2 · 2012-08-31T02:04:36.290

by Unicode you probably mean UTF16 in this case. Depending on the editor you use to view the file, you might need to put a byte order mark (BOM) at the beginning of the file, so the encoding and byte order is properly guessed. Else you need to explicitly set the encoding when viewing the file.

However, your question suggests that you have text before the Japanese text in the file. if you put that out as non-wide-string, that is the problem. Don't mix. If all text is wide, you may just lack the possibility to show Japanese characters in your editor of choice, or did not install support for asiatic languages in Windows.

Firefox usually supports exotic characters pretty well. drag the file there, check the encoding and see what you get.

score 0 · Answer 3 · answered Aug 31 '12 at 02:44

If you're using Visual Studio, go to your C++ project settings and set the language option to use Unicode instead of ANSI. Then Visual Studio will automatically define macros UNICODE and _UNICODE, and you should not define them yourself. You'll likely want to call setlocale so the VC++ library will convert Unicode strings in memory to multibyte ANSI strings in text files, instead of losing everything after the first non-ASCII character.

If you're using some other compiler with MFC, then I guess you have to define both macros UNICODE and _UNICODE yourself, and I'm not sure what else you have to do.

If you want your text files to contain Unicode instead of multibyte ANSI, then you don't want the VC++ library to convert them, but you'll have to do something else instead. Take a look at the Community Content section of MSDN page wofstream.

How can I print a Unicode CString (in Japanese) to a text file?

3 Answers3