Looking for UTF8-aware formatting functions like printf(), etc

Question

I discovered an interesting problem when processing UTF-8 strings containing non-ASCII chars with C standard library formatting functions like sprintf():

The functions of the printf() family are not aware of utf-8 and process everything based on the number of bytes, not chars. Therefore the formatting is incorrect.

Simple example:

#include <stdio.h>

int main(int argc, char *argv[])
{
    const char* testMsg = "Tääääßt";
    char buf[1024];
    int len;

    sprintf(buf, "|%7.7s|", testMsg);
    len = strlen(buf);
    printf("Result=\"%s\", len=%d", buf, len);

    return 0;
}

The result is:

 Result="|Täää|", len=7

Most probably some of you will recommand to convert the application from char to wchar_t and use fwprintf(), etc., but that's absolutely impossible because of huge existing applications. I could imagine writing a wrapper that uses these functions internally, but this would be tricky and very inefficient.

So the best solution would be a UTF-8-aware replacement for the formatting functions of the Standard C Library.

Currently I'm working on QNX 6.4, but replies for other operating systems. e.g. Linux, are also very welcome.

Answer 1

Well, once you ask printf to do intelligent padding of Unicode characters, you run into major problems. As they say,

w͢͢͝h͡o͢͡ ̸͢k̵͟n̴͘ǫw̸̛s͘ ̀́w͘͢ḩ̵a҉̡͢t ̧̕h́o̵r͏̵rors̡ ̶͡͠lį̶e͟͟ ̶͝in͢ ͏t̕h̷̡͟e ͟͟d̛a͜r̕͡k̢̨ ͡h̴e͏a̷̢̡rt́͏ ̴̷͠ò̵̶f̸ u̧͘ní̛͜c͢͏o̷͏d̸͢e̡͝?͞

• How many Unicode characters are in Tääääßt? Well, it could be anywhere from 7 to 11, depending on how it's encoded. Each ä can be written as U+00E4, which is one character, or it could be written as U+0061 U+0308, which is two characters. So your next hope is to count grapheme clusters. (No, normalization won't make the problem go away.)

• But, how wide is a grapheme cluster? Obviously, a is one column wide. U+200B should be zero columns wide, it's a "zero-width" space. Should each ひらがな be two columns wide? They usually are in terminal emulators. What happens when you format ひらがな as 7 columns, do you get "ひらが ", which adds a space, or do you get "ひらが", which is only 6 columns?

• If you cut something up which mixes RTL and LTR text, should you reset the text direction afterwards? What are you going to do? (Some terminal emulators, such as Apple's, support a mixture of left-to-right and right-to-left text.)

• What is your goal by truncating text? Are you trying to show the user a string in limited space, or are you trying to write a format that uses fixed-width fields?

Basically, if you want to cut Unicode text into chunks, you shouldn't be doing it with something as simple as printf (or wprintf, which is quite possibly worse). Use LibICU (website) to iterate over the breaks you want. Writing a UTF-8 aware version of printf is asking for all sorts of trouble that you don't want.

Answer 2

The following C99 code snippet defines the function u8printf where format specifiers such as %10s yield 10 utf-8 code points, that is characters rather than bytes. Don't forget to set the locale with setlocale(LC_ALL,"") somewhere before this routine is called. This works because the wprintf uses wchar_t internally. You can define u8fprintf and u8sprintf in a similar way. If you want to write this without C99 variable length arrays than a suitable combination of malloc/free is also possible.

int u8printf(char *fmt,...){
    va_list ap;
    va_start(ap,fmt);
        int n=mbstowcs(0,fmt,0);
        if(n==-1) return -1;
        wchar_t wfmt[n+1];
        mbstowcs(wfmt,fmt,n+1);
        for(int m=128;m<=32768;m*=2){
            wchar_t wbuf[m];
            int r=vswprintf(wbuf,m,wfmt,ap);
            if(r!=-1) {
                char buf[m*4];
                wcstombs(buf,wbuf,m*4);
                fputs(buf,stdout);
                return r;
            }
        }
        return -1;
    va_end(ap);
}