Alphabetical order in Chinese - java.text.Collator

Question

I've been testing alphabetical sorting in Chinese (if I may call it so). This is how Excel sorts some example words:

啊<波<词<的<俄<佛<歌<和<及<课<了<馍<呢<票<气<日<四<特<瓦<喜<以<只

0<2<85<!<@<版本<标记<成员<错误<导出<导航<Excel 文件<访问<分类<更改<规则<HTML<基本<记录<可选<快捷方式<类别<历史记录<密码<目录<内联<内容<讨论<文件<页面<只读

and this is what came out of Collections.sort(list, simplified_chinese_collator_comparator) (the first offending character in bold):

啊<波<词<的<俄<佛<歌<和<及<课<了<呢<票<气<日<四<特<瓦<喜<以<只<馍

!<@<0<2<85<Excel 文件<HTML<版本<标记<成员<错误<导出<导航<访问<分类<更改<规则<基本<记录 <可选<快捷方式<类别<历史记录<密码<目录<内联<内容<讨论<文件<页面<只读

I don't know anything about Chinese. Does anyone know why Collator output it's different, or what is it based on?

Are there any other libraries for language-based sorting?

score 3 · Answer 1 · answered Nov 08 '12 at 09:30

There isn't a Collator in Java 6 or 7 which will sort the Chinese in the same order as the first sample.

public static void main(String... args) {
    String text1 = "啊<波<词<的<俄<佛<歌<和<及<课<了<馍<呢<票<气<日<四<特<瓦<喜<以<只";
    findLocaleForSortedOrder(text1);
    String text2 = "啊<波<词<的<俄<佛<歌<和<及<课<了<呢<票<气<日<四<特<瓦<喜<以<只<馍";
    findLocaleForSortedOrder(text2);
}

private static void findLocaleForSortedOrder(String text) {
    System.out.println("For " + text + " found...");
    String[] preSorted = text.split("<");
    for (Locale locale : Collator.getAvailableLocales()) {
        String[] sorted = preSorted.clone();
        Arrays.sort(sorted, Collator.getInstance(locale));
        if (Arrays.equals(preSorted, sorted))
            System.out.println("Locale " + locale + " has the same sorted order");
    }
    System.out.println();
}

prints

For 啊<波<词<的<俄<佛<歌<和<及<课<了<馍<呢<票<气<日<四<特<瓦<喜<以<只 found...

For 啊<波<词<的<俄<佛<歌<和<及<课<了<呢<票<气<日<四<特<瓦<喜<以<只<馍 found...
Locale zh_CN has the same sorted order
Locale zh has the same sorted order
Locale zh_SG has the same sorted order

Appreciate your effort, but this does not answer the question, which was "why is the output different". You only proved that it is different, which was stated in the question itself. — MaDa, Nov 08 '12 at 11:06
Its appears to me that the Excel sorter is converting the Chinese characters to Latin characters and them comparing. I don't know how Java is comparing the letters, but I would expect it is comparing the character based on how the Chinese would do it ?? — Peter Lawrey, Nov 08 '12 at 11:16

score 3 · Accepted Answer · answered Nov 08 '12 at 20:15

Why it is different? Because there are several different methods of sorting ideographic characters or even entire words. The ones that stuck in my mind are:

by number of strokes
by using Latin transliteration and then ordering it "naturally" (according to rules specific for Chinese language of course)

There are other methods as well, for example Unicode Technical Report #35 mentions some of them (more by coincidence, not necessary on purpose), but you'd have to have plenty of time to go through it.

To answer your question, on why these sorting orders are different, it just because Java contains its own collation rules and it does not rely on Operating System's ones (as Excel does). These rules might be different. You might also want to try out ICU, which is the source of classes and rules in Java (and is usually a step ahead than JDK).

I've tested with ICU4J as well - it gives a proper order for the first string, but it doesn't seem better in sorting Latin characters mixed in - it continues to put them first, before all Chinese symbols. Still, ICU4J is the best I can do without learning Chinese. — MaDa, Nov 09 '12 at 07:58

Alphabetical order in Chinese - java.text.Collator

2 Answers2