same strings, different charset, not equals

Question

I have a weird problem.

I have an application that crawl a webpage to get a list o names. Than this list is passed to another application that using those names, ask for information to a site, using its API's.

When I compare some strings in the first webpage to some others grabbed by API's usually I get wrong results.

I tried to get character value letter by letter I got this:

Rocco De Nicola
82 111 99 99 111 160 68 101 32 78 105 99 111 108 97 1st web page
82 111 99 99 111 32 68 101 32 78 105 99 111 108 97 2nd

As you can see, in the first string a space is codified by 160 (non-breaking space) instead of 32.

I can I codify correctly the first set of Strings?

I have also tried to set the Charset to UTF-8 but it didn't worked.

Maybe I just have to replace 160 to 32 ?

score 2 · Answer 1 · answered Sep 19 '12 at 11:23

I would at first trim and replace complicated characters from the strings to compare. After this step follows the equals call. This brings also the advantages in cases you have language specific replacements in your text. It's also a good idea to convert the resulting strings to lower case.

Normally I use something like that ....

private String removeExtraCharsAndToLower(String str) {
    str=str.toLowerCase();
    str=str.replaceAll("ä", "ae");
    str=str.replaceAll("ö", "oe");
    str=str.replaceAll("ü", "ue");
    str=str.replaceAll("ß", "ss");
    return str.toLowerCase().replaceAll("[^a-z]","");
}

score 0 · Answer 2 · answered Sep 19 '12 at 09:20

Using brute force. This lists all the character set which convert 160 to 32 when encoding.

String s = "" + (char) 160;
for (Map.Entry<String, Charset> stringCharsetEntry : Charset.availableCharsets().entrySet()) {
    try {
        ByteBuffer bytes = stringCharsetEntry.getValue().encode(s);
        if (bytes.get(0) == 32)
            System.out.println(stringCharsetEntry.getKey());
    } catch (Exception ignored) {
    }
}

prints nothing.

If I change the condition to

if (bytes.get(0) != (byte) 160)
    System.out.println(stringCharsetEntry.getKey()+" "+new String(bytes.array(), 0));

I get quite a few examples.

same strings, different charset, not equals

2 Answers2