C String encoding UTF8 without libiconv

Question

I know this question has been answered to most of the languages but i have not found a solution for C or i am misunderstanding something.

I have declared a "char *strSSID". I have a function in a web server that process the POST params of the form page. This function return in "srtSSID" a param named SSID which is used to configure the SSID name of a wireless. But the problem is that special characters in the SSID like "ñ" are seen with Linux properly and with my phone or Windows like strange characters.

I know the problem is the encoding. In my plattform (mips and uclibc library) a special character like 'ñ' is stored in two positions (i mean for example "srtSSID[0]" and "strSSID[1]" to store the first 'ñ' character of the SSID) of strSSID.

I need some help.

Thanks

The bytes are the same, it's (probably) just a display issue. For display, you may be able to use `mbstowcs` or a similar conversion function [that might also help if a Windows API function expects a UCS-2 or UTF-16 encoded string]. — Daniel Fischer, May 23 '13 at 13:06
Can you clarify your problem a little? Generally, the SSID is a string of bytes and whereas your Linux distro obviously chooses to interpret that as UTF-8, Android assumes it is plain ASCII, which is why you may be seeing different things on different devices. I don't understand how this translates to a problem with your C code. — Bob Sammers, May 23 '13 at 13:07
@BobSammers My problem is that i want that the SSID can be seen properly in all OS, so i guess that changing the encoding of the string from UTF-8 to ASCII (then the SSID string in the Wifi frame will be in ASCII instead UTF-8) will be well interpreted in all the OS. So the solution (im not sure if this is the solution) how to change the encoding of the string from UTF8 to ASCII? — MABC, May 23 '13 at 14:17

score 1 · Answer 1 · answered May 23 '13 at 16:55

The only way to ensure characters are displayed correctly in all devices is to use characters that appear in ASCII, because (as I think you've identified) you won't be able to force Windows etc. to interpret them as UTF-8. Your problem is that characters such as 'ñ' don't appear in ASCII, so you can't translate unicode strings with such characters into standard ASCII.

The first 128 UTF-8 chars are the same as ASCII, so no translating of the string needs to occur... but you will need to remove or replace any chars that have values above 127.

If all the devices you are likely to use have the same non-English default language, it may be the case that they all use the same extended ASCII version (strictly, ASCII is a 7-bit code, but most implementations use the top bit to add another 128 characters which usually include accented characters and are often dependent on the device's default language). I'm straying into speculation here though! And your extended ASCII character set is unlikely to be compatible with devices which assume the SSID is UTF-8, of course!

score 1 · Accepted Answer · answered May 23 '13 at 18:27

In order to use UTF-8 characters in an SSID, the 802.11 spec requires that the UTF-8 SSID subfield of the Extended Capabilities element needs to be set. If you're finding that UTF-8 SSIDs don't work on certain devices, it's possible that they haven't set the Extended Capabilities element correctly, or perhaps they just don't support that capability.

The 802.11 spec doesn't say how the characters should be interpreted if the UTF-8 SSID capability isn't enabled, but I suspect that typically means only ASCII characters are valid, in which case your ñ is never going to work.

That said, you might have some success just converting to Latin1. If you don't have access to a library that will perform character set conversions, here's a basic conversion function that should be good enough for your needs.

void utf8tolatin1(char *s) {
  size_t i = 0, j = 0;
  char c;
  do {
    c = s[i++];
    if ((c&0xFC) == 0xC0 && s[i])
      c = (c<<6) + (s[i++]&0x3F);
    s[j++] = c;
  } while(c != 0);
}

Note that this only converts UTF-8 characters that are in the Latin1 range - everything else is left as is. If your SSID contains characters outside that range, there's probably nothing you can do to get it to work.

score 0 · Answer 3 · answered May 23 '13 at 13:43

0

If you're posting this through a webpage, you have to make sure that the encoding declared by the page is indeed utf-8. There are several ways to achieve that:

Use the Content-Type header in the response: Content-Type: text/html; charset=UTF-8
Use a meta tag in the HTML page. This is dependent on the HTML version you use. in HTML 4 it's <meta http-equiv="Content-type" content="text/html;charset=UTF-8">

The browsers are configured to assume different defaults for encoding, so a likely explanation is that your windows browser is expecting ISO-8859-1 encoding.

See this w3c page for more explanation on how to declare the encoding: http://www.w3.org/International/questions/qa-html-encoding-declarations

answered May 23 '13 at 13:43

Frederik Deweerdt

4,943
2
29
31

My problem is not SSID displaying in the web browser. My problem is with the SSID string travelling in the Wifi frame. I think this string is encoded in UTF-8 and then some OS are not supporting this encoding, so I want change char * srtSSID from UTF-8 to ASCII. – MABC May 23 '13 at 14:22
@user2119381 i think you're mistaken: the decoding is done by the client application, not the OS. You can't display 'ñ' in ASCII because it's not an ASCII character: http://en.wikipedia.org/wiki/ASCII – Frederik Deweerdt May 23 '13 at 16:10

C String encoding UTF8 without libiconv

3 Answers3

Linked