How to convert an UTF-8 string?

Question

I have to use a method that sends a zero terminated string:

ssize_t send_utf8 (char *buffer)

Where buffer is a string coded as "utf-8" or "ISO-8859-1". This function sends a zero terminated string using another function that sends a buffer (this function is transparent and the buffer may be binary data):

ssize_t send (void *buffer, size_t  num)

Where buffer is the buffer to be send and num is the number of bytes.

I've implemented this way, but before calling the send() method I have to convert the buffer coded as UTF-8.

ssize_t send_utf8 (char *buffer) {

    // UTF-8 conversion?

    net_http_send(buffer, strlen(buffer));

    return connection->content_len;   // number of bytes written
}

Should I use any library or is there a quick way to convert this buffer? Thanks in advance.

*All* data in a computer is ultimately "binary". Can you please elaborate? Perhaps show an example string with corresponding "binary" data? — Some programmer dude, May 22 '14 at 18:44
UTF-8 already is an encoding specified down to the bit level. Define "binary data". — , May 22 '14 at 18:45
You are asking for trouble, there are few applications around that will treat utf-8 encoded strings as binary data properly. If it doesn't fall over on the data not being properly utf-8 encoded then the data will get arbitrarily corrupted when the string is normalized. Boilerplate is to encode the binary data first, base64 is *very* common. — Hans Passant, May 22 '14 at 19:12
@HansPassant I think we have a valid UTF-8 string being sent over a channel that handles "binary" data (which I would normally take to mean "any bytes whatsoever with no interpretation assumed"), *not* arbitrary binary data being treated as a UTF-8 string. — , May 22 '14 at 19:39

score 0 · Answer 1 · answered May 22 '14 at 18:46

0

UTF-8 is already 'binary' in that it is represented by a number of bytes, so no need to 'encode'.

However, if you want to send / receive a buffer, you need to have a protocol which defines the start / end of the buffer. In the case of UTF-8, this would be the zero-terminating byte. For buffers, the normal convention would be to prefix it with a length-field instead of searching for a zero byte at the end, but also other conventions do exist.

So it depends not on UTF-8 on how to do this, but on the buffer-sending-convention.

answered May 22 '14 at 18:46

Rutger Nijlunsing

4,861
1
21
24

1

I think it's misleading to say UTF-8 is zero terminated. UTF-8 is a mapping between bytes and codepoints, how you store those bytes (in particular, how you denote the length/end of the byte string) is orthogonal. The codepoint 0, corresponding to the byte 0, has no particular meaning and UTF-8 (and most other unicode-related algorithms) doesn't care where and how often it occurs. You can, and often should, represent UTF-8 strings with length prefixing. – May 22 '14 at 18:50
@delnan: Zero terminating a UTF8 string is perfectly fine. It was designed to be compatible with C string handling. But you're right. UTF8 does not require termination or length. Those things are details it isn't concerned with. – Zan Lynx May 22 '14 at 18:54
1

@ZanLynx Zero termination for UTF-8 is *exactly* as well as for, say, ASCII. It works for both, and "when in Rome do as the Romans do", but in a vacuum (and in any language except C), it's generally worse than an explicit length. – May 22 '14 at 19:03

score 0 · Answer 2 · answered May 22 '14 at 19:36

As Rutger Nijlunsing said, a UTF-8 string is already binary.

Most methods that transfer data stored in buffers use a pointer and length parameter to transfer the data. In your case the pointer would the start of your string and you can use the null terminator to find the length if you don't already have it stored somewhere in your code.

score 0 · Answer 3 · answered May 24 '14 at 20:22

Where buffer is a string coded as "utf-8" or "ISO-8859-1".

This is going to get you in trouble. You need to pick one or the other and be consistent with it. If buffer is UTF-8 encoded then you can pass it to net_http_send() as-is:

ssize_t send_utf8 (char *buffer)
{
    net_http_send(buffer, strlen(buffer));
    ...
}

But if it is any other encoding, you have to convert it first:

ssize_t send_utf8 (char *buffer)
{
    char *utf8_buffer = convert_to_utf8(buffer, strlen(buffer));
    net_http_send(utf8_buffer, strlen(utf8_buffer));
    ...
}

How you actually do that conversion depends on your development environment, though. C has no concept of UTF-8 (that was not added until C++11), so you will have to use a third-party library, such as iconv or ICU, or else write your own conversion code (UTF-8 is fairly easy to implement by hand). But either way, you need to know the encoding that buffer uses, because you have to decode it to UTF-32 before you can then re-encode it as UTF-8 (libraries handle that detail for you).

How to convert an UTF-8 string?

3 Answers3