iconv encoding conversion problem

Question

I am having trouble converting strings from utf8 to gb2312. My convert function is below

void convert(const char *from_charset,const char *to_charset, char *inptr, char *outptr)
{
    size_t inleft = strlen(inptr);
    size_t outleft = inleft;
    iconv_t cd;     /* conversion descriptor */

    if ((cd = iconv_open(to_charset, from_charset)) == (iconv_t)(-1)) 
    {
            fprintf(stderr, "Cannot open converter from %s to %s\n", from_charset, to_charset);
            exit(8);
    }

    /* return code of iconv() */
    int rc = iconv(cd, &inptr, &inleft, &outptr, &outleft);
    if (rc == -1) 
    {
            fprintf(stderr, "Error in converting characters\n");

            if(errno == E2BIG)
                    printf("errno == E2BIG\n");
            if(errno == EILSEQ)
                    printf("errno == EILSEQ\n");
            if(errno == EINVAL)
                    printf("errno == EINVAL\n");

            iconv_close(cd);
            exit(8);
    }
    iconv_close(cd);
}

This is an example of how I used it:

int len = 1000;
char *result = new char[len];
convert("UTF-8", "GB2312", some_string, result);

edit: I most of the time get a E2BIG error.

Answer 1

outleft should be the size of the output buffer (e.g. 1000 bytes), not the size of the incoming string.

When converting, the string length usually changes in the process and you cannot know how long it is going to be until afterwards. E2BIG means that the output buffer wasn't large enough, in which case you need to give it more output buffer space (notice that it has already converted some of the data and adjusted the four variables passed to it accordingly).

Answer 2

As others have noted, E2BIG means that the output buffer wasn't large enough for the conversion and you were using the wrong value for outleft.

But I've also noticed some other possible problems with your function. Namely, with the way your function works, your caller has no way of knowing how many bytes are in the output string. Your convert() function neither nul-terminates the output buffer nor does it have a means of telling its caller the number of bytes it wrote to outptr.

If you want to deal with nul-terminates strings (and it appears that's what you want to do since your input string is nul-terminated), you might find the following approach to be much better:

char *
convert (const char *from_charset, const char *to_charset, const char *input)
{
 size_t inleft, outleft, converted = 0;
 char *output, *outbuf, *tmp;
 const char *inbuf;
 size_t outlen;
 iconv_t cd;

 if ((cd = iconv_open (to_charset, from_charset)) == (iconv_t) -1)
  return NULL;

 inleft = strlen (input);
 inbuf = input;

 /* we'll start off allocating an output buffer which is the same size
  * as our input buffer. */
 outlen = inleft;

 /* we allocate 4 bytes more than what we need for nul-termination... */
 if (!(output = malloc (outlen + 4))) {
  iconv_close (cd);
  return NULL;
 }

 do {
  errno = 0;
  outbuf = output + converted;
  outleft = outlen - converted;

  converted = iconv (cd, (char **) &inbuf, &inleft, &outbuf, &outleft);
  if (converted != (size_t) -1 || errno == EINVAL) {
   /*
    * EINVAL  An  incomplete  multibyte sequence has been encoun­-
    *         tered in the input.
    *
    * We'll just truncate it and ignore it.
    */
   break;
  }

  if (errno != E2BIG) {
   /*
    * EILSEQ An invalid multibyte sequence has been  encountered
    *        in the input.
    *
    * Bad input, we can't really recover from this. 
    */
   iconv_close (cd);
   free (output);
   return NULL;
  }

  /*
   * E2BIG   There is not sufficient room at *outbuf.
   *
   * We just need to grow our outbuffer and try again.
   */

  converted = outbuf - out;
  outlen += inleft * 2 + 8;

  if (!(tmp = realloc (output, outlen + 4))) {
   iconv_close (cd);
   free (output);
   return NULL;
  }

  output = tmp;
  outbuf = output + converted;
 } while (1);

 /* flush the iconv conversion */
 iconv (cd, NULL, NULL, &outbuf, &outleft);
 iconv_close (cd);

 /* Note: not all charsets can be nul-terminated with a single
  * nul byte. UCS2, for example, needs 2 nul bytes and UCS4
  * needs 4. I hope that 4 nul bytes is enough to terminate all
  * multibyte charsets? */

 /* nul-terminate the string */
 memset (outbuf, 0, 4);

 return output;
}