9

I have Base64 encoded data that is in UTF-16 I am trying to decode the data but most libraries only support UTF-8. I believe I have to drop the null bites but I am unsure how.

Currently I am using David Chambbers Polyfill for Base64, but I have also tried other libraries such as phpjs.org, none of which support UTF-16.

One thing to point out is on Chrome the atob method works with out problem, Firefox I get results described here, and in IE I am only returned the first character.

Any help is greatly appreciated

Community
  • 1
  • 1
Don P
  • 570
  • 1
  • 5
  • 12
  • 2
    I'm a bit confused as to what you are trying to do. You have Base64 data, but it's in UTF-16? I'm not sure, but I have a hunch that you can just chop off every other byte and be left with what you need. – Brad Jan 29 '13 at 21:12
  • 1
    Convertions can be easily done using PHP : http://php.net/manual/en/function.mb-convert-encoding.php – sdespont Jan 29 '13 at 21:14
  • 1
    @Brad chopping off every other byte won't do if the data is non-ascii. – John Dvorak Jan 29 '13 at 21:16
  • @JanDvorak, If it is Base-64 data, it *is* ASCII. Am I not understanding something? – Brad Jan 29 '13 at 21:23
  • @Brad I have updated my question to make it a little more clear, I believe. – Don P Jan 29 '13 at 21:23
  • 2
    @sdespont, What does PHP have to do with JavaScript?... – Brad Jan 29 '13 at 21:23
  • @Brad it's a suggestion to offload the conversion to server. – John Dvorak Jan 29 '13 at 21:24
  • 1
    @JanDvorak, I see that, but there are plenty of reasons why you may want to do this in JavaScript. – Brad Jan 29 '13 at 21:24
  • 2
    @Brad I guess it's base64-encoded UTF-16 (which could contain any text), not UTF-16-encoded base64 (which would be kinda wasteful). – John Dvorak Jan 29 '13 at 21:25
  • @JanDvorak, My read is the other way around. "I have Base64 encoded data that is in UTF-16". The reason I think that is because it wouldn't matter what character set his base64 library supported otherwise. – Brad Jan 29 '13 at 21:26
  • @Brad I believe that every normal Base64-decoding library takes a string as an input, not a byte stream. Thus, the input encoding won't matter. – John Dvorak Jan 29 '13 at 21:30
  • @user1227074, It would be nice if you could clarify your question. Do you have base64 encoded data currently stored within a UTF-16 string? Or do you want to decode some UTF-16 string currently encoded with base64? – Brad Jan 29 '13 at 23:41
  • @Brad I have a UTF-16 string that is Base64 encoded – Don P Jan 30 '13 at 17:37
  • @user1227074, Then it doesn't matter what your Base64 decoder does... – Brad Jan 30 '13 at 17:38

1 Answers1

26

You want to decode UTF-16, not convert to UTF-8. Decoding means that the result is a string of abstract characters. Of course there is an internal encoding for strings as well, UTF-16 or UCS-2 in javascript, but that's an implementation detail.

With strings the goal is that you don't have to worry about encodings but just about manipulating characters "as they are". So you can write string methods that don't need to decode input at all. Of course there are many edge cases where this falls apart.

You cannot decode utf-16 just by removing nulls. I mean this will work fine for the first 256 code points of unicode, but you will get garbage when any of the other ~110000 characters in unicode are used. You cannot even get the most popular non-ASCII characters like em dash or any smart quotes working.

Also, looking at your example, it looks like UTF-16LE.

//Braindead decoder that assumes fully valid input
function decodeUTF16LE( binaryStr ) {
    var cp = [];
    for( var i = 0; i < binaryStr.length; i+=2) {
        cp.push( 
             binaryStr.charCodeAt(i) |
            ( binaryStr.charCodeAt(i+1) << 8 )
        );
    }

    return String.fromCharCode.apply( String, cp );
}

var base64decode = atob; //In chrome and firefox, atob is a native method available for base64 decoding

var base64 = "VABlAHMAdABpAG4AZwA";
var binaryStr = base64decode(base64);
var result = decodeUTF16LE(binaryStr);

Now you can even get smart quotes working:

var base64 = "HCBoAGUAbABsAG8AHSA="
var binaryStr = base64decode(base64);
var result = decodeUTF16LE(binaryStr);
//"“hello”"
Esailija
  • 138,174
  • 23
  • 272
  • 326