1

Does anyone know why this happens

    const encoder = new TextEncoder()
    const decoder = new TextDecoder()
    const array = new Uint8Array([200])
    console.log(encoder.encode(decoder.decode(array))) // prints [239, 191, 189]

As far as I know, decoding and encoding anything should give me the exact same value. Should not it just print [200] back?

bugwheels94
  • 30,681
  • 3
  • 39
  • 60

1 Answers1

3

You are decoding [200] which is an invalid UTF-8 sequence and results in �

const decoder = new TextDecoder()
const array = new Uint8Array([200])
console.log(decoder.decode(array)) // prints �

Since the binary representation of 200 is 11001000, the decoder thinks this is a two-byte UTF-8 character:

In UTF-8 encoding, a byte starting with 110 indicates that it's the first byte of a two-byte encoded character.

Then when you try to encode � you just get the following output [239, 191, 189] or 0xEF 0xBF 0xBD in hexadecimal

const encoder = new TextEncoder()
const array = new Uint8Array([])
console.log(encoder.encode("�"))
Asleepace
  • 3,466
  • 2
  • 23
  • 36
  • 1
    oh nice! Thanks for the explanation! I want to store this array as string in redis, do you know what approach I should take then please – bugwheels94 Apr 03 '23 at 23:23
  • @bugwheels94 you should use `encoder.encode([200])` first, might be that you are running `decode` then `encode` vs. the other way around. If that doesn't work, what about just using `JSON.stringify`? – Asleepace Apr 03 '23 at 23:27
  • Hmm, in that case .toString() is also a more compact solution but I was looking for something compressed as TextDecoder cause .toString adds a comma in between values hence increasing a bit – bugwheels94 Apr 03 '23 at 23:29
  • @bugwheels94 edited my last comment, but I think the root of the issue is you are trying to `decode` before `encoding`, if you flip those it should be fine! – Asleepace Apr 03 '23 at 23:30
  • 1
    encoder then decoder is pretty smart. It is giving the output exactly like native .toString() though. Thanks for the help! – bugwheels94 Apr 03 '23 at 23:34
  • @bugwheels94 you want to encode this data in another format. The most common one for this task is base64. See https://stackoverflow.com/questions/9267899/arraybuffer-to-base64-encoded-string – Kaiido Apr 04 '23 at 00:40
  • @Kaiido yes base64 is also a great option. Though, I discovered that I can insert Typed array in redis directly by converting it to buffer. so solved using that. But thank you so much for advice! – bugwheels94 Apr 05 '23 at 02:06