86

I'm going to be storing a large array of byte values (most likely over a million) in Javascript. If I use a normal array with normal numbers, that will take 8 MB, because numbers are stored as IEEE doubles, but if I can store it as bytes, it will be only 1 MB.

I'd like to avoid wasting that much space for obvious reasons. Is there a way to store bytes as bytes instead of doubles? Browser compatibility isn't an issue for me, as long as it works in Chrome. This is in HTML5, if that makes a difference.

Kendall Frey
  • 43,130
  • 20
  • 110
  • 148
  • 12
    [Typed arrays.](https://developer.mozilla.org/en-US/docs/JavaScript_typed_arrays) – Pointy Sep 08 '12 at 15:28
  • http://stackoverflow.com/questions/5612637/storing-byte-array-in-html5-sqlite-database and http://siphon9.net/loune/2011/05/javascript-arraybuffer-binary-handling-in-javascript/ these links may help you – Gyan Chandra Srivastava Sep 08 '12 at 15:35
  • @Pointy: typed array for bytes? http://www.khronos.org/registry/typedarray/specs/latest/#7 could you please elaborate? i would like to understand that – naveen Sep 08 '12 at 16:02
  • @naveen Type array for bytes (8 bits) is `Int8Array` or `Uint8Array` depending on whether you want you byte to be signed or not. – HoLyVieR Sep 08 '12 at 16:08
  • 7
    Re: "as long as it works in Chrome" -- I'm tempted to give you a -1 for that (I haven't but I'm tempted), because that's the same attitude that left us with a legacy of sites that only work in IE6. If what you want can *only* be done in Chrome, then so be it, but don't limit your site. There are several browsers out there, and they all work with HTML5. Write your site for as broad an audience as possible. – Spudley Sep 08 '12 at 18:12
  • 20
    @Spudley It's not intended to be a public site. It's written for an audience of just me. – Kendall Frey Sep 08 '12 at 18:16

4 Answers4

90

By using typed arrays, you can store arrays of these types:

Type Value Range Size(bytes)
Int8Array -128 to 127 1
Uint8Array 0 to 255 1
Uint8ClampedArray 0 to 255 1
Int16Array -32768 to 32767 2
Uint16Array 0 to 65535 2
Int32Array -2147483648 to 2147483647 4
Uint32Array 0 to 4294967295 4
Float32Array -3.4E38 to 3.4E38 4
Float64Array -1.8E308 to 1.8E308 8
BigInt64Array -2^63 to 2^63 - 1 8
BigUint64Array 0 to 2^64 - 1 8

Demo in Stack Snippets & JSFiddle

var array = new Uint8Array(100);
array[42] = 10;
console.log(array[42]);
KyleMit
  • 30,350
  • 66
  • 462
  • 664
Kendall Frey
  • 43,130
  • 20
  • 110
  • 148
  • 11
    Yes! That cut the time generating an array of 800,000+ elements from 2.4 seconds to 0.12 seconds. Using `Float32Array` makes an amazing difference! – Robbie Wxyz Nov 05 '13 at 19:27
  • 1
    I'm having issues converting a Blob that's returned by a fetch() promise to a C# byte[]. Do you have any wisdom on what approach I should take. – Reahreic Mar 02 '21 at 21:05
19
var array = new Uint8Array(100);    
array[10] = 256;
array[10] === 0 // true

I verified in firefox and chrome, its really an array of bytes :

var array = new Uint8Array(1024*1024*50);  // allocates 50MBytes
reuns
  • 854
  • 10
  • 14
2

You could store the data in an array of strings of some large fixed size. It should be efficient to access any particular character in that array of strings, and to treat that character as a byte.

It would be interesting to see the operations you want to support, perhaps expressed as an interface, to make the question more concrete.

Hew Wolff
  • 1,489
  • 8
  • 17
1

I wanted a more exact and useful answer to this question. Here's the real answer (adjust accordingly if you want a byte array specifically; obviously the math will be off by a factor of 8 bits : 1 byte):

class BitArray {
  constructor(bits = 0) {
    this.uints = new Uint32Array(~~(bits / 32));
  }

  getBit(bit) {
    return (this.uints[~~(bit / 32)] & (1 << (bit % 32))) != 0 ? 1 : 0;
  }

  assignBit(bit, value) {
    if (value) {
      this.uints[~~(bit / 32)] |= (1 << (bit % 32));
    } else {
      this.uints[~~(bit / 32)] &= ~(1 << (bit % 32));
    }
  }

  get size() {
    return this.uints.length * 32;
  }

  static bitsToUints(bits) {
    return ~~(bits / 32);
  }
}

Usage:

let bits = new BitArray(500);
for (let uint = 0; uint < bits.uints.length; ++uint) {
  bits.uints[uint] = 457345834;
}
for (let bit = 0; bit < 50; ++bit) {
  bits.assignBit(bit, 1);
}
str = '';
for (let bit = bits.size - 1; bit >= 0; --bit) {
  str += bits.getBit(bit);
}
str;

Output:

"00011011010000101000101100101010
 00011011010000101000101100101010
 00011011010000101000101100101010
 00011011010000101000101100101010
 00011011010000101000101100101010
 00011011010000101000101100101010
 00011011010000101000101100101010
 00011011010000101000101100101010
 00011011010000101000101100101010
 00011011010000101000101100101010
 00011011010000101000101100101010
 00011011010000101000101100101010
 00011011010000101000101100101010
 00011011010000111111111111111111
 11111111111111111111111111111111"

Note: This class is really slow to e.g. assign bits (i.e. ~2s per 10 million assignments) if it's created as a global variable, at least in the Firefox 76.0 Console on Linux... If, on the other hand, it's created as a variable (i.e. let bits = new BitArray(1e7);), then it's blazingly fast (i.e. ~300ms per 10 million assignments)!

For more info, see here:

Note that I used Uint32Array because there's no way to directly have a bit/byte array (that you can interact with directly) and because even though there's a BigUint64Array, JS only supports 32 bits:

Bitwise operators treat their operands as a sequence of 32 bits

...

The operands of all bitwise operators are converted to...32-bit integers

Andrew
  • 5,839
  • 1
  • 51
  • 72