9

From the node docs regarding the creation of typed arrays from Buffers:

The buffer's memory is interpreted as an array, not a byte array. That is, new Uint32Array(new Buffer([1,2,3,4])) creates a 4-element Uint32Array with elements [1,2,3,4], not an Uint32Array with a single element [0x1020304] or [0x4030201].

This contrasts to plain javascript, where creating a typed array view from an ArrayBuffer uses the ArrayBuffer's memory as bytes (like a reinterpret_cast in C++). I need this behavior in node when operating on node Buffers.

I could convert the Buffer to an ArrayBuffer, but this is too slow for my application. (I've tried many methods -- but they're all O(n) time.) (Edit: the fastest method I've found is this, which is a single memmove op and pretty fast, but still has at least momentary 2x memory consumption until the reference to the original Buffer is released.)

Is there any (fast/O(1)) way to get a typed array from a Buffer, using the Buffer's contents as bytes instead of elements? (The needed typed array element size is >1 byte, needless to say.)

ZachB
  • 13,051
  • 4
  • 61
  • 89
  • What type of operation is supposed to do with the array? If the read-only, then why not refer to elements in the buffer directly through the offset? Ex.: `buf.readUInt32LE(i*4)` where `i` is array index? – stdob-- Jul 17 '15 at 11:47
  • 1
    I need to iterate through the array quickly and repeatedly. The `read*` methods are really slow. – ZachB Jul 17 '15 at 15:37
  • I think that there is no place more quickly: If we need a Int32 value of the index `i`: `var k = i*4; return buf[k] | buf[k+1]<<8 | buf[k+2]<<16 | buf[k+3]<<256;` – stdob-- Jul 17 '15 at 16:55
  • I invented a bicycle: if we look at the module `buffer.js` - is exactly how implemented reading. – stdob-- Jul 17 '15 at 17:08

2 Answers2

7

As far as I know, it is not possible to do this without making a copy of the data in memory. Even your example new Uint32Array(new Buffer([1,2,3,4])) internally does this (meaning it's not O(1)).

Note that typed arrays are just views of an ArrayBuffer (not Buffer, that's not possible). new Uint32Array(array) creates an ArrayBuffer of 4 * array.length bytes. You can access it with uint32Array.buffer. The constructor treats your Buffer no different than a normal Array.

The best solution I know is the one you already found.

Another issue with using Uint32Array for what you try to do is that it depends on platform byte order. You could either iterate over the Buffer like this or use DataView if you want to be safe.

Community
  • 1
  • 1
Jan Schär
  • 808
  • 6
  • 9
3

As of node 4.0, Buffers are Uint8Arrays and new views can be constructed on them directly:

var b = new Buffer([1,2,3,4]);
new Uint32Array(b.buffer, b.byteOffset);

This mini-example assumes that b is larger than ~4096 (the threshold for a Buffer owning its own chunk of memory vs. sharing a larger chunk). See Convert a binary NodeJS Buffer to JavaScript ArrayBuffer for more info -- generally you should call b.slice() to operate on only the memory you own.

ZachB
  • 13,051
  • 4
  • 61
  • 89
  • NB: `new Uint32Array(b.buffer, b.byteOffset).length` -> `2046` ... doesnt work. You are right -- can be constructed directly, but "as uint8array" is not bytes. – shaunc Jan 22 '18 at 02:26
  • @shaunc it does work, look at the contents of the result. The unexpected length is a side-effect of node's shared array buffer allocator. Use `b.slice` at least when your buffer is small, or always to be safe. – ZachB Jan 22 '18 at 06:58
  • `new Uint32Array(b.buffer, b.byteOffset).slice(0,4)` -> `Uint32Array [ 67305985, 0, 8192, 0 ]` ? the same when I rewrite both "b"s as `b.slice(0,4)`? (NB: node 9.3.0) OOPS ... I guess thats right :) Grumble ... won't let me correct my mistake by upvote until you edit -- any edit will do and I'll upvote. Sorry – shaunc Jan 24 '18 at 01:22
  • No prob, clarified. Cheers. – ZachB Jan 24 '18 at 01:48