What is a buffer in Node.js?

Question

As you can read in the Node.js documentation on the Buffer class, a buffer

is similar to an array of integers but corresponds to a raw memory allocation outside the V8 heap.

So far, so good.

What now puzzles me is the question what a buffer is technically speaking. Is it an array with just some additional functions for creating and converting to strings using specific encodings?

Or is there "more" to it?

[Deep-dive of buffers in NodeJS](https://medium.freecodecamp.org/do-you-want-a-better-understanding-of-buffer-in-node-js-check-this-out-2e29de2968e8) — Nino Filiu, Apr 09 '19 at 11:19

phihag · Accepted Answer · 2013-01-27T19:10:12.897

35

A Buffer is a chunk of memory, just like you would have it in C/C++. You can interpret this memory as an array of integer or floating point numbers of various lengths, or as a binary string. Unlike higher-level data structures like arrays, a buffer is not resizable.

It corresponds roughly to:

char* or char[] in C/C++
byte[] in Java
A mutable bytes or a non-resizable bytearray in Python
Strings in php if they were mutable

edited Jan 27 '13 at 19:10

answered Jan 27 '13 at 19:04

phihag

278,196
72
453
469

1

Okay, the comparison to a char[] or char* in C makes sense to me. Thanks for clarifying this! This analogy helps me to understand the meaning of buffers :-). – Golo Roden Jan 27 '13 at 19:08
Does it make any difference if I save a buffer: `new Buffer(intArray)` or if I convert the intArray first to a decimalArray e.g. with `decimal >>> 0` for every element? Or asked in a different way, the buffer is always binary data, so if I don't convert it just gets converted automatically. – Andi Giga May 25 '16 at 15:49
@AndiGiga Sorry, I don't quite understand your question. I am unaware of any `decimalArray` class in JavaScript. I'd advise you to simply ask a question of your own (not a comment), and describe in detail what intArray and decimalArray are. – phihag May 25 '16 at 16:14
1

@phihag Could you give an example like where we supposed to use buffer over regular data type like array itself. – Sachin Bhandari Jun 03 '19 at 10:24
1

@SachinBhandari Say you want to read a 1GB binary file. You can represent it as an array of integers or a Buffer. With a Buffer, you need roughly 1GB storage (+ on the order of 32 Bytes overhead). Getting an element from the buffer will be one boundary check and one memory access. ... – phihag Jun 03 '19 at 11:06
1

@SachinBhandari ... If you were to use an array of integers between 0 and 255, your memory requirements would be 1G * pointer size. On a 64 Bit system, that works out to 8GB. There's also negligible overhead (the numbers 0-255 in memory, and the integer array), but unless you're careful also some non-negligible overhead, caused by overprovisioning of the array. Also, getting an element from the array will entail *two* memory accesses; one for the element and one for the item. – phihag Jun 03 '19 at 11:07
@phihag how buffers are so efficient in terms of memory and how do they keep the track of next element for traversal or so – Sachin Bhandari Jun 04 '19 at 05:51
@SachinBhandari Internally, a Buffer is a raw memory region. If you read from it (say with an `Uint8Array`), the raw memory is interpreted. That is opposed to an array, where every element is an object. It needs to be this way because an array can contain arbitrary objects, like `[1, 2, 3, 'surprise', ['array']]`. Any iterators over the Buffer or views into it are not the Buffer's problem, and are managed elsewhere. Access into raw memory is simple: If I want element 42 of a `Uint32Array` to a buffer starting at 0x12340000, then I simply read the 32 Bits=4 Bytes at address 0x12340000 + 42 * 4 – phihag Jun 04 '19 at 08:31
@phihag Thank you so much for washing my doubts out, could you explain one more thing, ```1GB storage (+ on the order of 32 Bytes overhead)``` and ```1G * pointer size. On a 64 Bit system, that works out to 8GB```, I didn't quite get these memory calculations that you gave in your first comment to me. – Sachin Bhandari Jun 04 '19 at 11:39
@SachinBhandari Sorry, could you be a little bit more precise about your question? Every JavaScript object requires some amount of memory for its type, garbage collection, and properties (e.g. an array needs to know its allocated size and its actual size). This is the overhead, which I estimated at 32 Bytes. The data storage of a Buffer is simple; it needs 1GB to store 1GB. But every object of an array could be of an arbitrary type, so the usual implementation would be to have 1G pointers to arbitrary objects. On 64 Bit systems, a pointer is 8 Bytes long. – phihag Jun 04 '19 at 15:19
1

@phihag What is the difference between ```Buffer```, ```TypedArray``` and ```ArrayBuffer```? If we have ```ArrayBuffer``` then why ```Buffer``` class – Sachin Bhandari Jun 05 '19 at 13:29
2

@SachinBhandari node.js's `Buffer` predates `ArrayBuffer`. At the time, there was no `ArrayBuffer`, and that's why the `Buffer` class in node.js was created. Browser vendors did not like the `Buffer` interface, and created their own `ArrayBuffer`. The TypedArray family of classes provides different interpretations of the memory region of an `ArrayBuffer`. Nowadays, `Buffer` is just a `Uint8Array` (which itself is one of the TypedArray implementations). – phihag Jun 05 '19 at 19:35

score 17 · Answer 2 · answered Sep 10 '16 at 20:23

BUFFER is a temporary holding spot for data being moved from one place to another.

In order to understand what is Buffer, we need to know how a computer will process things. See the chart below.

The concept is like if you are watching a Youtube Video, you can start to watch a video without downloading the whole video. If your internet speed is too slow, you would see "buffering", that means the computer is trying to collect data in order for you to keep watching that video.

Somewhat understood. Can you specify a practical example(code) of this thing? — Krishnadas PC, Oct 07 '17 at 16:45

score 6 · Answer 3 · edited Mar 13 '18 at 16:43

6

Explanation from http://nodejitsu.com/...

Buffers are instances of the Buffer class in node, which is designed to handle raw binary data. Each buffer corresponds to some raw memory allocated outside V8. Buffers act somewhat like arrays of integers, but aren't resizable and have a whole bunch of methods specifically for binary data. In addition, the "integers" in a buffer each represent a byte and so are limited to values from 0 to 255 (2^8 - 1), inclusive.

What is a buffer in Node.js?

3 Answers3

Linked