througt module accepts only 8/16 files

Question

I am currently working with throught2 module to write gulp plugin to replace something in the buffer, below is a simplified version of it

module.exports = function(replaceData: DataChange | DataChange[]) {
  let count = 0;
  return through.obj(function (file: any, encoding: any, callback: any) {
    this.push(file);
    callback(null, file);
    console.log('@COUNT: ', ++count);
  });
};

I found out that it will pick up only 8 files and will continue, ignoring the others, my output will be like:

@COUNT: 1
@COUNT: 2
...
@COUNT: 8

If I don't return anything in callback

module.exports = function(replaceData: DataChange | DataChange[]) {
  let count = 0;
  return through.obj(function (file: any, encoding: any, callback: any) {
    this.push(file);
    callback();
    console.log('@COUNT: ', ++count);
  });
};

it will process 16 files. How to increase the number of files through will pick up?

Answer 1

What I found out for through2 is expecting streams that inherits from Readable & Writable.

Default Readable/Writable hihghWaterMark is 16K. Without a consumer for the through2 transform stream relieving pressure, it hits the high water mark and buffers after 16 records. In case of returning file in callback(case 1) it is 8 files in and 8 files out, that's why it's picking only 8.

The solution will be to pass to through2 options, i.e.

return through.obj({highWaterMark: 32}, function (file: any, encoding: any, callback: any) {

and it will pick up 32 files in my case.

Supported options include:

• highWaterMark (defaults to 16)
• defaultEncoding (defaults to 'utf8')
• encoding - 'utf8', 'base64', 'utf16le', 'ucs2' etc. If specified, a StringDecoder decoder will be attached to the stream.
• readable {boolean}
• writable {boolean}
• allowHalfOpen {boolean} If set to false, then the stream will automatically end the readable side when the writable side ends and vice versa.

Hope this will help you and save your time