0

I hope to figure out a node.js system with workerpool to handle cpu intensive tasks, but there is something confused me about the cpu usage in multiple routes. A scenario is like this:

route1.js:
  const workerpool = require('workerpool');
  const pool = workerpool.pool(__dirname + '/job1.js');
  pool.exec.......
route2.js:
  const workerpool = require('workerpool');
  const pool = workerpool.pool(__dirname + '/job2.js');
  pool.exec.......
route3.js:
  const workerpool = require('workerpool');
  const pool = workerpool.pool(__dirname + '/job3.js');
  pool.exec.......

When the node.js use these three files, they will create their own workerpool, and since worker_thread number and its control is via node.js internals, is this possible to create threshold problem? And how to use workerpool in a correct way, thanks a lot.

abramhum
  • 443
  • 2
  • 8
  • 20

1 Answers1

1

What I'd do is only have one pool of worker. Workers can expose multiple functions so your worker could expose job1, job2 and job3 without problem. If you create a pool for each, you'll need to consider that pools may work against each other...

Consider your pools getting 100% of your CPUs, that means you can require up to 300% what you have available if all 3 pools are full.

If you allocated them 33% each, that means you can require up to 100% which is good, but if only job1 is heavily required at one time it will only be able to use 33% of available resources.

By using a single pool, you can reach up to 100% without requiring more than 100% available.

Salketer
  • 14,263
  • 2
  • 30
  • 58
  • There somehow has an api called pool.terminate(), if the threadpool theory is valid in workerpool, why there needs a pool.terminate to delete all the worker_thread? Will that clean the other workerpool I declared in other functions? If not, if there possible a dynamic and static workerpool can exist at the same time, thanks. – abramhum May 18 '23 at 05:40
  • From what I understand it will terminate the pool, so only its worker_thread. I do not understand what you mean by dynamic and static workerpool. If you have a defined and precise use case, I suggest you explain it or else only generic/vague answers can be given. – Salketer May 19 '23 at 06:11