Stack Overflow
Stack Overflow

Questions tagged [worker-pool]

24 questions
12
votes
3 answers

Asynchronous multiprocessing with a worker pool in Python: how to keep going after timeout?

I would like to run a number of jobs using a pool of processes and apply a given timeout after which a job should be killed and replaced by another working on the next task. I have tried to use the multiprocessing module which offers a method to…
fuenfundachtzig
  • 7,952
  • 13
  • 62
  • 87
5
votes
2 answers

"Workerpool Worker terminated Unexpectedly" for Mocha tests in CircleCI

I have TypeScript tests running with Yarn and Mocha and they work fine locally. When I deploy via CircleCI, however, I get this: 1) Uncaught error outside test suite: Uncaught Workerpool Worker terminated Unexpectedly exitCode: `null` …
tlhinman
  • 73
  • 8
4
votes
1 answer

Optimal size of worker pool

I'm building a Go app which uses a "worker pool" of goroutines, initially I start the pool creating a number of workers. I was wondering what would be the optimal number of workers in a mult-core processor, for example in a CPU with 4 cores ? I'm…
guilhebl
  • 8,330
  • 10
  • 47
  • 66
2
votes
0 answers

How to share singleton across multiple threads in nodejs

I'm trying to have a singleton work across multiple threads in node/ts. But for some reason only main event loop has access to it. Is there any way I can have a singleton updated in the main loop, and have the changes reflected in a worker…
stefan_thedev
  • 167
  • 6
2
votes
1 answer

Using WorkerPool to multithread through a list of URLs

I'm trying to use multithreads to go through a txt file of urls and scrape the contents found at each url. This works for about 20 URLs (not consistent how many) but then consistently gets stuck on the last url in the file. It doesn't seem to be…
Jolijt Tamanaha
  • 333
  • 2
  • 9
  • 23
2
votes
1 answer

Java wordcount: a mediocre implementation

I implemented a wordcount program with Java. Basically, the program takes a large file (in my tests, I used a 10 gb data file that contained numbers only), and counts the number of times each 'word' appears - in this case, a number (23723 for…
DanGordon
  • 671
  • 3
  • 8
  • 26
1
vote
1 answer

Broken Pipe Error when writing to stdin of a python application from a go application

I have implemented a worker pool to submit my jobs to the python script. func NewWorker(index int, workerConfig config.AppConfig, logger log.PrefixedLogger) error { var worker entities.Worker worker.ID = index worker.Config =…
mster3313
  • 33
  • 5
1
vote
0 answers

Why we need async Callback in Node JS Since Event Loop offers Worker Pool to handle expensive task?

I was studying how Node JS improves performance for multiple concurrent request! After reading couple of blogs I found out that: When any request come an event is triggered and corresponding callback function is placed in Event Queue. An event…
Bishal Imtiaz
  • 479
  • 1
  • 5
  • 10
1
vote
0 answers

Out of Memory Exception - Unhandled RangeError

I've created an NodeJS service that reads messages from an ActiveMQ queue, set up remotely, and picks up these messages, processes it and pushes some data to a GUI. The issue i'm facing is while trying to process these messages on NodeJS end when…
sanmyth
  • 85
  • 5
1
vote
1 answer

Download M total files with N simultaneous Async HTTP clients, where M is large and N is configurable

I'm trying to write a script which will download at most N files simultaneously via HTTP. I've previously used AnyEvent::Worker::Pool to manage a pool of BLOCKING tasks. I've also used AnyEvent::HTTP in combination with AnyEvent->condvar to manage…
David-SkyMesh
  • 5,041
  • 1
  • 31
  • 38
0
votes
0 answers

Golang worker pool execute tasks in specific order

I have records in my table, the table has column called flow which takes values in range from 1 to 10. Every 3rd row has flow 3, every 5th row has flow 5 and so on for other rows and flows: +------+--------+ | id | flow | +------+--------+ | 1…
nrgx
  • 325
  • 3
  • 13
0
votes
0 answers

How to override a function from an external library on nodejs

I want to override the .exec function on workerpool but it's calls my console.log twice, can't figure out why : export const queues = workerpool.pool({ maxWorkers: require("os").cpus().length, }); const originalExec = queues.exec; queues.exec =…
GhostLexly
  • 1
  • 1
0
votes
1 answer

Does workerpool declared in multiple routes can still maintain its cpu usage without caring about threshold

I hope to figure out a node.js system with workerpool to handle cpu intensive tasks, but there is something confused me about the cpu usage in multiple routes. A scenario is like this: route1.js: const workerpool = require('workerpool'); const…
abramhum
  • 443
  • 2
  • 8
  • 20
0
votes
0 answers

Passing Electron NativeImage to workerpool without having problem

I capture the page with Electron and try to pass this NativeImage object to a workerpool. I am passing as an PNG image since I need in this format from the worker. However, when I try to use it in Sharp, I receive an error saying ReferenceError:…
Wokers
  • 107
  • 3
  • 13
0
votes
0 answers

How to keep workerpool active and user can send persistant input to stdin?

Is workerpool capable of keeping the spawn function alive and can the user send input to stdin? If not what are the best options to handle a large amount of alive child process in node js
Mark Wasfy
  • 167
  • 1
  • 13
1
2