1

I can assure you, this is not your typical "how to watch for changes on a JS variable" post!

So I have some worker threads gathering data for a parent thread in Node.js.

The parent thread processes the data in ticks of a set timeframe, just keeping a general eye on things.

Occasionally the data the worker threads send to the parent thread is really interesting, and when the parent thread gets interesting data I want it to be processed immediately. (Without waiting for the next tick to roll around.)

What I am stuck on is how to most quickly and efficiently communicate to the main worker thread that dataIsInteresting = true.

The worker threads know that their data is interesting immediately, but the main thread only knows that when it processes it's tick a few hundred milliseconds later generally.

I am using SharedArrayBuffers to share memory between the worker threads and the parent thread.

The frustrating thing is that the parent thread ACTUALLY HAS THE INTERESTING DATA IMMEDIATELY, it's in there, it just doesn't know it until the timer goes off and tells it to process the data in that SharedArrayBuffer.

I know I can send a message from a worker thread to a parent thread in a number of ways, but that takes about 0.75ms no matter how I do that. (Speed seems to be the same for JS postMessage, as well as a local Websocket client/server setup... which makes sense.)

Which is fine, but I've found a much faster way to communicate (that, at least) between threads and just hoping there may be others I have not thought of?

I can run a recursive loop of the same function called isDataInteresting() inside the parent thread, constantly calling itself with process.nextTick(() => {isDataInteresting();});, but that maxes out the CPU and I worry that it will slow down other important calculations that need to happen regularly in the parent thread.

I can switch it up calling setTimeout(function(){isDataInteresting();},0); every few thousand calls to isDataInteresting() to cut the CPU usage down to a constant 25% on the parent thread which is the route I am probably going to take but that just seems so kludgy.

I can't pass off the aggressive isDataInteresting() check to another worker thread as that creates the same problem I had in the first place... which is how to inform the parent thread of interesting data immediately!

Proxy objects are not allowed to be cloned/posted via javascript postMessages unfortunately either.

The isDataInteresting() check is the fastest by a large margin. With the parent thread being notified of interesting data in about 10,000 nanoseconds as opposed to 750,000 nanoseconds. So if the kludge is the only thing that works, I guess I'll take it.

Is there another way to check for updated data on a variable that I am not considering, I hope?

Thanks for reading!

randy
  • 156
  • 9
  • 1
    In my opinion you should change technology to java or c#. Accurate timing in javascript with precision under 1ms is not achievable as far I know. But when many threads share the same data (like in scenario which you describe) you should use [Atomic](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Atomics) to avoid conflicts. – Robert Jun 11 '21 at 21:14
  • 2
    Have you tried using [atomics](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Atomics) to "watch" the array for changes? You pretty much HAVE to use atomics anyway if you're going to be reading/writing to shared memory from different threads. As Robert said, a cooperative single-threaded model like nodejs is never going to be about precise and super fast timing. If the main thread just happens to be doing something else at the moment your data is updated by the other thread, even the fastest notification scheme possible isn't going to trigger right away. – jfriend00 Jun 11 '21 at 21:25
  • I am using atomics to create the mutex. How would I watch a SharedArrayBuffer for changes? Which method? – randy Jun 11 '21 at 21:39
  • Just tested await/notify and the communication takes about 400,000ns on average. Thanks for the tip! Had not thought of that! Still much slower than brute force checking though unfortunately :S – randy Jun 11 '21 at 22:21

0 Answers0