1

I'm developing a game that uses hand detection

I'm attempting use Web workers so that hand detection doesn't block the main thread as per mediapipe docs

I'm wondering if anyone is able to produce an example as to how this would be done.

the detectForVideo method takes two params, a reference to the dom element and a timestamp.

Since Web Workers can't access dom elements, I'm getting an error.

This is my code

const worker = new Worker('landmarker/worker');


class UIManager {
  constructor() {
    this.$video = document.getElementById("webcam");
  }

  bindEnableCam() {
//this method is called elsewhere, it attaches the `predictWebcam` method to run once the webcam is ready
        const handleStream = (stream) => {
          this.$video.srcObject = stream;
          this.$video.addEventListener(
            "loadeddata",
            this.predictWebcam.bind(this)
          );
        };

        navigator.mediaDevices
          .getUserMedia(constraints)
          .then((stream) => handleStream(stream))
  }

  predictWebcam() {
   
    const nowInMs = Date.now();
    worker.postMessage({vid: this.$video}) // this line of code throws the error
    window.requestAnimationFrame(this.predictWebcam.bind(this));
    
  }
  
}

Produces this error.

caught (in promise) DOMException: Failed to execute 'postMessage' on 'Worker': HTMLVideoElement object could not be cloned.
    at UIManager.predictWebcam

I also get an error if I try to pass the landmarker or detectForVideo method

UIManager.js?t=1683753166973:81 Uncaught (in promise) DOMException: Failed to execute 'postMessage' on 'Worker': () => o.wasmBinaryPath.toString() could not be cloned.
    at UIManager.predictWebcam

Lasty, I tried just sending over a string and running everything from the worker and setting the worker to {type: "module"}.

//in my worker.js

import handLandmarker from ".";
import { ui } from "../UIManager";

self.onmessage = (e) => {
 
 if(e.data === 'detect'){
  handLandmarker.detectForVideo(ui.$video, Date.now())
 }
}

However this now produces this error:

vision_bundle.js:1 Uncaught TypeError: Failed to execute 'importScripts' on 'WorkerGlobalScope': Module scripts don't support importScripts().
    at o (vision_bundle.js:1:466026)
    at Array.map (<anonymous>)
    at r (vision_bundle.js:1:466152)
    at createTaskRunner (vision_bundle.js:1:467669)
    at TaskRunner.createInstance (vision_bundle.js:1:467909)
    at VisionTaskRunner.createVisionInstance (vision_bundle.js:1:472322)
    at b.createFromOptions (vision_bundle.js:1:553534)
    at index.js:9:45
o

This one seems to be a problem with workers not fully supporting modules, however I can't think of how I'd get around that when I have to import the HandLandmarker and FilesetResolver from the vision package

SaintPreston
  • 103
  • 6
  • For the first part I can help: the `HTMLVideoElement` is not a transferable object, as in https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Transferable_objects and so you need to call something like `createImageBitmap` and transfer that. However I'm also hitting the importScripts error when trying to run inference in a worker, and it seems unsupported, for now. – David Baynard Aug 23 '23 at 01:18
  • I've now resolved this for a different task (FaceLandmarker), but it isn't pretty. It isn't in a state to post as a full answer, at this time. So in brief: vendor the code to create the task class, replace the call to `importScripts` with `await import(script).then(({ default: f }) => (self.ModuleFactory = f));`, and fix whatever follows. This works in chrome, at least, though I'm not sure it should, as I believe dynamic import _shouldn't_ work in workers. – David Baynard Aug 23 '23 at 22:03

0 Answers0