How can I get continuous landmarks from video for Mediapipe Handpose

Question

I am new to Javascript. I am trying to get output from MediaPipe Handpose. When I give an image into this model, I easily get an output. But when I try for a video, it doesn't work. This is the head

<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-core"></script>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-converter"></script>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-backend-webgl"></script>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/handpose"></script>

My video ID details

<a href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/video">Source</a><br>
<video id="video" src="o_new.mp4" width="640" height="480" controls> 
</video>
<canvas id="canvas" style="overflow:auto"></canvas>

Inside the script

<script>
     const video = document.getElementById("video");   

     
    async function main() {
  // Load the MediaPipe handpose model.
  const model = await handpose.load(maxContinuousChecks = 60);
        console.log('Model Loaded')

  // Pass in a video stream (or an image, canvas, or 3D tensor) to obtain a
  // hand prediction from the MediaPipe graph.
  const predictions = await model.estimateHands(video);
        console.log('Estimated Hand')
        console.log(predictions)
  if (predictions.length > 0) {
    /*
    `predictions` is an array of objects describing each detected hand, for example:
    [
      {
        handInViewConfidence: 1, // The probability of a hand being present.
        boundingBox: { // The bounding box surrounding the hand.
          topLeft: [162.91, -17.42],
          bottomRight: [548.56, 368.23],
        },
        landmarks: [ // The 3D coordinates of each hand landmark.
          [472.52, 298.59, 0.00],
          [412.80, 315.64, -6.18],
          ...
        ],
        annotations: { // Semantic groupings of the `landmarks` coordinates.
          thumb: [
            [412.80, 315.64, -6.18]
            [350.02, 298.38, -7.14],
            ...
          ],
          ...
        }
      }
    ]
    */

    for (let i = 0; i < predictions.length; i++) {
      const keypoints = predictions[i].landmarks;
        console.log('keypoints Loop')

      // Log hand keypoints.
      for (let i = 0; i < keypoints.length; i++) {
        const [x, y, z] = keypoints[i];
        console.log(`Keypoint ${i}: [${x}, ${y}, ${z}]`);
          
      }
    }
  }
}
    
 
main();
</script>

How can I get continuous landmarks in an output object for video?

Here is the error.

What is the error/output you are getting? – Divya Kaushik Jul 30 '20 at 09:43 — Divya Kaushik, Jul 30 '20 at 09:43
@DivyaKaushik https://i.stack.imgur.com/nJ06H.png – Kanchon Kanti Podder Jul 31 '20 at 12:16 — Kanchon Kanti Podder, Jul 31 '20 at 12:16

NoxAnura · Accepted Answer · 2020-08-12T23:02:18.393

I am updating my previous answer. You don't want to use set interval like I put in my former solution. When I used it for more than a few minutes it filled up my gpu memory and crashed webgl. I was able to comb through the developers demo.js file, and found a solution. In your js file replace your main() function with the below code:



const state = {
  backend: 'webgl'
};


let model;


async function main() {
  await tf.setBackend(state.backend);
  model = await handpose.load();

  landmarksRealTime(video);
}

const landmarksRealTime = async (video) => {
  async function frameLandmarks() {
    const predictions = await model.estimateHands(video);
    
    if (predictions.length > 0) {
      const result = predictions[0].landmarks;
      console.log(result, predictions[0].annotations);

    }
    rafID = requestAnimationFrame(frameLandmarks);
  };

  frameLandmarks();
};


video.addEventListener("loadeddata", main);

This console logs continuous landmarks. It doesn't log anything if a hand is not detected. Also, it looks like the developers updated the recommended script tags a few days ago, so I would recommend updating your index.html file. They should be:

<!-- Require the peer dependencies of handpose. -->
<script src="https://unpkg.com/@tensorflow/tfjs-core@2.1.0/dist/tf-core.js"></script>
<script src="https://unpkg.com/@tensorflow/tfjs-converter@2.1.0/dist/tf-converter.js"></script>

<!-- You must explicitly require a TF.js backend if you're not using the tfs union bundle. -->
<script src="https://unpkg.com/@tensorflow/tfjs-backend-webgl@2.1.0/dist/tf-backend-webgl.js"></script>
<!-- Alternatively you can use the WASM backend: <script src="https://unpkg.com/@tensorflow/tfjs-backend-wasm@2.1.0/dist/tf-backend-wasm.js"></script> -->

<script src="https://unpkg.com/@tensorflow-models/handpose@0.0.6/dist/handpose.js"></script></head>

How can I get continuous landmarks from video for Mediapipe Handpose

1 Answers1