Use posenet from tensorflow.js in electron

Question

I am trying to use the posenet MobileNetV1 network in an electron app. I want to be able to read image from file system (it does not matter if it is a png or jpg), and run it through the network.

What have I done so far:

I am using the following modules:

import * as posenet from '@tensorflow-models/posenet';
var imageToBase64 = require('image-to-base64');
var toUint8Array = require('base64-to-uint8array')

And initializing the network with:

var net = posenet.load();

In order to read image I am converting it to base64 than to Uint8Array, than I am using them to create an object {data: bytes, width: width, height: height}, which is fitting in the definition of ImageData.

Everything is running but the results in percentages are very low:

{
  score: 0.002851587634615819,
  keypoints: [
    { score: 0.0007664567674510181, part: 'nose', position: [Object] },
    {
      score: 0.0010295170359313488,
      part: 'leftEye',
      position: [Object]
    },
    {
      score: 0.0006740405224263668,
      part: 'rightEye',
      position: [Object]
    },

Notice that in the future I intend to build this app so modules like Canvas are no good since it does not build well.

If someone could give me a working poc it would be great since I am working on that for a very long time.

Answer 1

electron has two separated contexts; one that can be considered as a server side context called the main context and the renderer context in which the browser and its scripts are called. Though the question is not precise enough, it is trying to execute posenet in the main context of electron which can be compared as if one is trying to run this code in nodejs

posenet in the main renderer

const data = Buffer.from(base64str, 'base64')
const t = tf.node.decodeImage(data)
const net = await posenet.load()
const poses = net.estimateMultiplePoses(t, {
      flipHorizontal: false,
      maxDetections: 2,
      scoreThreshold: 0.6,
      nmsRadius: 20})
  })
  // do whatever with the poses

posenet from a script executed by the browser

const im = new Image()
im.src = base64str
const net = await posenet.load()
im.onload = async() => {
 const poses = await net.estimateMultiplePoses(im, {
      flipHorizontal: false,
      maxDetections: 2,
      scoreThreshold: 0.6,
      nmsRadius: 20})
  })
  // do whatever with the poses
}

Answer 2

Even though you copied the structure, maybe PoseNet is checking the if the Object is of a certain class which it will not be unless you actually create ImageData object and then set the fields. That is my guess as to why it doesn't like it.

Have you tried:

let clamped = Uint8ClampedArray.from(someframeBuffer);
let imageData = new ImageData(clamped, width, height);

It seems PoseNet accepts ImageData|HTMLImageElement|HTMLCanvasElement|HTMLVideoElement objects that can be passed to its prediction function.