0

I'm trying to create an in-browser background removal using tensorflow.js body-pix

While the online demo reaches ~10 frames per second on my computer, my code takes ~3 seconds for a single frame.

I mostly followed the example on the official github

Here is my code:

const remove_background = async (img) => {
    console.time("ML");
    console.time("loadModel");
    const net = await bodyPix.load({
        architecture: 'MobileNetV1',
        outputStride: 16,
        multiplier: 0.5,
        quantBytes: 2
      });
      console.timeEnd("loadModel");
      console.time("segmentPerson");
    const segmentation = await net.segmentPerson(img.imageData, {
        segmentationThreshold: 0.7,
        internalResolution: 0.25
    });
    console.timeEnd("segmentPerson");
    console.time("ApplyMask");
    for (var i=0;i<segmentation.data.length;i++) {
        if(segmentation.data[i] === 0) {
            img.imageData.data[i*4] = 255;
            img.imageData.data[i*4+1] = 0;
            img.imageData.data[i*4+2] = 0;
        }
    }
    console.timeEnd("ApplyMask");

    console.timeEnd("ML");

    return img;
}

And here are the times:

loadModel: 129.35498046875 ms
segmentPerson: 2755.817138671875 ms
ApplyMask: 4.910888671875 ms
ML: 2890.7060546875 ms

I started with a 1700x1700 Image with internal resolution of 0.1, and worked down to a 200x200 image with 0.25 resolution, with no significant improvement.

What am I doing wrong?

I'm using tfjs@1.2 & body-pix@2.0

LobsterMan
  • 1,158
  • 1
  • 11
  • 18
  • Could be the type of backend you are using run `import {getBackend} from '@tensorflow/tfjs-core';` and tell me what the result of `console.log(getBackend())` is. – yudhiesh May 21 '21 at 05:47
  • i'm running it in the browser `````` – LobsterMan May 22 '21 at 08:49
  • Just adding this comment for the benefit of anyone else who comes across this message, I believe yudiesh is referring to the tensorflow backend as discussed in their documentation here - https://www.tensorflow.org/js/guide/platform_environment Running it on CPU vs. GPU vs. WASM can have dramatic effects on performance. – PhysicalEd Oct 28 '21 at 19:14

1 Answers1

1

I have found in my implementation using bodypix that the first detection takes quite a bit longer than subsequent ones. I believe they do one-time initialization on the first call to segmentPerson. We are segmenting video. I found I could save about 500ms of startup time of the first detection just by passing a blank image to segmentPerson while the stream is being acquired before passing the video frame by frame when the stream has started.

PhysicalEd
  • 784
  • 7
  • 10