What exactly does the nextFrame() method in TensorFlow.js do?

Question

I'm experimenting with processing live video feeds using TensorFlow.js.

I'm using something like the following based off other examples I've seen:

while(true) {
  const results = await model.classify(videoElem);
  console.log(results);
  await tf.nextFrame();
}

I'm trying to understand exactly what tf.nextFrame() does.

I'm thinking that when I'm running model.classify(videoElem) it takes a single frame from the video stream and processes it with the model.

I imagine there are two main scenarios:

1. The video produces frames faster than JavaScript is classifying them.
2. JavaScript is processing frames faster than they are produced.

Is the tf.nextFrame() method something to handle scenario #2, so that a single frame is never processed twice?

The documentation describes it by saying:

Returns a promise that resolve when a requestAnimationFrame has completed.

This is simply a sugar method so that users can do the following: await tf.nextFrame();

I'm having trouble interpreting what that means. Can anyone confirm if what I described is what tf.nextFrame() does? If my interpretation is wrong than what exactly does tf.nextFrame() do?

Answer 1

Upon re-reading the definition provided by the documentation I realized that requestAnimationFrame was a browser API that I was not familiar with.

The window.requestAnimationFrame() method tells the browser that you wish to perform an animation and requests that the browser calls a specified function to update an animation before the next repaint. The method takes a callback as an argument to be invoked before the repaint.

It seems window.requestAnimationFrame() returns on the next available re-paint if I'm interpreting it right. So this should stop scenario #2 described in my question where a single frame is processed twice.

Answer 2

The requestAnimationFrame() function provides a hook into the javascript event loop that allows execution of a callback function right before the next render. This can be used as a performant way to update animations before they are drawn to the screen without introducing the risk of forced reflows.

However in this context, the most important detail of rAF is that after the callback function is executed control of the main thread goes to the browser rendering process before any other code executes. This mitigates blocking, which can cause the browser's UI to stutter or even freeze.

In the example, awaiting the call to tf.nextFrame() allows the UI to update each time before executing the next iteration of the loop. This probably does have the effect of preventing the given second scenario, assuming that allowing the browser to render its next frame updates the input videoElem. However, this technique can be useful to prevent blocking the UI during long running operations even when input doesn't depend on visual elements of the screen, although it's important to note that this potentially caps the loop performance to the refresh rate of the monitor of the host machine.