Interacting programmatically with ipywidgets Video widget in Jupyter notebook

Question

I was trying to build a Jupyter notebook solution for outlier analysis of video dataset. I wanted to use Video widget for that purpose, but I didn't find in the documentation how to get a current video frame and/or scroll to needed position by calling some widget's method. My problem is very similar (virtually the same) to these unanswered questions one and two.

I managed to implement the idea by saving the video frames to numpy array and employing matplotlib's imshow function, but video playing is very jittering. I used blitting technique to get some extra fps, but it didn't help much, and in comparison, Video widget produces a smoother experience. It looks like Video widget is essentially a wrapper for browser's built-in video player.

Question: How can I get control of widget's playing programmatically so I can synchronize multiple widgets? Could some %%javascript magic help with a IPython.display interaction?

Here below is a Python code snippet just for illustration purposes, to give you an idea of what I want to achieve.

%matplotlib widget

from videoreader import VideoReader # nice pithonic wrapper for video reading with opencv
import numpy as np
import matplotlib.pyplot as plt
from ipywidgets import IntSlider, Play, link, HBox, VBox 

# prepare buffered video frames
vr = VideoReader('Big.Buck.Bunny.mp4')

fps = vr.frame_rate
frames = []
for frame in vr[0:300:1]:
    frames.append(frame[:,:,::-1]) # brg2rgb 
del vr

vi_buff = np.stack(frames, axis=0) # dimensions (T, H, W, C)  

# simulate random signal for demonstration purposes
t = np.linspace(0.0, vi_buff.shape[0], num=vi_buff.shape[0]*10)
s = np.sin(2*np.pi*t)*np.random.randn(vi_buff.shape[0]*10)

plt.ioff()
fig = plt.figure(figsize=(11, 8))
ax1 = plt.subplot2grid((6, 6), (0, 0), rowspan=2, colspan=3)
ax2 = plt.subplot2grid((6, 6), (0, 3), colspan=3)
ax3 = plt.subplot2grid((6, 6), (1, 3), colspan=3)
plt.ion()

# initial plots 
img = ax1.imshow(vi_buff[0,...])
l0 = ax2.plot(t, s)
l1 = ax3.plot(t, -s)

# initial scene
lo_y, hi_y = ax2.get_ybound()
ax2.set_xbound(lower=-12., upper=2.)
ax3.set_xbound(lower=-12., upper=2.)

def update_plot(change):
    val = change.new
    img.set_data(vi_buff[val,...])
    ax2.axis([val - 12, val + 2, lo_y, hi_y])
    ax3.axis([val - 12, val + 2, lo_y, hi_y])
    fig.canvas.draw_idle()

player = Play(
    value=0, #intial frame index
    min=0,
    max=vi_buff.shape[0]-1,
    step=1,
    interval=int(1/round(fps)*1000) #referesh interval in ms
)
fr_slider = IntSlider(
    value=0,
    min=0,
    max=vi_buff.shape[0]-1
)
fr_slider.observe(update_plot, names='value')

link((player,"value"), (fr_slider,"value"))

VBox([HBox([player, fr_slider]), fig.canvas])

Answer 1

I had to learn it the hard way and write my own custom Video widget. Thanks to the authors of ipywidgets module, it was not from the scratch. I also found out that, what I wanted to do, possibly could be done in the easier way with PyViz Panel Video. I am sharing my solution if anyone interested and it could save you the time learning the front-end and Backbone.js.

Caution: it won't run in JupyterLab, you need to switch to Jupyter Classic Mode. Please share a solution in the comments, if you know how to fix: “Javascript Error: require is not defined”

Widget Model on the Python side

from traitlets import Unicode, Float, Bool
from ipywidgets import Video, register

@register
class VideoE(Video):
    _view_name   = Unicode('VideoEView').tag(sync=True)
    _view_module = Unicode('video_enhanced').tag(sync=True)
    _view_module_version = Unicode('0.1.1').tag(sync=True)
    
    playing = Bool(False, help="Sync point for play/pause operations").tag(sync=True)
    rate    = Float(1.0, help="Sync point for changing playback rate").tag(sync=True)
    time    = Float(0.0, help="Sync point for seek operation").tag(sync=True)
    
    @classmethod
    def from_file(cls, filename, **kwargs):
        return super(VideoE, cls).from_file(filename, **kwargs)

Widget View on the front-end

%%javascript
require.undef('video_enhanced');

define('video_enhanced', ["@jupyter-widgets/controls"], function(widgets) {

    var VideoEView = widgets.VideoView.extend({
        
        events: {
            // update Model when event is generated on View side
            'pause'      : 'onPause',
            'play'       : 'onPlay',
            'ratechange' : 'onRatechange',
            'seeked'     : 'onSeeked',
        },
    
        initialize: function() {
            // propagate changes from Model to View
            this.listenTo(this.model, 'change:playing', this.onPlayingChanged); // play/pause
            this.listenTo(this.model, 'change:rate',    this.onRateChanged);    // playbackRate
            this.listenTo(this.model, 'change:time',    this.onTimeChanged);    // currentTime
        },
        
        // View -> Model
        onPause: function() {
            this.model.set('playing', false, {silent: true}); 
            this.model.save_changes();
        },
        // View -> Model
        onPlay: function() {
            this.model.set('playing', true, {silent: true}); 
            this.model.save_changes();
        },
        // Model -> View    
        onPlayingChanged: function() {
            if (this.model.get('playing')) {
                this.el.play();
            } else {
                this.el.pause();
            }
        },
        // View -> Model
        onRatechange: function() {
            this.model.set('rate', this.el.playbackRate, {silent: true}); 
            this.model.save_changes();
        },
        // Model -> View
        onRateChanged: function() {
            this.el.playbackRate = this.model.get('rate'); 
        },
        // View -> Model        
        onSeeked: function() {
            this.model.set('time', this.el.currentTime, {silent: true}); 
            this.model.save_changes();
        },
        // Model -> View
        onTimeChanged: function() {                  
            this.el.currentTime = this.model.get('time'); 
        },
    });

    return {
        VideoEView : VideoEView,
    }
});

And the actual job I needed to get done

from ipywidgets import link, Checkbox, VBox, HBox

class SyncManager():
    '''
    Syncing videos needs an explicit setting of "time" property or clicking on the progress bar, 
    since the property is not updated continuously, but on "seeked" event generated
    '''
    def __init__(self, video1, video2):
        self._links = []
        self._video1 = video1
        self._video2 = video2
        
        self.box = Checkbox(False, description='Synchronize videos')
        self.box.observe(self.handle_sync_changed, names=['value'])
        
    def handle_sync_changed(self, value):
        if value.new:
            for prop in ['playing', 'rate', 'time']:
                l = link((self._video1, prop), (self._video2, prop))
                self._links.append(l)
        else:
            for _link in self._links:
                _link.unlink()
            self._links.clear()


video1 = VideoE.from_file("images/Big.Buck.Bunny.mp4")
video1.autoplay = False
video1.loop = False
video1.width = 480

video2 = VideoE.from_file("images/Big.Buck.Bunny.mp4")
video2.autoplay = False
video2.loop = False
video2.width = 480

sync_m = SyncManager(video1, video2)

VBox([sync_m.box, HBox([video1, video2])])