Building a custom video widget by subclassing from standard Video widget

The basic example of HelloWidget proposes to start with extending DOMWidgetModel from ‘@jupyter-widgets/base’, but building any nontrivial widget from scratch is by no means an easy task. I wanted to build a tool for outlier analysis in my video datasets, and standard Video widget seemed to be a good fit for that. I needed 2 videos to run synchronously side by side, microdelays were not important at all, just a visual inspection on human perception level. After many trials and errors I managed to reuse the standard Video widget extending it with control points of interest, but even though it does the job it has ugly hacks, I don’t know how to remove, and internal error behavior not manifested as software failures. Could someone take a look at code below and comment how to improve it. I purposely left some console.log calls uncommented so one can inspect the output.

a) The problem is that by changing model’s state, for example setting syncTime (corresponds to currentTime property of html video element) starts seek callback, it generates “seeking/seeked” events triggering updateSyncTime handler. How can I suppress this double reaction?

The above described behavior is a reason behind having separate ‘paused’ and ‘playing’ properties to keep track whether video is playing. Ideally I could use just one, and have something similar to commented updatePlaying method.

b) There was no need for try/catch wrapping around this.model.save_changes(); in updateSyncTime method, but initially I tried to call this.touch(); for synchronization with backend model and it crashed. I believe it’s the result of wrong subclassing. Could you explain what’s wrong with that?

c) It doesn’t want to run in JupyterLab, saying “Javascript Error: require is not defined”, only in the Classic Notebook mode. Is there any easy fix?

from traitlets import Unicode, Float, Bool
from ipywidgets import Video, register

@register
class VideoE(Video):
    _view_name   = Unicode('VideoEView').tag(sync=True)
    _view_module = Unicode('video_enhanced').tag(sync=True)
    _view_module_version = Unicode('0.1.1').tag(sync=True)
    
    paused   = Bool(True, help="Sync point for pausing").tag(sync=True)
    playing  = Bool(False, help="Sync point for playing").tag(sync=True)
    syncTime = Float(0.0, help="Sync point for seek operation").tag(sync=True)
    playbackRate = Float(1.0, help="Sync point for changing playback rate").tag(sync=True)
    
    @classmethod
    def from_file(cls, filename, **kwargs):
        return super(VideoE, cls).from_file(filename, **kwargs)
%%javascript
require.undef('video_enhanced');

define('video_enhanced', ["@jupyter-widgets/controls"], function(widgets) {

    var VideoEView = widgets.VideoView.extend({
        
        events: {
            // update Model when View handled the event
            'pause'      : 'updatePaused',
            'play'       : 'updatePlaying',
            'ratechange' : 'updatePlaybackRate',
            'seeked'     : 'updateSyncTime',
        },
    
        initialize: function() {
            // propagate changes from Model -> View
            this.listenTo(this.model, 'change:paused', this.onPausedChanged); 
            this.listenTo(this.model, 'change:playing', this.onPlayingChanged); // play/pause
            this.listenTo(this.model, 'change:playbackRate', this.onPlaybackRateChanged); // set rate
            this.listenTo(this.model, 'change:syncTime', this.seek);
        },

        updatePaused: function() {
            // console.log( 'Entered "updatePaused" func...')
            this.model.set('paused', true, {silent: true}); 
            this.model.set('playing', false, {silent: true}); 
            this.model.save_changes();
        },
        
        updatePlaying: function() {
            // console.log( 'Entered "updatePlaying" func...')
            this.model.set('paused', false, {silent: true}); 
            this.model.set('playing', true, {silent: true}); 
            this.model.save_changes();
        },
        
        onPausedChanged: function() {
            // console.log( 'Reacting on "paused" changed...')
            this.el.pause();
            this.model.set('paused', true, {silent: true}); 
            this.model.set('playing', false, {silent: true}); 
            this.model.save_changes();
        },
            
        onPlayingChanged: function() {
            // console.log( 'Reacting on "playing" changed...')
            this.el.play();
            this.model.set('paused', false, {silent: true}); 
            this.model.set('playing', true, {silent: true}); 
            this.model.save_changes();
        },
        
        onPlaybackRateChanged: function() {
            // console.log( 'Reacting on "playbackRate" changed...')
            this.el.playbackRate = this.model.get('playbackRate'); 
        },
        
        updatePlaybackRate: function() {
            // console.log( 'Entered "updatePlaybackRate" func...')
            this.model.set('playbackRate', this.el.playbackRate, {silent: true}); 
            this.model.save_changes();
        },
        
//         updatePlaying: function() {
//             // console.log( 'Reacting on "playing" changed...')
//             this.model.set('playing', !this.model.get('playing'), {silent: true}); 
//             this.model.save_changes();
//         },
        
        seek: function() {                  
            this.el.currentTime = this.model.get('syncTime'); 
        },        
        
        updateSyncTime: function() {
            //make silent to prevent calling 'seek' function
            this.model.set('syncTime', this.el.currentTime, {silent: true}); 
            
            // sync with backend
            try {
                // this.touch();
                this.model.save_changes();
            } catch (error) {
                console.error(error);
            }
            console.log(`video.currentTime = ${this.model.get('syncTime')}`);
        },
    });

    return {
        VideoEView : VideoEView,
    }
});
from ipywidgets import VBox, HBox
from ipywidgets import link, Checkbox

class SyncManager():
    def __init__(self, video1, video2):
        self._links = []
        self._video1 = video1
        self._video2 = video2
        
        self.box = Checkbox(False, description='Synchronize videos')
        self.box.observe(self.handle_sync_changed, names=['value'])
        
    def handle_sync_changed(self, value):
        if value.new:
            for prop in ['paused', 'playing', 'playbackRate', 'syncTime']:
                l = link((self._video1, prop), (self._video2, prop))
                self._links.append(l)
        else:
            for _link in self._links:
                _link.unlink()
            self._links.clear()

# ipywidgets.Video has no option to control muting

video1 = VideoE.from_file("images/Big.Buck.Bunny.mp4")
video1.autoplay = False
video1.loop = False
video1.width = 480

video2 = VideoE.from_file("images/Big.Buck.Bunny.mp4")
video2.autoplay = False
video2.loop = False
video2.width = 480

sync_m = SyncManager(video1, video2)

VBox([sync_m.box, HBox([video1, video2])])

It seems that what I’ve considered a wrong cycling behavior actually is the expected Model/View interaction. So I figured out how to change the code, and now it doesn’t look that ugly. Sorry to those who followed the question I changed properties names to make them shorter.

from traitlets import Unicode, Float, Bool
from ipywidgets import Video, register

@register
class VideoE(Video):
    _view_name   = Unicode('VideoEView').tag(sync=True)
    _view_module = Unicode('video_enhanced').tag(sync=True)
    _view_module_version = Unicode('0.1.1').tag(sync=True)
    
    playing = Bool(False, help="Sync point for play/pause operations").tag(sync=True)
    rate    = Float(1.0, help="Sync point for changing playback rate").tag(sync=True)
    time    = Float(0.0, help="Sync point for seek operation").tag(sync=True)
    
    @classmethod
    def from_file(cls, filename, **kwargs):
        return super(VideoE, cls).from_file(filename, **kwargs)
%%javascript
require.undef('video_enhanced');

define('video_enhanced', ["@jupyter-widgets/controls"], function(widgets) {

    var VideoEView = widgets.VideoView.extend({
        
        events: {
            // update Model when event is generated on View side
            'pause'      : 'onPause',
            'play'       : 'onPlay',
            'ratechange' : 'onRatechange',
            'seeked'     : 'onSeeked',
        },
    
        initialize: function() {
            // propagate changes from Model to View
            this.listenTo(this.model, 'change:playing', this.onPlayingChanged); // play/pause
            this.listenTo(this.model, 'change:rate',    this.onRateChanged);    // playbackRate
            this.listenTo(this.model, 'change:time',    this.onTimeChanged);    // currentTime
        },
        
        // View -> Model
        onPause: function() {
            this.model.set('playing', false, {silent: true}); 
            this.model.save_changes();
        },
        // View -> Model
        onPlay: function() {
            this.model.set('playing', true, {silent: true}); 
            this.model.save_changes();
        },
        // Model -> View    
        onPlayingChanged: function() {
            if (this.model.get('playing')) {
                this.el.play();
            } else {
                this.el.pause();
            }
        },
        // View -> Model
        onRatechange: function() {
            this.model.set('rate', this.el.playbackRate, {silent: true}); 
            this.model.save_changes();
        },
        // Model -> View
        onRateChanged: function() {
            this.el.playbackRate = this.model.get('rate'); 
        },
        // View -> Model        
        onSeeked: function() {
            this.model.set('time', this.el.currentTime, {silent: true}); 
            this.model.save_changes();
        },
        // Model -> View
        onTimeChanged: function() {                  
            this.el.currentTime = this.model.get('time'); 
        },
    });

    return {
        VideoEView : VideoEView,
    }
});
from ipywidgets import link, Checkbox

class SyncManager():
    '''
    Syncing videos needs an explicit setting of "time" property or clicking on the progress bar, 
    since the property is not updated continuously, but on "seeked" event generated
    '''
    def __init__(self, video1, video2):
        self._links = []
        self._video1 = video1
        self._video2 = video2
        
        self.box = Checkbox(False, description='Synchronize videos')
        self.box.observe(self.handle_sync_changed, names=['value'])
        
    def handle_sync_changed(self, value):
        if value.new:
            for prop in ['playing', 'rate', 'time']:
                l = link((self._video1, prop), (self._video2, prop))
                self._links.append(l)
        else:
            for _link in self._links:
                _link.unlink()
            self._links.clear()
from ipywidgets import VBox, HBox

video1 = VideoE.from_file("images/Big.Buck.Bunny.mp4")
video1.autoplay = False
video1.loop = False
video1.width = 480

video2 = VideoE.from_file("images/Big.Buck.Bunny.mp4")
video2.autoplay = False
video2.loop = False
video2.width = 480

sync_m = SyncManager(video1, video2)

VBox([sync_m.box, HBox([video1, video2])])

The only questions left how to make it running in JupyterLab and why this.touch() doesn’t make its job? :slight_smile:

For those interested in this topic, there is a related discussion on Stackoverflow and so you may want to also look there.

1 Like

Thank you. It’s actually my question. I didn’t mention that I started with Stackowerflow, and when stuck for many nights I had to deep dive into details :slight_smile:

Surprisingly, the next code snippet fixes warnings in VS Code and widget is rendered correctly. But it’s not enough for JupyterLab, it looks like, the widget should be packaged as a separate widget package.

from IPython.display import display, HTML
requirejs = '<script src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js" integrity="sha512-c3Nl8+7g4LMSTdrm621y7kf9v3SDPnhxLNhcjFJbKECVnmZHTdo+IRO05sNLTH/D3vA6u1X32ehoLC7WFVdheg==" crossorigin="anonymous"></script>'
display(HTML(requirejs))
1 Like