How to change a VideoOverlay's window handle after it has already been set?

Question

Background

I'm looking for a way to change the window that my video is being rendered into. This is necessary because there are some situations where the window can be destroyed, for example when my application switches into fullscreen mode.

Code

When the canvas is realized, the video source and sink are connected. Then when the prepare-window-handle message is emitted, I store a reference to the VideoOverlay element that sent it. Clicking the "switch canvas" button calls set_window_handle(new_handle) on this element, but the video continues to render in the original canvas.

import sys

import gi
gi.require_version('Gtk', '3.0')
gi.require_version('Gst', '1.0')
gi.require_version('GstVideo', '1.0')
from gi.repository import Gtk, Gst, GstVideo
Gst.init(None)


if sys.platform == 'win32':
    import ctypes

    PyCapsule_GetPointer = ctypes.pythonapi.PyCapsule_GetPointer
    
    PyCapsule_GetPointer.restype = ctypes.c_void_p
    PyCapsule_GetPointer.argtypes = [ctypes.py_object]

    gdkdll = ctypes.CDLL('libgdk-3-0.dll')
    gdkdll.gdk_win32_window_get_handle.argtypes = [ctypes.c_void_p]
    
    def get_window_handle(widget):
        window = widget.get_window()
        if not window.ensure_native():
            raise Exception('video playback requires a native window')
        
        window_gpointer = PyCapsule_GetPointer(window.__gpointer__, None)
        handle = gdkdll.gdk_win32_window_get_handle(window_gpointer)
        
        return handle
else:
    from gi.repository import GdkX11

    def get_window_handle(widget):
        return widget.get_window().get_xid()


class VideoPlayer:
    def __init__(self, canvas):
        self._canvas = canvas
        self._setup_pipeline()
    
    def _setup_pipeline(self):
        # The element with the set_window_handle function will be stored here
        self._video_overlay = None
        
        self._pipeline = Gst.ElementFactory.make('pipeline', 'pipeline')
        src = Gst.ElementFactory.make('videotestsrc', 'src')
        video_convert = Gst.ElementFactory.make('videoconvert', 'videoconvert')
        auto_video_sink = Gst.ElementFactory.make('autovideosink', 'autovideosink')

        self._pipeline.add(src)
        self._pipeline.add(video_convert)
        self._pipeline.add(auto_video_sink)
        
        # The source will be linked later, once the canvas has been realized
        video_convert.link(auto_video_sink)
        
        self._video_source_pad = src.get_static_pad('src')
        self._video_sink_pad = video_convert.get_static_pad('sink')
        
        self._setup_signal_handlers()
    
    def _setup_signal_handlers(self):
        self._canvas.connect('realize', self._on_canvas_realize)
        
        bus = self._pipeline.get_bus()
        bus.enable_sync_message_emission()
        bus.connect('sync-message::element', self._on_sync_element_message)
    
    def _on_sync_element_message(self, bus, message):
        if message.get_structure().get_name() == 'prepare-window-handle':
            self._video_overlay = message.src
            self._video_overlay.set_window_handle(self._canvas_window_handle)
    
    def _on_canvas_realize(self, canvas):
        self._canvas_window_handle = get_window_handle(canvas)
        self._video_source_pad.link(self._video_sink_pad)
        
    def start(self):
        self._pipeline.set_state(Gst.State.PLAYING)
    

window = Gtk.Window()
vbox = Gtk.Box(orientation=Gtk.Orientation.VERTICAL)
window.add(vbox)

canvas_box = Gtk.Box()
vbox.add(canvas_box)

canvas1 = Gtk.DrawingArea()
canvas1.set_size_request(400, 400)
canvas_box.add(canvas1)

canvas2 = Gtk.DrawingArea()
canvas2.set_size_request(400, 400)
canvas_box.add(canvas2)

player = VideoPlayer(canvas1)
canvas1.connect('realize', lambda *_: player.start())

def switch_canvas(btn):
    handle = get_window_handle(canvas2)
    print('Setting handle:', handle)
    player._video_overlay.set_window_handle(handle)

btn = Gtk.Button(label='switch canvas')
btn.connect('clicked', switch_canvas)
vbox.add(btn)

window.connect('destroy', Gtk.main_quit)
window.show_all()
Gtk.main()

Problem / Question

Calling set_window_handle() a 2nd time seems to have no effect - the video continues to render into the original window.

I've tried setting the pipeline into PAUSED, READY, and NULL state before calling set_window_handle(), but that didn't help.

I've also tried to replace the autovideosink with a new one as seen here, but that doesn't work either.

How can I change the window handle without disrupting the playback too much? Do I have to completely re-create the pipeline?

I'm testing on Windows, so I can't guarantee that the `get_window_handle` works on Mac/linux. Please let me know if there's something wrong with it and I'll try to fix it. — Aran-Fey, Dec 31 '21 at 09:12
Apparently the `set_window_handle` solution works with some sinks and not others. On my Windows PC it's a `GstD3D11VideoSink`, which seems to ignore the 2nd call to `set_window_handle`, but on a different (linux) PC this code works. I'd like to find a solution that works reliably with any sink. — Aran-Fey, Dec 31 '21 at 14:09
@CristiFati I don't know if it works on *every* linux PC, but an acquaintance tested it for me and it works on theirs. — Aran-Fey, Jan 05 '22 at 07:50

score 0 · Answer 1 · answered Dec 29 '21 at 09:14

0

Looking at the source code, it appears that at least GL-based implementations of VideoOverlay element update the window id on expose event.

So you could try calling:

player._video_overlay.expose()

to reinitialize the GL scene after the window handle has been changed.

If that does not work, you can create a new VideoOverlay element and add it dynamically without stopping the graph.

answered Dec 29 '21 at 09:14

jpa

10,351
1
28
45

Thanks for the answer. Unfortunately `.expose()` doesn't do anything for me; I guess it probably depends on the element whether that works or not. So it looks like I'll have to create a new autovideosink, but I haven't quite figured out how to unlink elements in a PLAYING pipeline yet. As far as I can tell, the tutorial you linked only shows how to link elements at runtime, not how to unlink them. Could you help me with this? I have already posted [a related question](https://stackoverflow.com/questions/70508580/output-selector-makes-video-freeze) yesterday, so you could write an answer there. – Aran-Fey Dec 29 '21 at 09:30
@Aran-Fey https://stackoverflow.com/questions/3074145/dynamically-unlink-elements-in-a-running-gstreamer-pipeline has something, but I haven't tried it myself. – jpa Dec 29 '21 at 09:41
I've already seen that question, but I still can't figure it out :( – Aran-Fey Dec 29 '21 at 09:55
Yeah, it gets difficult sometimes. In one application I used application sinks and passed around buffers between pipelines manually so that I could stop them separately. But that probably breaks up audio/video sync. – jpa Dec 29 '21 at 10:39
Adding the call to `expose()` made the video in the left canvas stop, and the video in the right canvas looks more correct, so almost there but not quite. This must be getting _somewhere_... – Sylvester Kruin Jan 05 '22 at 23:38

How to change a VideoOverlay's window handle after it has already been set?

Background

Code

Problem / Question

1 Answers1