ffmpeg piped to python and displayed with cv2.imshow slides rightward and changes colors

Question

Code:

import cv2
import time
import subprocess
import numpy as np

w,h = 1920, 1080
fps = 15

def ffmpegGrab():
    """Generator to read frames from ffmpeg subprocess"""
    cmd = f'.\\Resources\\ffmpeg.exe -f gdigrab -framerate {fps} -offset_x 0 -offset_y 0 -video_size {w}x{h} -i desktop -pix_fmt bgr24 -vcodec rawvideo -an -sn -f image2pipe -' 

    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
    while True:
        raw_frame = proc.stdout.read(w*h*3)
        frame = np.fromstring(raw_frame, np.uint8)
        frame = frame.reshape((h, w, 3))
        yield frame

# Get frame generator
gen = ffmpegGrab()

# Get start time
start = time.time()

# Read video frames from ffmpeg in loop
nFrames = 0
while True:
    # Read next frame from ffmpeg
    frame = next(gen)
    nFrames += 1

    frame = cv2.resize(frame, (w // 4, h // 4))

    cv2.imshow('screenshot', frame)

    if cv2.waitKey(1) == ord("q"):
        break

    fps = nFrames/(time.time()-start)
    print(f'FPS: {fps}')


cv2.destroyAllWindows()

The code does display the desktop capture however the color format seems to switch and the video scrolls rightward as if it is repeated. Am I going about this in the correct way?

I eould use `np.frombuffer()` rather than `np.fromstring()` and I would avoid `shell=True` where possible. — Mark Setchell, Sep 09 '21 at 22:10
Try saving the frames as PNGs immediately after `frame = next(gen)` using `cv2.imwrite(f'frame-{nFrames}.png', frame)` and inspecting the PNGs. — Mark Setchell, Sep 09 '21 at 23:20
This is for a realtime application - wouldn't saving this to a file cause delay? I'm aiming for 30-60fps. I'm also looking for why this issue occurs as it seems there was only one other post related to this where the suggestion was to open a pipe as binary which I already tried. — Michael, Sep 09 '21 at 23:51
The point of saving a PNG is to debug/check to find the problem. We can speed it up later. I think the code originates from mine... https://stackoverflow.com/a/60266466/2836621 — Mark Setchell, Sep 09 '21 at 23:56
Images show the same deal, video comes up as corrupted file/fileextension. — Michael, Sep 10 '21 at 02:59

score 2 · Accepted Answer · answered Sep 10 '21 at 09:20

The cause for the problem is: stderr=subprocess.STDOUT.

The argument stderr=subprocess.STDOUT redirects stderr to stdout.
stdout is used as a PIPE for reading the output video from FFmpeg sub-process.
FFmpeg writes some text to stderr and the text is "mixed" with the raw video (due to the redirection). The "mixing" causes weird slides and colors changes.

Replace proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True) with:

proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)

Minor corrections:

As Mark Setchell commended, use np.frombuffer() instead of np.fromstring() and avoid shell=True.

Replace -f image2pipe with -f rawvideo.
The output format is raw video and not an image (the code is working with image2pipe, but rawvideo is more correct).

Complete updated code:

import cv2
import time
import subprocess
import numpy as np

w,h = 1920, 1080
fps = 15

def ffmpegGrab():
    """Generator to read frames from ffmpeg subprocess"""
    # Use "-f rawvideo" instead of "-f image2pipe" (command is working with image2pipe, but rawvideo is the correct format).
    cmd = f'.\\Resources\\ffmpeg.exe -f gdigrab -framerate {fps} -offset_x 0 -offset_y 0 -video_size {w}x{h} -i desktop -pix_fmt bgr24 -vcodec rawvideo -an -sn -f rawvideo -'

    #proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)

    # Don't use stderr=subprocess.STDOUT, and don't use shell=True
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)

    while True:
        raw_frame = proc.stdout.read(w*h*3)
        frame = np.frombuffer(raw_frame, np.uint8)  # Use frombuffer instead of fromarray
        frame = frame.reshape((h, w, 3))
        yield frame

# Get frame generator
gen = ffmpegGrab()

# Get start time
start = time.time()

# Read video frames from ffmpeg in loop
nFrames = 0
while True:
    # Read next frame from ffmpeg
    frame = next(gen)
    nFrames += 1

    frame = cv2.resize(frame, (w // 4, h // 4))

    cv2.imshow('screenshot', frame)

    if cv2.waitKey(1) == ord("q"):
        break

    fps = nFrames/(time.time()-start)
    print(f'FPS: {fps}')


cv2.destroyAllWindows()

ffmpeg piped to python and displayed with cv2.imshow slides rightward and changes colors

1 Answers1