ffmpeg: What is the best practice to keep a live connection/socket with a camera, and save time on ffprobe

Question

Today... I used the following command: with subprocess.PIPE and subprocess.Popen in python 3:

ffmpeg -i udp://{address_of_camera} \
  -vf select='if(eq(pict_type,I),st(1,t),gt(t,ld(1)))' setpts=N/FRAME_RATE/TB \
  -f rawvideo -an -vframes {NUM_WANTED_FRAMES} pipe:`

This command helps me to capture NUM_WANTED_FRAMES frames from a live camera at a given moment.

However... it takes me about 4 seconds to read the frames, and about 2.5 seconds to open a socket between my computer and the camera's computer.

Is there a way, to have a socket/connection always open between my computer and the camera's computer, to save the 2.5 seconds?

I read something about fifo_size and overrun_fatal. I thought that maybe I can set fifo_size to be equal to NUM_WANTED_FRAMES, and overrun_fatal to True? Will this solve my problem? Or is there a different and simpler/better solution?

Should I try to record always (no -vframes flag) store the frames in a queue(With max size), and upon a wish to slice the video, read from my queue buffer? Will it work well with the keyframe?

Also... What to do when ffmpeg fails? restart the ffmpeg command?

score 1 · Answer 1 · answered May 15 '22 at 01:16

FFmpeg itself is an one-n-done type of app. So, to keep the camera running, the best option is to "record always (no -vframes flag)" and handle whether to drop/record frames in Python.

So, a rough sketch of the idea:

import subprocess as sp
from threading import Thread, Event
from queue import Queue

NUM_WANTED_FRAMES = 4 # whatever it is

width = 1920
height = 1080
ncomp = 3 # rgb

framesize = width*height*ncomp # in bytes
nbytes = framesize * NUM_WANTED_FRAMES 

proc = Popen(<ffmpeg command>, stdout=sp.PIPE)
stdout = proc.stdout

buffer = Queue(NUM_WANTED_FRAMES)
req_frame = Event() # set to record, default to drop

def reader():
    while True:
        if req_frame.is_set():
            queue.put(stdout.read(nbytes))
            record_frame.clear()
        else:
            # frames not requested, drop 
            stdout.read(framesize)
    
rd_thread = threading.Thread(target=reader)
rd_thread.start()

...

# elsewhere in your program, do this when you need to get the camera data
req_frame.set()
framedata = queue.get()

....

Will it work well with the keyframe?

Yes, if your FFmpeg command has -discard nokey it'll read just keyframes.

What to do when ffmpeg fails? restart the ffmpeg command?

Have another thread to monitor the health of proc (Popen object) and if it is dead, you need to restart subprocess with the same command and overwrite with the new stdout. You probably want to protect your code with try-except blocks as well. Also, adding timeouts to queue ops would be a good idea, too.

Hello @kesh First, let me say thanks!! :) Your suggestion works perfectly! We count the time boost as a big victory!. Now, it takes exactly the number of: NUM_WANTED_FRAMES divided by FPS How would you suggest: - To monitor the subprocess? What type of exception does it throw, when it exits unexpectedly? - What does stdout.read() return/throw when it fails? None or some specific exception type? - I used bool should_record_frames instead of req_frame. Is it thread safe, and a good solution? Or should I use the Event you suggested? - Clearing state/restarting upon crash? — Jeff Strongman, May 27 '22 at 10:04
`Popen` has `wait()` function which you can call in your monitor thread to wait for the ffmpeg to be done, then check its `returncode` (non-zero = abnormal termination). As for `bool` vs `Event`, let's just say that I follow language-agnostic textbooks when it comes to multithreading but the GIL lets you get by much easier in Python. — kesh, May 27 '22 at 13:46
Hello @kesh, I liked your idea with returncode. I learned that while True could be replaced with while proc.poll() is not None. While this lets us know when proc finished running, there are additional cases to handle: 1. When .read() returns None rather than a frame , 2. When read() is in a deadlock… because stdin (the ffmpeg unexpectedly takes too long or failed due to an odd case). Is there timeout field? 3. Should I use asyncio.subprocess rather than a thread? 4. Should I use ping to the udp address, to see it’s ok . 5. How to deal with buffer overflow? — Jeff Strongman, May 28 '22 at 12:01

ffmpeg: What is the best practice to keep a live connection/socket with a camera, and save time on ffprobe

1 Answers1