1

I am using Python 3.9 and Open-CV (cv2) to read frames from a video stream and save them as JPGs.

My program seems to run OK. It captures the video stream fine, obtains frames, and saves them as JPGs.

However, the frames it is obtaining from the stream are out-of-date - sometimes by several minutes. The clock in the video stream is running accurately, but the clock displays in the JPGs are all identical (to the second - but one or more minutes prior to the datetime in the program's "print()" output (and the saved JPG file time), and moving objects that were in view at the time they were saved are missing completely.

Strangely:

  1. The JPG images are not identical in size. They grow by 10K - 20K as the sequence progresses. Even though they look identical to the eye, they show significant difference when compared using CV2 - but no difference if compared using PIL (which is about 10 - 15 times slower for image comparisons).
  2. The camera can be configured to send a snapshot by email when it detects motion. These snapshots are up-to-date, and show moving objects that were in frame at the time (but no clock display). Enabling or disabling this facility has no effect on the out-of-date issue with JPGs extracted from the video stream. And, sadly, the snapshots are only about 60K, and too low resolution for our purposes (which is an AI application that needs images to be 600K or more).

The camera itself is ONVIF - and things like PTZ work nicely from Python code. Synology Surveillance Station works really well with it in every aspect. This model has reasonably good specs - zoom and good LPR anti-glare functionality. It is made in China - but I don't want to be 'a poor workman who blames his tools'.

Can anyone spot something in the program code that may be causing this?

Has anyone encountered this issue, and can suggest a work-around or different library / methodology?

(And if it is indeed an issue with this brand / model of camera, you are welcome to put in a plug for a mid-range LPR camera that works well for you in an application like this.)

Here is the current program code:

import datetime
from time import sleep

import cv2

goCapturedStream = None
# gcCameraLogin, gcCameraURL, & gcPhotoFolder are defined in the program, but omitted for simplicity / obfuscation.

def CaptureVideoStream():
    global goCapturedStream
    print(f"CaptureVideoStream({datetime.datetime.now()}):  Capturing video stream...")
    goCapturedStream = cv2.VideoCapture(f"rtsp://{gcCameraLogin}@{gcCameraURL}:554/stream0")
    if not goCapturedStream.isOpened():  print(f"Error:  Video Capture Stream was not opened.")
    return

def TakePhotoFromVideoStream(pcPhotoName):
    llResult = False ;  laFrame = None
    llResult, laFrame = goCapturedStream.read()
    print(f"TakePhotoFromVideoStream({datetime.datetime.now()}):  Result is {llResult},  Frame data type is {type(laFrame)}, Frame length is {len(laFrame)}")
    if not ".jpg" in pcPhotoName.lower():  pcPhotoName += ".jpg"
    lcFullPathName = f"{gcPhotoFolder}/{pcPhotoName}"
    cv2.imwrite(lcFullPathName, laFrame)

def ReleaseVideoStream():
    global goCapturedStream
    goCapturedStream.release()
    goCapturedStream = None

# Main Program:  Obtain sequence of JPG images from captured video stream
CaptureVideoStream()
for N in range(1,7):
    TakePhotoFromVideoStream(f"Test{N}.jpg")
    sleep(2)             # 2 seconds
ReleaseVideoStream()
Graeme
  • 43
  • 1
  • 8
  • 1
    There's likely some buffering involved. IMHO the assumption that if you read a frame, sleep for 2 seconds, and read another frame, the second one will be the most recent frame captured by the camera, is flawed. In fact, if you only want a frame once every 2 seconds, streaming might not be good choice. Does the camera provide means to retrieve individual images? – Dan Mašek Dec 05 '22 at 10:31
  • 1
    I'd say, if you want to stick with `VideoCapture` and a stream, then don't `sleep`. Read the frames as fast as you can to stay in sync with the camera, keep track of elapsed time, and once the desired amount of time passes, save the most recently read frame (discard the others). – Dan Mašek Dec 05 '22 at 10:39
  • No problem. As for the changing size of JPEGs... hard to say without seeing the data. Save a couple of frames that demonstrate this (but use PNG, since it's loss-less) and add them to your question. It might be an effect caused by (inter frame) compression of the stream. – Dan Mašek Dec 05 '22 at 11:08
  • 1
    Thanks for your suggestions, Dan. I will give these a go in the morning. (It is midnight here in Auckland, and time I turned in.) I have added some info about snapshots to the main question. The 2 seconds was for test / demo purposes. In our application, we try to capture images as fast as possible. But the AI takes about a second or so to do its bit. I'll try running repeated frame-reads in a different thread (if it doesn't slow the AI down too much). – Graeme Dec 05 '22 at 11:11
  • Yes, that suggestion was a good one. I changed the program to .read() frames continually in a loop (with a sleep of 0.1), and save an image on every 20th pass through the loop, The clock display seems to be about right. – Graeme Dec 06 '22 at 08:20
  • But changing them to PNGs doesn't seem to fix the issue of the percentage difference creeping up significantly with subsequent saved frames (when compared with the first frame). I will do some further experimentation with this tomorrow (in daylight). – Graeme Dec 06 '22 at 08:27
  • Reading frames rapidly from the stream caused some low level Python error messages: [h264 @ 0000028822776280] left block unavailable for requested intra mode [h264 @ 0000028822776280] error while decoding MB 0 61, bytestream 544009 [h264 @ 00000288227774c0] error while decoding MB 62 64, bytestream -31 [h264 @ 00000186e9ea3600] reference picture missing during reorder [h264 @ 00000186e9ea3600] Missing reference picture, default is 65617 [h264 @ 00000186e9ea3a80] mmco: unref short failure [h264 @ 00000186e9ea3a80] illegal short term buffer state detected – Graeme Dec 06 '22 at 08:31
  • But releasing and re-capturing the video stream allowed the program to carry on, and it seemed to settle down after a number of minutes (these errors then occurred less frequently). – Graeme Dec 06 '22 at 08:33
  • Those errors and warnings look like ffmpeg log messages (the library used as a backend for VideoCapture). So it's going to be either related to that, the camera, or the communication between them, I'd say. I don't think I can help with that, but a separate question focused just on this specific might be useful. Mention the exact model of the camera, if you post one. – Dan Mašek Dec 06 '22 at 11:18
  • Using PNGs wasn't meant to fix the issue, it was only so that you may upload few frames for others to inspect in a state that matches what you have right after capturing (and not further modified by another pass of lossy compression, which would happen if you used JPEG). – Dan Mašek Dec 06 '22 at 11:20

1 Answers1

1

Dan Masek's suggestions were very valuable.

The program (now enhanced significantly) saves up-to-date images correctly, when triggered by the camera's inbuilt motion detection (running in a separate thread and communicating through global variables).

The key tricks were:

  1. A much faster loop reading the frames (and discarding most of them). I reduced the sleep to 0.1 (and even further to 0.01), and saved relatively few frames to JPG files only when required
  2. Slowing down the frame rate on the camera (from 25 to 10 fps - even tried 5 at one point). This meant that the camera didn't get ahead of the software and send unpredictable frames.
Graeme
  • 43
  • 1
  • 8