1

I am trying to store video file from multiple sources (RGB, depth, and infrared) from kinect sensors.

This is the image that I visualized using cv2.imshow command using the following code:

cv2.imshow("ir", ir / 65535.)
cv2.imshow("depth", depth / 4500.)
cv2.imshow("color", color)

ir, depth both are array with size of (height, width), float32. color is a array with size of (height, width, 3), where 3 is the RGB channel and uint8 type from 0-255. Since ir and depth's value is large, we need to normalize them using the code above. And this code gave the above figures.

Now I want to store a series of image array as a video file. I use the following code:

ir_video = cv2.VideoWriter('ir.mp4', cv2.VideoWriter_fourcc(*'MP42'), fps, (height, width), False)
depth_video = cv2.VideoWriter('depth.mp4', cv2.VideoWriter_fourcc(*'MP42'), fps, (height, width), False)
color_video = cv2.VideoWriter('color.mp4', cv2.VideoWriter_fourcc(*'MP42'), fps, (height, width), True)

for loop: (pseudo for loop for this part, basically write every frame into the video)
    ir_video.write(ir / 65535.)
    depth_video.write(depth / 4500.)
    color_video.write(color)

ir_video.release()
depth_video.release()
color_video.release()

Color video works very well, looks very similar to the cv2.imshow command. However, ir and depth video are corrupted. All 0kb. I tried to change the fourcc code to cv2.VideoWriter_fourcc(*'mp4v'). This time the ir one saved a video that I can play. But it is very different from the cv2.imshow result. It is shown here.

I'm wondering how I can correct save the result as I viewed in cv2.imshow command. What fourcc code should be used? Thanks a lot!

Daniel Konečný
  • 3
  • 7
kamicj
  • 11
  • 1
  • 3
  • 1
    Read imshow and videowriter docs. Afair imshow expects floating points to be black <= 0 and white >= 1, while VideoWriter interpretes floats as black <= 0 and white >= 255.0 – Micka Apr 20 '22 at 19:55
  • You may convert to `uint8` in range [0, 255] before saving: Example instead of writing: `depth / 4500.`, write: `(depth * (255.0/4500.0)).clip(0, 255).astype(np.uint8)` – Rotem Apr 20 '22 at 21:19
  • Thanks you@Micka for explaining. – kamicj Apr 21 '22 at 22:43
  • Thanks a lot@Rotem. Your solution is perfect yet simple! Works now! – kamicj Apr 21 '22 at 22:44
  • @Rotem I'm wondering after saving the video, how can I revert it back to 'depth' value? Now when I read the saved depth video using the command, it return me a (height, width,3) array. How do I revert it back to the (height, width) float array? The calculation part is clear I just need to do the math inversion of '(depth * (255.0/4500.0)).clip(0, 255).astype(np.uint8)', but how about from 3 channels to one? Thanks a lot! – kamicj May 25 '22 at 22:11
  • ok. Figure it out. data = double(rgb2gray(frame))/(255/4500); Convert it back to double first is very important. – kamicj May 25 '22 at 22:36

3 Answers3

0

I worked on a similar project using other depth cameras (Orbbec, Asus Xtion) and afaik videowriter class of OpenCV does not support 16-bit depth images, that's why as suggested in the comments you should convert to 8 bit. You can take a look here for what I was using to save such a video (It's about using OpenNI2 but the main concept is there).

rok
  • 2,574
  • 3
  • 23
  • 44
0

Starting with OpenCV 4.7.0 it is possible to write 16-bit depth videos, see the pull request which added support for it.

For VideoWriter you have to:

  • specify CAP_FFMPEG because this only seems to be supported for FFmpeg at the moment
  • use the FFV1 codec
  • specify {VIDEOWRITER_PROP_DEPTH, CV_16U, VIDEOWRITER_PROP_IS_COLOR, false} as params

For VideoCapture (reading) you have to:

  • specify CAP_FFMPEG
  • specify {CAP_PROP_CONVERT_RGB, false} as params
    Note that this will print a warning like "VIDEOIO/FFMPEG: BGR conversion turned OFF ..." to the console.

There appear to be some limitations with this though, see the description of the pull request.

That pull request also added a unit test which contains a VideoWriter & VideoCapture round trip.

Marcono1234
  • 5,856
  • 1
  • 25
  • 43
0

Solution provided in the comments to the question is sufficient, however, other answers developed this question to a one that focuses also on how to save video with higher bit depth. That is an interesting topic that is not yet discussed anywhere I've seen, so I'd like to continue on that. Author of this question could also benefit from this since uint16 video format preserves more information of the original float32 (which is also the reason why IR and point cloud data are not saved in uint8 with Kinect).

@Marcono1234 provided an idea how to save video with OpenCV VideoWriter (since version 4.7.0) but didn't provide the Python code. I find it not trivial to program it correctly and it took me some time, so I'd like to share my code.

This is a fully working example of reading an image from your webcam, converting it to monochrome 16-bit depth and saving it as such. Run to record the video and use keyboard letter q to stop recording. The most important part is obviously the VideoWriter definition.

import cv2
import numpy as np

video_capture = cv2.VideoCapture(0)

if not video_capture.isOpened():
    print("Error reading video file")

video_width = int(video_capture.get(cv2.CAP_PROP_FRAME_WIDTH))
video_height = int(video_capture.get(cv2.CAP_PROP_FRAME_HEIGHT))

video_writer = cv2.VideoWriter(
    filename="video.mkv",
    apiPreference=cv2.CAP_FFMPEG,
    fourcc=cv2.VideoWriter_fourcc(*"FFV1"),
    fps=10.0,
    frameSize=(video_width, video_height),
    params=[
        cv2.VIDEOWRITER_PROP_DEPTH,
        cv2.CV_16U,
        cv2.VIDEOWRITER_PROP_IS_COLOR,
        0,  # false
    ],
)

while True:
    ret, frame = video_capture.read()
    if ret:
        # Convert the webcam image to mono 16-bit.
        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        frame = frame.astype(np.uint16)
        frame *= 2 ** 8

        video_writer.write(frame)

        cv2.imshow('Frame', frame)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    else:
        break

video_capture.release()
video_writer.release()
cv2.destroyAllWindows()

To make the example complete, you'd also like to read the video somehow. Here's again a fully working example of OpenCV VideoCapture with monochrome 16-bit depth video.

import cv2

video_capture = cv2.VideoCapture(
    filename="video.mkv",
    apiPreference=cv2.CAP_FFMPEG,
    params=[
        cv2.CAP_PROP_CONVERT_RGB,
        0,  # false
    ],
)

if not video_capture.isOpened():
    print("Error reading video file")

while True:
    ret, frame = video_capture.read()
    if ret:
        cv2.imshow('Frame', frame)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    else:
        break

video_capture.release()
cv2.destroyAllWindows()

Be aware that produced video might not be playable with every video player since FFV1 encoding of 16-bit depth mono video is not so common yet. VLC media player had this discussed and supports it since version 3.0.18 and some improvements might also come in version 4.0 when it is released.

Daniel Konečný
  • 3
  • 7