Can't write frame output with background removed to output video

Question

I'm attempting to use opencv and rembg to write an output video that walks through each frame and once it reaches a specific point in the video (start_frame) it runs rembg (remove background) on every frame until it reaches end_frame. It writes the output inline with the other frames. Essentially it will hit a point in the video and the background will disappear, once it reaches another point (end_frame) the background will reappear. I seem to be successful and it writes the video but none of the frames seem affected by the rembg that is run on frames x to x of the video. Am I missing something obvious? Here's my code:

import cv2
from rembg import remove

video_path = "D:\\frontloader.mp4"
video = cv2.VideoCapture("D:\\frontloaderShort.mp4")

w = int(video.get(3))
h = int(video.get(4))

print(w, h)

videoOut = cv2.VideoWriter('FrontLoaderOut.avi', cv2.VideoWriter_fourcc('M','J','P','G'), 24, (w, h))
    
start_frame = 200
end_frame = 230

while(True):
    sucess, frame = video.read()
    if(sucess):
        if video.get(cv2.CAP_PROP_POS_FRAMES) >= start_frame and video.get(cv2.CAP_PROP_POS_FRAMES) <= end_frame:
            print('no background')
            bg_removed = remove(frame)
            cv2.imshow("Frame", bg_removed)
            key = cv2.waitKey(1)
            videoOut.write(bg_removed)
        else:
            print('background')
            cv2.imshow("Frame", frame)
            key = cv2.waitKey(1)
            videoOut.write(frame)
    else:
        break

video.release()
videoOut.release()
cv2.destroyAllWindows() ```

score 2 · Answer 1 · answered Dec 07 '22 at 22:37

The output of format of bg_removed = remove(frame) has 4 color channels and applies BGRA pixel format.

The last channel applies Alpha (transparency channel).
alpha = 0 is fully transparent.
alpha = 255 is fully opaque.
Values between 0 and 255 are scaled linearly.

MJPEG codec does not support transparency.
The best option is to scale the BGR values, by the alpha channel for getting black background (instead of transparent background).

if video.get(cv2.CAP_PROP_POS_FRAMES) >= start_frame and video.get(cv2.CAP_PROP_POS_FRAMES) <= end_frame:
    print('no background')
    bg_removed = remove(frame)
    
    # The format of bg_removed is BGRA, A channel applies Alpha (transparency channel).
    alpha = bg_removed[:, :, 3]
    alpha = cv2.cvtColor(alpha, cv2.COLOR_GRAY2BGR)  # Duplicate alpha to 3 channels before scaling.
    
    # Scale BGR channel by alpha channel (the background is going to be black).
    bg_removed_bgr = cv2.multiply(alpha, bg_removed[:, :, 0:3], scale=1/255)  # bg_removed_bgr = alpha/255 * bg_removed[:, :, 0:3]
    
    cv2.imshow("Frame", bg_removed_bgr)
    key = cv2.waitKey(1)
    videoOut.write(bg_removed_bgr)

Sample bg_removed_bgr frame:

Awesome, Thanks so much for the response. That worked exactly as I had hoped, though I believe I have a lot of googling to begin to understand what's going on. — jport, Dec 07 '22 at 23:58
The principle is simple: We want to scale red, green ,blue components of each pixel by the corresponding alpha component. Value of alpha is supposed to be in range [0, 1], but due to the usage of uint8 type alpha range is [0, 255], so we have do divide alpha by 255 before the scaling. `bg_removed[:, :, 0:3]` is the BGR components without alpha. `alpha = cv2.cvtColor(alpha, cv2.COLOR_GRAY2BGR)` duplicates alpha to be [alpha, alpha, alpha] (we need it because `cv2.multiply` requires the multiplied images to be in the same dimensions). — Rotem, Dec 08 '22 at 08:13

Can't write frame output with background removed to output video

1 Answers1