1

I am trying to remove a background from a video using ffmpeg and a PY library that does that, the PY lib (backgroundremover) just creates a matte.mp4 file as an output, having the background as black and the person as white silhouette.

PY lib: https://github.com/nadermx/backgroundremover#advance-usage-for-video

What I am doing at the moment:

Shrink & convert the video to MP4

ffmpeg -i ios.mov -s 320x240 -filter:v fps=30 -vf scale=320:-2 edited.mp4

Create the matte video

backgroundremover -i edited.mp4 -wn 4 -mk -o matte.mp4

Create video with alpha channel (the problem)

ffmpeg -i edited.mp4 -i matte.mp4 -filter_complex "[0:v][1:v]alphamerge" -shortest -c:v qtrle -an output.mov

Last command fails with invalid frame sizes, how do I force a frame size or skip this check?

Error:

[swscaler @ 0x7ff5c957b000] No accelerated colorspace conversion found from yuv420p to argb.
[Parsed_alphamerge_0 @ 0x7ff5c4e6d480] Input frame sizes do not match (320x240 vs 426x320).
[Parsed_alphamerge_0 @ 0x7ff5c4e6d480] Failed to configure output pad on Parsed_alphamerge_0
Error reinitializing filters!
Failed to inject frame into filter network: Invalid argument

Answer:

ffmpeg -y -i edited.mp4 -i matte.mp4 -f lavfi -i color=c=black:s=320x240 -filter_complex "[1:v]scale=320:240,setsar=1:1,split[vs][alpha];[0:v][vs]alphamerge[vt];[2:v][vt]overlay=shortest=1[rgb];[rgb][alpha]alphamerge" -shortest -c:v hevc_videotoolbox -allow_sw 1 -alpha_quality 0.75 -vtag hvc1 -pix_fmt yuva420p -an output.mov
Alex
  • 1,210
  • 3
  • 21
  • 34

1 Answers1

1

The error Input frame sizes do not match (320x240 vs 426x320) is "self explained".

  • The resolution of edited.mp4 is 320x240.
  • The resolution of matte.mp4 is 426x320.
    I don't know why backgroundremover modifies the resolution from 320x240 to 426x320.

The rest of the messages are just warnings.

I am not sure about it, but I think the first FFmpeg command should be:

ffmpeg -y -i ios.mov -filter:v fps=30 -vf scale=320:240,setsar=1:1 edited.mp4

It's not solving the issue - the resolution of matte.mp4 is still 426x320.
It could be a bug in backgroundremover...

You may solve the error message using scale filer.
The alpha merge should be followed by an overlay filter:

ffmpeg -y -i edited.mp4 -i matte.mp4 -f lavfi -i color=c=black:s=320x240 -filter_complex "[1:v]scale=320:240,setsar=1:1[vs];[0:v][vs]alphamerge[vt];[2:v][vt]overlay=shortest=1" -shortest -c:v qtrle -an output.mov

Sample output:
enter image description here

Rotem
  • 30,366
  • 4
  • 32
  • 65
  • ffmpeg .... -shortest -c:v qtrle -an output.mov Kind of works, as in, it does create a working .mov video file, but I cannot retain the transparency in Safari. Like in: https://rotato.app/blog/transparent-videos-for-the-web – Alex Apr 13 '22 at 11:13
  • The posted command replaces the transparent background with black. For keeping transparency remove the overlay filter: `ffmpeg -y -i edited.mp4 -i matte.mp4 -filter_complex "[1:v]scale=320:240,setsar=1:1[vs];[0:v][vs]alphamerge" -shortest -c:v qtrle -pix_fmt rgba -an output.mov`. I didn't test the new command, so it's not going to work as is... – Rotem Apr 13 '22 at 11:40
  • Yeah, that kind of worked, there is no background if I use a 3rd party video player. Wouldn't play at all on Quicktime Player, and plays video WITH background on Safari. So everywhere is a different result :| – Alex Apr 14 '22 at 10:30
  • I also tried using: avconvert --preset PresetHEVC1920x1080WithAlpha --source output.mov --output output2.mov But it says that the present cannot be applied to that video (as in there's no alpha channel available) – Alex Apr 14 '22 at 10:31
  • HEVC with alpha is a different problem. The issue with my command with transparent background is compatibility with players that doesn't support alpha. What it should be is black without alpha, and transparent with alpha. New question maybe? – Rotem Apr 14 '22 at 11:14
  • How would you phrase this question? Also do you think it’s possible to do it? (I have access to apple hardware) – Alex Apr 15 '22 at 12:40
  • The Apple hardware encoder supports HEVC with alpha. I don't have access for testing it. – Rotem Apr 15 '22 at 13:07
  • 1
    I can help you ceating video with black background and alpha. I suppose it is required for compatibility with players that don't support alpha. (The zero pixels of the alpha channel should be black in the RGB channels). – Rotem Apr 15 '22 at 13:14
  • yeah, the main issue is getting transparency working with iOS devices, and Safari browser. I do have access to apple hardware, however I am still unable to do the proper video processing :| No matter what I tried, it still has background. – Alex Apr 29 '22 at 08:01
  • Try: `ffmpeg -y -i edited.mp4 -i matte.mp4 -f lavfi -i color=c=black:s=320x240 -filter_complex "[1:v]scale=320:240,setsar=1:1,split[vs][alpha];[0:v][vs]alphamerge[vt];[2:v][vt]overlay=shortest=1[rgb];[rgb][alpha]alphamerge" -shortest -c:v qtrle -pix_fmt argb -an output.mov` – Rotem Apr 29 '22 at 08:32
  • Try: `ffmpeg -y -i edited.mp4 -i matte.mp4 -f lavfi -i color=c=black:s=320x240 -filter_complex "[1:v]scale=320:240,setsar=1:1,split[vs][alpha];[0:v][vs]alphamerge[vt];[2:v][vt]overlay=shortest=1[rgb];[rgb][alpha]alphamerge" -shortest -c:v libvpx-vp9 -pix_fmt yuva420p -an output.webm` – Rotem Apr 29 '22 at 08:33
  • sadly this doesn't really work for me :(. You can see at: https://scraprobots.world/test and I also found a good article: https://rotato.app/blog/transparent-videos-for-the-web – Alex Apr 29 '22 at 08:57
  • the .mov video does not play at all, in any player I give it to, but there were no errors and it has size. – Alex Apr 29 '22 at 08:58
  • 1
    According to [this post](https://stackoverflow.com/questions/61661140/convert-webm-to-hevc-with-alpha), with Apple try: `ffmpeg -y -i edited.mp4 -i matte.mp4 -f lavfi -i color=c=black:s=320x240 -filter_complex "[1:v]scale=320:240,setsar=1:1,split[vs][alpha];[0:v][vs]alphamerge[vt];[2:v][vt]overlay=shortest=1[rgb];[rgb][alpha]alphamerge" -shortest -c:v hevc_videotoolbox -allow_sw 1 -alpha_quality 0.75 -vtag hvc1 -pix_fmt yuva420p -an output.mov` – Rotem Apr 29 '22 at 09:29
  • oh my god! That worked! Oh thank you so much! I've had so many attempts trying to get this work! Thank you! – Alex Apr 29 '22 at 10:09
  • Hey @Rotem ! By any chance do you know why audio is lost after conversion? :( – Alex Jun 21 '22 at 09:27
  • `-an` is used for removing the audio. – Rotem Jun 21 '22 at 09:48
  • that worked. tysm! – Alex Jun 21 '22 at 10:12
  • @Alex - I have a mac too but this command throws an error for me. Do you have more info on your ffmpeg version etc or run into this?: `Incompatible pixel format 'yuva420p' for codec 'hevc_videotoolbox', auto-selecting format 'bgra' zsh: illegal hardware instruction ffmpeg -y -i color.mp4 -i alpha.mp4 -f lavfi -i color=c=black:s=720x1280` – James Jan 16 '23 at 21:14