Prevent ffmpeg from changing the intensity of colors while downscaling the resolution of the video

Question

I have a use case where I need to downscale a 716x1280 mp4 video to 358x640 (half of the original). Command that I used is

ffmpeg -i ./input.mp4 -vf "scale=640:640:force_original_aspect_ratio=decrease,pad=ceil(iw/2)*2:ceil(ih/2)*2" ./output.mp4

Out of 10 sample videos, 2 of the them suffered impact on colors. Below I have attached a comparison from the one which was impacted the most.

NOTE: The one on the right is a frame from the original video and the frame on the left is the one from the processed (down scaled) video. Notice the colors red and green in the image (even the skin color and hair color were changed).

What I am looking for is

Is there any way I can prevent changes like these happening? Probably some flag on saturation, brightness, contrast or any other parameter.
I am assuming that ffmpeg uses some default settings while downscaling a video. What made ffmpeg change colors only for these two videos? If it made similar changes for the rest of the videos as well, how to predict this behaviour before hand?

EDIT:

What I already have Tried?

-crf with values 0 and 18.
-preset veryslow as mentioned here

None helped

Mediainfo input V/S output

param	input	output
color range	Limited	NA (attribute not in description)
color primaries	BT.2020	NA (attribute not in description)
transfer characteristics	HLG	NA (attribute not in description)
matrix coefficients	BT.2020 non-constant	NA (attribute not in description)
bit deapth	8	8

Logs of the ffmpeg command

ffmpeg -i ./input.mp4 -vf "scale=640:640:force_original_aspect_ratio=decrease,pad=ceil(iw/2)*2:ceil(ih/2)*2" -movflags +faststart ./output.mp4
ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with Apple clang version 12.0.0 (clang-1200.0.32.28)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.3.1_9 --enable-shared --enable-pthreads --enable-version3 --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-libsoxr --enable-videotoolbox --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from './input.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.45.100
  Duration: 00:00:30.05, start: 0.000000, bitrate: 10366 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, bt2020nc/bt2020/arib-std-b67), 716x1280, 10116 kb/s, 30 fps, 30 tbr, 19200 tbn, 38400 tbc (default)
    Metadata:
      handler_name    : Core Media Video
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 245 kb/s (default)
    Metadata:
      handler_name    : Core Media Audio
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[libx264 @ 0x7faab4808800] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x7faab4808800] profile High, level 3.0, 4:2:0, 8-bit
[libx264 @ 0x7faab4808800] 264 - core 161 r3027 4121277 - H.264/MPEG-4 AVC codec - Copyleft 2003-2020 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=12 lookahead_threads=2 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to './output.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.45.100
    Stream #0:0(und): Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 358x640, q=-1--1, 30 fps, 15360 tbn, 30 tbc (default)
    Metadata:
      handler_name    : Core Media Video
      encoder         : Lavc58.91.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : Core Media Audio
      encoder         : Lavc58.91.100 aac
[mp4 @ 0x7faab5808800] Starting second pass: moving the moov atom to the beginning of the file
frame=  901 fps=210 q=-1.0 Lsize=    3438kB time=00:00:30.02 bitrate= 938.0kbits/s speed=7.01x
video:2933kB audio:472kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.974633%
[libx264 @ 0x7faab4808800] frame I:6     Avg QP:22.60  size: 20769
[libx264 @ 0x7faab4808800] frame P:228   Avg QP:24.84  size:  7657
[libx264 @ 0x7faab4808800] frame B:667   Avg QP:27.59  size:  1697
[libx264 @ 0x7faab4808800] consecutive B-frames:  0.9%  0.9%  1.0% 97.2%
[libx264 @ 0x7faab4808800] mb I  I16..4:  9.5% 64.6% 26.0%
[libx264 @ 0x7faab4808800] mb P  I16..4:  2.5% 12.2%  2.5%  P16..4: 37.2% 20.6% 11.2%  0.0%  0.0%    skip:13.7%
[libx264 @ 0x7faab4808800] mb B  I16..4:  0.4%  2.1%  0.2%  B16..8: 42.2%  7.1%  1.2%  direct: 1.8%  skip:44.9%  L0:39.4% L1:52.8% BI: 7.8%
[libx264 @ 0x7faab4808800] 8x8 transform intra:72.2% inter:74.2%
[libx264 @ 0x7faab4808800] coded y,uvDC,uvAC intra: 61.8% 67.2% 20.2% inter: 16.7% 13.9% 1.3%
[libx264 @ 0x7faab4808800] i16 v,h,dc,p: 24% 19%  7% 50%
[libx264 @ 0x7faab4808800] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 16% 15%  6%  9% 11%  7% 10%  6%
[libx264 @ 0x7faab4808800] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 16% 13%  7%  9% 10%  7%  9%  4%
[libx264 @ 0x7faab4808800] i8c dc,h,v,p: 53% 16% 26%  5%
[libx264 @ 0x7faab4808800] Weighted P-Frames: Y:3.9% UV:1.8%
[libx264 @ 0x7faab4808800] ref P L0: 57.8% 19.5% 14.8%  7.8%  0.1%
[libx264 @ 0x7faab4808800] ref B L0: 90.7%  7.2%  2.1%
[libx264 @ 0x7faab4808800] ref B L1: 95.3%  4.7%
[libx264 @ 0x7faab4808800] kb/s:799.80
[aac @ 0x7faab2036a00] Qavg: 189.523

does this have to be done with ffmpeg? I found another question doing something similar here: https://video.stackexchange.com/questions/28889/ffmpeg-does-not-preserve-colors-after-resizing. Specifically the "Addendum to Ad 1" section at the bottom of the answer. — Ian Chu, Feb 17 '21 at 14:34
@IanChu I tried it while I was browsing the existing questions but wasn't able to run a successful iteration of `-profile:v main444-10`. I did try the `Ad 1` part though. It did not gave any better result. — dravit, Feb 17 '21 at 15:17
The color change is probably related to the color parameters of the encoded video stream: `Color range`, `Color primaries`, `Transfer characteristics` and `Matrix coefficients`. Can you please use [Mediainfo](https://mediaarea.net/en/MediaInfo) tool, and post these parameters of `input.mp4` and `output.mp4`? Post other parameters that might be important like `Bit depth` (if not 8 bits). — Rotem, Feb 17 '21 at 17:51
There is also a codec missing. Try using: `ffmpeg -i ./input.mp4 -vf "scale=640:640:force_original_aspect_ratio=decrease,pad=ceil(iw/2)*2:ceil(ih/2)*2" -vcodec libx264 -crf 17 -pix_fmt yuv420p ./output.mp4` — Rotem, Feb 17 '21 at 17:58
Show the complete log from the ffmpeg command in your question. — llogan, Feb 17 '21 at 18:27
@Rotem @llogan have added the logs and mediainfo attributes. Also, @Rotem using `-vcodec libx264` does not help. What could've helped is `-vcodec copy`. But stream copy is not supported with filtergraph. PS: if I run `ffmpeg -i input.mp4 output.mp4`, above image can be referred for comparison. But if I run `ffmpeg -i -vcodec copy input.mp4 output.mp4`, the output video get's fixed, i.e. there is no color change in the output video. — dravit, Feb 18 '21 at 07:26
I posted an answer. There is no guarantee that it's going to work, because there are too many variables, and I don't have your video file for validating the solution. — Rotem, Feb 18 '21 at 14:15
I quick ran your shared command and it did fix the issue for this video (big thanks to you). Though, I'd still like to experiment and play around with it to understand why these params were required for this specific video and are not required in general (other 8 samples did not show any such variation in output). Thanks a lot for helping me fix this one :D — dravit, Feb 18 '21 at 19:32
The metadata parameters of the other 8 samples could be different. The codec (h264 or h265 or vp9...) might be different. The bit depth (8 or 10) also meters. There are cases when the resolution of the output video makes a different (HD or SD resolution). There is also a chance that the differences of the input and output are so small that you can't see them (the dynamic range of the captured scene affects the differences). — Rotem, Feb 19 '21 at 09:11

Rotem · Accepted Answer · 2022-08-29T19:46:46.607

We may use Bit Stream Video Filter for setting h264 metadata.

When a video player plays a video file, it looks for metadata that attached to the video stream (h264 metadata for example).
The H.264 metadata parameters that affects the colors and brightness are: video_full_range_flag, colour_primaries, transfer_characteris and matrix_coefficients.

If the parameters are not set, there are defaults.
The defaults for low resolution video are "Limited Range" BT.601 (in most player - I am not sure about MAC OS).
The default gamma curve (affects the brightness) is sRGB gamma curve.
The player converts the pixels from YUV color space to RGB (for displaying the video). The conversion formula is done according to the metadata.

Your input video file input.mp4 has H.264 metadata parameters that are far from the default.

We can assume that scale video filter does not change the color characteristics (the filter applies the YUV elements without converting to RGB).

The characteristics of input.mp4 applies BT.2020, and HLG gamma curve, but converted as if they were default (BT.601 and sRGB gamma), so the colors and brightness are very different from what they should have been.

When FFmpeg encodes a video stream, it does not copy the metadata parameters from the input to the output - we need to set the parameters explicitly.

The solution is using a Bit Stream Video Filter for setting the metadata parameters.

Try using the following command:

ffmpeg -i ./input.mp4 -vf "scale=640:640:force_original_aspect_ratio=decrease,pad=ceil(iw/2)*2:ceil(ih/2)*2" -vcodec libx264 -crf 17 -pix_fmt yuv420p -bsf:v h264_metadata=video_full_range_flag=0:colour_primaries=9:transfer_characteristics=18:matrix_coefficients=9 ./output.mp4

video_full_range_flag=0 applies "limited color range".
colour_primaries=9 applies BT.2020 colour primaries.
transfer_characteristics=18 applies HLG gamma (see ITU-T Rec. Series H)
matrix_coefficients=9 applies BT.2020 matrix coefficients.

Most of the parameters are documented in ITU-T Rec. H.264 (section E.2.1).

Checking the parameters of output.mp4 using MediaInfo tool:

Color range                              : Limited
Color primaries                          : BT.2020
Transfer characteristics                 : HLG
Matrix coefficients                      : BT.2020 non-constant

Do you think it's possible to copy only the `h264_metadata` (and not the whole codec) from the input file? In other words, for the above attributes, use the same values from input.mp4. — dravit, Feb 20 '21 at 07:46
It would have been useful feature, but I can't find out how to do it using FFmpeg. — Rotem, Feb 20 '21 at 21:47

Prevent ffmpeg from changing the intensity of colors while downscaling the resolution of the video

1 Answers1

Linked