4

I'm new to development with ffmpeg and I was wondering if it is possible to get the same bit exact output by running ffmpeg with the libx264 encoder on different machines (compiled using different compilers) with the same input file and same configuration options?

If it is possible, are there additional configuration options that need to be set when compiling libx264/ffmpeg? If it is not possible, why?

Below is the output from my running two versions off ffmpeg with libx264 that yield different MD5 hashes of the outputs.

ffmpeg installed via homebrew on OSX

ffmpeg version 3.4 Copyright (c) 2000-2017 the FFmpeg developers
  built with Apple LLVM version 9.0.0 (clang-900.0.38)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/3.4 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --disable-lzma --enable-nonfree

ffmpeg on Alpine Linux (in a Docker container based on https://github.com/jrottenberg/ffmpeg/blob/master/docker-images/3.4/alpine/Dockerfile)

ffmpeg version 3.4 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 6.2.1 (Alpine 6.2.1) 20160822
  configuration: --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --disable-lzma --enable-nonfree --prefix=/opt/ffmpeg

Test run with ffmpeg on OSX

ffmpeg -i seg.ts -c:v libx264 -s 1280x720 -minrate 6000k -maxrate 6000k -bufsize 6000k -r 60 -threads 1 -f md5 -

Output

MD5=b7c84c0bae5da6749e389a5b69d88582

Test run with ffmpeg on Alpine Linux (in a Docker container based on https://github.com/jrottenberg/ffmpeg/blob/master/docker-images/3.4/alpine/Dockerfile)

docker run --rm -v /code/:/tmp/workdir -w=/tmp/workdir ffmpeg -i seg.ts -c:v libx264 -s 1280x720 -minrate 6000k -maxrate 6000k -bufsize 6000k -r 60 -threads 1 -f md5 -

Output

MD5=c46af8fbdbbb2bfbb9f7042ab28accfc

pirs
  • 2,410
  • 2
  • 18
  • 25
yfu
  • 73
  • 7
  • Out of curiosity: why do you need this? – sascha Oct 23 '17 at 21:00
  • And for debugging: calculate this hash for decoding-only. I can imagine there will already be some differences (in some case: e.g. some audio codecs). – sascha Oct 23 '17 at 21:15
  • Trying to check if encoding was performed properly on one machine with another. Running ffmpeg without `-c:v libx264` results in the same hash with each machine. – yfu Oct 23 '17 at 21:20
  • just to be clear and exclude any obvious stuff: are you able to produce the same MD5 using the same ffmpeg commandline on the same OS? – Harry Oct 23 '17 at 21:59
  • Have you tried with a fixed bit rate `-b:v` and a constant bit rate `-vbr off` ? – pirs Oct 23 '17 at 22:56
  • If it's to know if it's the same video, you could try to use only one pixel as reference like `-vf "crop=1:1:0:0"` (w,h,x,y) on one frame `-t 1` at SS times `-ss 0.5` – pirs Oct 23 '17 at 22:59

2 Answers2

2

If you use different builds of libx264 on different CPUs nobody can guarantee you bit identical output. With same libx264 build on different CPUs (different instruction set extensions support) you also can get different output but can try to fix it by using --cpu-independent param (for ffmpeg through -x264-params) during encoding (on both).

As for why it can be not possible with different libx264 builds is because different compilers can generate different float-math code which will differ in rounding errors (use x87 or sse) and so nothing is guaranteed. You can try to minimize probability by disabling AQ, MBTree or using CQP ratecontrol but that is all.

UPDATE. Also don't forget about using same threads number on different CPUs instead of auto.

nobody555
  • 2,239
  • 18
  • 18
  • I even suspect that you can't get the same result on the same machine compare to the thread activity at the time you were encoding the file. It ll need many experiments to ensure this fact. – pirs Oct 23 '17 at 23:10
  • 1
    On same machine and same libx264 build output is deterministic if your input to libx264 is deterministic and you don't use VBV or --non-deterministic param. VBV is deterministic if you use single thread encode. – nobody555 Oct 24 '17 at 06:41
1

I ran into this issue when testing on docker machines running on i7 vs i9s. At first I tried disabling different cpu capabilities in https://github.com/FFmpeg/FFmpeg/blob/master/configure

But then I noticed that both machines were using the same cpu capabilities. I saw that from the logs outputted by running the command. So then I saw in the logs that they had different thread and lookahead_thread counts. So I updated by specifying the counts with this param -x264-params threads=6:lookahead_threads=1

See this post https://streaminglearningcenter.com/blogs/ffmpeg-command-threads-how-it-affects-quality-and-performance.html

Daniel Kobe
  • 9,376
  • 15
  • 62
  • 109