ffmpeg with Quick Sync Video (qsv) gets stuck

Question

I've compiled ffmpeg in an Arch Linux distribution with support to Intel Quick Sync Video. But ffmpeg gets stuck when I try to run it:

$ ffmpeg -debug -hwaccel qsv -qsv_device /dev/dri/renderD128 -c:v h264_qsv -i test_video.mp4 -c:v hevc_qsv output.mp4

ffmpeg version N-109562-g0431f9805e Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12.2.0 (GCC)
  configuration: --prefix=/usr --extra-cflags=-I/opt/cuda/include --extra-ldflags=-L/opt/cuda/lib64 --enable-lto --disable-rpath --enable-gpl --enable-version3 --enable-nonfree --enable-shared --disable-static --disable-stripping --enable-gray --enable-alsa --enable-avisynth --enable-bzlib --enable-chromaprint --enable-frei0r --enable-gcrypt --enable-gmp --enable-gnutls --enable-iconv --enable-ladspa --enable-lcms2 --enable-libaom --enable-libaribb24 --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcelt --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libdavs2 --enable-libdc1394 --enable-libfdk-aac --enable-libflite --enable-fontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libiec61883 --enable-libilbc --enable-libjack --enable-libjxl --enable-libklvanc --enable-libkvazaar --enable-liblensfun --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --disable-libopencv --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --disable-libopenvino --enable-libopus --enable-libplacebo --enable-libpulse --enable-librabbitmq --enable-librav1e --enable-librist --enable-librsvg --enable-librubberband --enable-librtmp --enable-libshine --enable-libsmbclient --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libsvthevc --enable-libsvtvp9 --disable-libtensorflow --enable-libtesseract --enable-libtheora --disable-libtls --enable-libtwolame --enable-libuavs3d --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxavs2 --enable-libxcb --enable-libxcb-shm --enable-libxcb-xfixes --enable-libxcb-shape --enable-libxvid --enable-libxml2 --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-lzma --enable-decklink --disable-mbedtls --enable-libmysofa --enable-openal --enable-opencl --enable-opengl --disable-openssl --enable-pocketsphinx --enable-sndio --enable-sdl2 --enable-vapoursynth --enable-vulkan --enable-xlib --enable-zlib --enable-amf --enable-cuda-nvcc --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-libdrm --enable-libvpl --enable-libnpp --enable-nvdec --enable-nvenc --enable-omx --enable-rkmpp --enable-v4l2-m2m --enable-vaapi --enable-vdpau
  libavutil      57. 43.100 / 57. 43.100
  libavcodec     59. 56.100 / 59. 56.100
  libavformat    59. 34.102 / 59. 34.102
  libavdevice    59.  8.101 / 59.  8.101
  libavfilter     8. 53.100 /  8. 53.100
  libswscale      6.  8.112 /  6.  8.112
  libswresample   4.  9.100 /  4.  9.100
  libpostproc    56.  7.100 / 56.  7.100
 matched as AVOption 'debug' with argument '-hwaccel'.
Reading option 'qsv' ... matched as output url.
Reading option '-qsv_device' ... matched as option 'qsv_device' (set QSV hardware device (DirectX adapter index, DRM path or X11 display name)) with argument '/dev/dri/renderD128'.
Reading option '-c:v' ... matched as option 'c' (codec name) with argument 'h264_qsv'.
Reading option '-i' ... matched as input url with argument 'test_video.mp4'.
Reading option '-c:v' ... matched as option 'c' (codec name) with argument 'hevc_qsv'.
Reading option 'output.mp4' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option qsv_device (set QSV hardware device (DirectX adapter index, DRM path or X11 display name)) with argument /dev/dri/renderD128.
[AVHWDeviceContext @ 0x5590d517ee40] libva: VA-API version 1.17.0
[AVHWDeviceContext @ 0x5590d517ee40] libva: User requested driver 'iHD'
[AVHWDeviceContext @ 0x5590d517ee40] libva: Trying to open /usr/lib/dri/iHD_drv_video.so
[AVHWDeviceContext @ 0x5590d517ee40] libva: Found init function __vaDriverInit_1_16
[AVHWDeviceContext @ 0x5590d517ee40] libva: va_openDriver() returns 0
[AVHWDeviceContext @ 0x5590d517ee40] Initialised VAAPI connection: version 1.17
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x41524742 -> bgra.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x42475241 -> argb.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x41424752 -> rgba.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x52474241 -> abgr.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x58524742 -> bgr0.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x42475258 -> 0rgb.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x58424752 -> rgb0.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x52474258 -> 0bgr.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x30335241 -> unknown.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x30334241 -> unknown.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x30335258 -> x2rgb10le.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x30334258 -> unknown.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x36314752 -> unknown.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x50424752 -> unknown.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x50524742 -> unknown.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x56555941 -> unknown.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x30303859 -> gray.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x3231564e -> nv12.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x3132564e -> unknown.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x32595559 -> yuyv422.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x59565955 -> uyvy422.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x32315659 -> yuv420p.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x30323449 -> yuv420p.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x50313134 -> yuv411p.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x48323234 -> yuv422p.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x56323234 -> yuv440p.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x50343434 -> yuv444p.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x33434d49 -> unknown.
[AVHWDeviceContext @ 0x5590d517ee40] Format 0x30313050 -> p010le.
[AVHWDeviceContext @ 0x5590d517ee40] VAAPI driver: Intel iHD driver for Intel(R) Gen Graphics - 22.6.4 ().
[AVHWDeviceContext @ 0x5590d517ee40] Driver not found in known nonstandard list, using standard behaviour.
[AVHWDeviceContext @ 0x5590d517e940] Use Intel(R) oneVPL to create MFX session, API version is 2.6, the required implementation version is 1.3

CPU starts being used intensively and then nothing happens. Does anybody have any clue on it?

I have an Intel i7-8550U (8th gen Kaby Lake) processor with 4 cores (8 threads).

I have also used these options to ffmpeg, getting the same results:

ffmpeg -loglevel debug -init_hw_device qsv=hw -filter_hw_device hw -hwaccel qsv -hwaccel_output_format qsv -i test_video.mp4 -vf 'format=qsv,hwupload=extra_hw_frames=64' -c:v hevc_qsv -f mp4 output.mp4

With VAAPI, transcoding runs flowlessly:

ffmpeg -init_hw_device vaapi=intel:/dev/dri/renderD128 -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device intel -filter_hw_device intel -i test_video.mp4 -c:v hevc_vaapi -f mp4 output.mp4

Answer 1

My experience with ffmpeg's MFX and OneVPL integrations over the years show consistent success across multiple QSV-capable platforms using a hybrid strategy.

Here,we initialize a VAAPI's HWContext,followed by deriving a qsv device/instance for encoding via hwmap, rather than calling up QSV's HWContext directly. Using hwmap also allows further runtime flexibility as regards mapping modes, with the option to request zero-copy emulation via the direct mapping mode.

See the solution below,adapted from your command-line above:

ffmpeg -init_hw_device vaapi=intel:/dev/dri/renderD128 \
-hwaccel vaapi -hwaccel_output_format vaapi \
-hwaccel_device intel -filter_hw_device intel \
-i test_video.mp4 \
-vf 'scale_vaapi,hwmap=derive_device=qsv,format=qsv' \
-c:v hevc_qsv -f mp4 output.mp4

You can view the respective codec wrapper options via:

ffmpeg -h encoder=hevc_qsv

Sample output:

Encoder hevc_qsv [HEVC (Intel Quick Sync Video acceleration)]:
    General capabilities: delay hybrid 
    Threading capabilities: none
    Supported hardware devices: qsv qsv qsv 
    Supported pixel formats: nv12 p010le p012le yuyv422 y210le qsv bgra x2rgb10le vuyx xv30le
hevc_qsv encoder AVOptions:
  -async_depth       <int>        E..V....... Maximum processing parallelism (from 1 to INT_MAX) (default 4)
  -preset            <int>        E..V....... (from 0 to 7) (default 0)
     veryfast        7            E..V.......
     faster          6            E..V.......
     fast            5            E..V.......
     medium          4            E..V.......
     slow            3            E..V.......
     slower          2            E..V.......
     veryslow        1            E..V.......
  -forced_idr        <boolean>    E..V....... Forcing I frames as IDR frames (default false)
  -low_power         <boolean>    E..V....... enable low power mode(experimental: many limitations by mfx version, BRC modes, etc.) (default auto)
  -rdo               <int>        E..V....... Enable rate distortion optimization (from -1 to 1) (default -1)
  -max_frame_size    <int>        E..V....... Maximum encoded frame size in bytes (from -1 to INT_MAX) (default -1)
  -max_frame_size_i  <int>        E..V....... Maximum encoded I frame size in bytes (from -1 to INT_MAX) (default -1)
  -max_frame_size_p  <int>        E..V....... Maximum encoded P frame size in bytes (from -1 to INT_MAX) (default -1)
  -max_slice_size    <int>        E..V....... Maximum encoded slice size in bytes (from -1 to INT_MAX) (default -1)
  -mbbrc             <int>        E..V....... MB level bitrate control (from -1 to 1) (default -1)
  -extbrc            <int>        E..V....... Extended bitrate control (from -1 to 1) (default -1)
  -p_strategy        <int>        E..V....... Enable P-pyramid: 0-default 1-simple 2-pyramid(bf need to be set to 0). (from 0 to 2) (default 0)
  -b_strategy        <int>        E..V....... Strategy to choose between I/P/B-frames (from -1 to 1) (default -1)
  -dblk_idc          <int>        E..V....... This option disable deblocking. It has value in range 0~2. (from 0 to 2) (default 0)
  -low_delay_brc     <boolean>    E..V....... Allow to strictly obey avg frame size (default auto)
  -max_qp_i          <int>        E..V....... Maximum video quantizer scale for I frame (from -1 to 51) (default -1)
  -min_qp_i          <int>        E..V....... Minimum video quantizer scale for I frame (from -1 to 51) (default -1)
  -max_qp_p          <int>        E..V....... Maximum video quantizer scale for P frame (from -1 to 51) (default -1)
  -min_qp_p          <int>        E..V....... Minimum video quantizer scale for P frame (from -1 to 51) (default -1)
  -max_qp_b          <int>        E..V....... Maximum video quantizer scale for B frame (from -1 to 51) (default -1)
  -min_qp_b          <int>        E..V....... Minimum video quantizer scale for B frame (from -1 to 51) (default -1)
  -adaptive_i        <int>        E..V....... Adaptive I-frame placement (from -1 to 1) (default -1)
  -adaptive_b        <int>        E..V....... Adaptive B-frame placement (from -1 to 1) (default -1)
  -scenario          <int>        E..V....... A hint to encoder about the scenario for the encoding session (from 0 to 8) (default unknown)
     unknown         0            E..V.......
     displayremoting 1            E..V.......
     videoconference 2            E..V.......
     archive         3            E..V.......
     livestreaming   4            E..V.......
     cameracapture   5            E..V.......
     videosurveillance 6            E..V.......
     gamestreaming   7            E..V.......
     remotegaming    8            E..V.......
  -avbr_accuracy     <int>        E..V....... Accuracy of the AVBR ratecontrol (unit of tenth of percent) (from 0 to 65535) (default 0)
  -avbr_convergence  <int>        E..V....... Convergence of the AVBR ratecontrol (unit of 100 frames) (from 0 to 65535) (default 0)
  -skip_frame        <int>        E..V....... Allow frame skipping (from 0 to 3) (default no_skip)
     no_skip         0            E..V....... Frame skipping is disabled
     insert_dummy    1            E..V....... Encoder inserts into bitstream frame where all macroblocks are encoded as skipped
     insert_nothing  2            E..V....... Encoder inserts nothing into bitstream
     brc_only        3            E..V....... skip_frame metadata indicates the number of missed frames before the current frame
  -idr_interval      <int>        E..V....... Distance (in I-frames) between IDR frames (from -1 to INT_MAX) (default 0)
     begin_only      -1           E..V....... Output an IDR-frame only at the beginning of the stream
  -load_plugin       <int>        E..V....... A user plugin to load in an internal session (from 0 to 2) (default hevc_hw)
     none            0            E..V.......
     hevc_sw         1            E..V.......
     hevc_hw         2            E..V.......
  -load_plugins      <string>     E..V....... A :-separate list of hexadecimal plugin UIDs to load in an internal session (default "")
  -look_ahead_depth  <int>        E..V....... Depth of look ahead in number frames, available when extbrc option is enabled (from 0 to 100) (default 0)
  -profile           <int>        E..V....... (from 0 to INT_MAX) (default unknown)
     unknown         0            E..V.......
     main            1            E..V.......
     main10          2            E..V.......
     mainsp          3            E..V.......
     rext            4            E..V.......
     scc             9            E..V.......
  -tier              <int>        E..V....... Set the encoding tier (only level >= 4 can support high tier) (from 0 to 256) (default high)
     main            0            E..V.......
     high            256          E..V.......
  -gpb               <boolean>    E..V....... 1: GPB (generalized P/B frame); 0: regular P frame (default true)
  -tile_cols         <int>        E..V....... Number of columns for tiled encoding (from 0 to 65535) (default 0)
  -tile_rows         <int>        E..V....... Number of rows for tiled encoding (from 0 to 65535) (default 0)
  -recovery_point_sei <int>        E..V....... Insert recovery point SEI messages (from -1 to 1) (default -1)
  -aud               <boolean>    E..V....... Insert the Access Unit Delimiter NAL (default false)
  -pic_timing_sei    <boolean>    E..V....... Insert picture timing SEI with pic_struct_syntax element (default true)
  -transform_skip    <int>        E..V....... Turn this option ON to enable transformskip (from -1 to 1) (default -1)
  -int_ref_type      <int>        E..V....... Intra refresh type. B frames should be set to 0 (from -1 to 65535) (default -1)
     none            0            E..V.......
     vertical        1            E..V.......
     horizontal      2            E..V.......
     slice           3            E..V.......
  -int_ref_cycle_size <int>        E..V....... Number of frames in the intra refresh cycle (from -1 to 65535) (default -1)
  -int_ref_qp_delta  <int>        E..V....... QP difference for the refresh MBs (from -32768 to 32767) (default -32768)
  -int_ref_cycle_dist <int>        E..V....... Distance between the beginnings of the intra-refresh cycles in frames (from -1 to 32767) (default -1)

And for the filter(s) used:

ffmpeg -h filter=scale_vaapi

Sample output:

Filter scale_vaapi
  Scale to/from VAAPI surfaces.
    Inputs:
       #0: default (video)
    Outputs:
       #0: default (video)
scale_vaapi AVOptions:
   w                 <string>     ..FV....... Output video width (default "iw")
   h                 <string>     ..FV....... Output video height (default "ih")
   format            <string>     ..FV....... Output video format (software format of hardware frames)
   mode              <int>        ..FV....... Scaling mode (from 0 to 768) (default hq)
     default         0            ..FV....... Use the default (depend on the driver) scaling algorithm
     fast            256          ..FV....... Use fast scaling algorithm
     hq              512          ..FV....... Use high quality scaling algorithm
     nl_anamorphic   768          ..FV....... Use nolinear anamorphic scaling algorithm
   out_color_matrix  <string>     ..FV....... Output colour matrix coefficient set
   out_range         <int>        ..FV....... Output colour range (from 0 to 2) (default 0)
     full            2            ..FV....... Full range
     limited         1            ..FV....... Limited range
     jpeg            2            ..FV....... Full range
     mpeg            1            ..FV....... Limited range
     tv              1            ..FV....... Limited range
     pc              2            ..FV....... Full range
   out_color_primaries <string>     ..FV....... Output colour primaries
   out_color_transfer <string>     ..FV....... Output colour transfer characteristics
   out_chroma_location <string>     ..FV....... Output chroma sample location
   force_original_aspect_ratio <int>        ..FV....... decrease or increase w/h if necessary to keep the original AR (from 0 to 2) (default disable)
     disable         0            ..FV.......
     decrease        1            ..FV.......
     increase        2            ..FV.......
   force_divisible_by <int>        ..FV....... enforce that the output resolution is divisible by a defined integer when force_original_aspect_ratio is used (from 1 to 256) (default 1)

ffmpeg -h filter=hwmap

Sample output:

Filter hwmap
  Map hardware frames
    Inputs:
       #0: default (video)
    Outputs:
       #0: default (video)
hwmap AVOptions:
   mode              <flags>      ..FV....... Frame mapping mode (default read+write)
     read                         ..FV....... Mapping should be readable
     write                        ..FV....... Mapping should be writeable
     overwrite                    ..FV....... Mapping will always overwrite the entire frame
     direct                       ..FV....... Mapping should not involve any copying
   derive_device     <string>     ..FV....... Derive a new device of this type
   reverse           <int>        ..FV....... Map in reverse (create and allocate in the sink) (from 0 to 1) (default 0)

Extra notes:

Not all rate-control and encoding modes may be available on your platform. To verify what's supported, cross-verify vainfo's output with the support matrix hosted on intel-media-driver's project.

Intel's media delivery project on Github also provides reference command-lines outlining specific fix-ups for performance and quality trade-offs with respect to low-power encoding, rate control method availability, etc. It's a good start to grasp the basics on what may break in production.

Where possible, try to build the cartwheel-ffmpeg branch maintained by Intel for troubleshooting any QSV-related performance issues and likewise, report any bugs you may encounter there.