So I am running a go2rtc server and I'm receiving a rtsp stream from a camera and I want to draw a box on top of the video. The system has a Pentium Silver J5005 with iGPU. From what I understand I should be able to use hwmap instead of hwdownload/hwupload in this case because the iGPU and CPU share the same system memory. Anyway, leaving out the drawing part, I can tell that hardware decoding and encoding is working because ffmpeg only uses about 8% CPU. This is the ffmpeg command that I have working, but it is only decoding and re-encoding the video:
ffmpeg -hide_banner -v error -allowed_media_types video -loglevel verbose -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device /dev/dri/renderD128 \
-i rtsp://... -c:v h264_vaapi -g 50 -bf 0 -profile:v high -level:v 4.1 -sei:v 0 -an \
-filter_complex "[0:v]scale_vaapi,hwmap=mode=read+write+direct,format=nv12[in];\
[in]format=vaapi|nv12,hwmap[out]" -map "[out]" \
-c:v h264_vaapi -an -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://..."
Now i'm trying to insert a drawbox filter:
ffmpeg -hide_banner -v error -allowed_media_types video -loglevel verbose -hwaccel vaapi -hwaccel_output_format vaapi -hwaccel_device /dev/dri/renderD128 \
-i rtsp://... -c:v h264_vaapi -g 50 -bf 0 -profile:v high -level:v 4.1 -sei:v 0 -an \
-filter_complex "[0:v]scale_vaapi,hwmap=mode=read+write+direct,format=nv12[in];\
[in]drawbox=x=10:y=10:w=100:h=100:color=pink@0.5:t=fill[in2];\
[in2]format=vaapi|nv12,hwmap[out]" -map "[out]" \
-c:v h264_vaapi -an -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp://...
But this fails immedicately:
[h264 @ 0x55bf016ffc40] Reinit context to 2304x1296, pix_fmt: vaapi
[graph 0 input from stream 0:0 @ 0x55bf01fe9100] w:2304 h:1296 pixfmt:vaapi tb:1/90000 fr:20/1 sar:0/1
[auto_scale_0 @ 0x55bf01fee800] w:iw h:ih flags:'' interl:0
[Parsed_drawbox_3 @ 0x55bf01fe8180] auto-inserting filter 'auto_scale_0' between the filter 'Parsed_format_2' and the filter 'Parsed_drawbox_3'
[auto_scale_1 @ 0x55bf01feffc0] w:iw h:ih flags:'' interl:0
[Parsed_format_4 @ 0x55bf01fe8780] auto-inserting filter 'auto_scale_1' between the filter 'Parsed_drawbox_3' and the filter 'Parsed_format_4'
[auto_scale_0 @ 0x55bf01fee800] w:2304 h:1296 fmt:nv12 sar:0/1 -> w:2304 h:1296 fmt:yuv420p sar:0/1 flags:0x0
[Parsed_drawbox_3 @ 0x55bf01fe8180] x:10 y:10 w:100 h:100 color:0xC67B9B7F
[auto_scale_1 @ 0x55bf01feffc0] w:2304 h:1296 fmt:yuv420p sar:0/1 -> w:2304 h:1296 fmt:nv12 sar:0/1 flags:0x0
Last message repeated 3 times
[Parsed_hwmap_5 @ 0x55bf01fe8bc0] Failed to map frame: -38.
Error while filtering: Function not implemented
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
I found a similar question, but the solution of setting -hwaccel_output_format nv12 causes ffmpeg to fail (even if I don't include the drawbox step):
[Parsed_scale_vaapi_0 @ 0x55ab1c1d2540] auto-inserting filter 'auto_scale_0' between the filter 'graph 0 input from stream 0:0' and the filter 'Parsed_scale_vaapi_0'
Impossible to convert between the formats supported by the filter 'graph 0 input from stream 0:0' and the filter 'auto_scale_0'
It seems like the problem is the nv12 pixel format. I tried countless of ways to convert to e.g. rgb24 but everything I tried just caused ffmpeg to fail.
