2

I'm trying to duplicate a USB webcam device V4L2 stream (/dev/video0) to a V4L2 Loopback Device (/dev/video99) at the highest resolution and framerate possible with the hardware available on a Raspberry Pi 4 running the latest Raspbian (also the minimum CPU load). I'm using FFMPEG version 7:4.1.6-1~deb10u1+rpt2 and v4l2loopback version 0.12.5.1.

The output from: ffmpeg -f v4l2 -list_formats all -i /dev/video0 gives me the following:

[video4linux2,v4l2 @ 0xc1e1c0] Compressed:       mjpeg :          Motion-JPEG : 1600x1200 3264x2448 2592x1944 2048x1536 1280x960 1024x768 800x600 640x480 320x240 1600x1200
[video4linux2,v4l2 @ 0xc1e1c0] Raw       :     yuyv422 :           YUYV 4:2:2 : 1600x1200 3264x2448 2592x1944 2048x1536 1280x960 1024x768 800x600 640x480 320x240 1600x1200

3264x2448px is the native CCD resolution, and the image is being used for machine vision purposes, so I'd like to use this maximum resolution at the highest framerate I can. The reason for the loopback device is so that I can view/stream the videofeed (with GStreamer) and take high resolution PNG snapshots simultaneously, using the command ffmpeg -f v4l2 -video_size 3264x2448 -i /dev/video99 -frames 1 capture.png -y From my understanding this means there is an MJPEG stream at this resolution and according to the command v4l2-ctl -d /dev/video0 --list-formats-ext I can manage 15fps:

ioctl: VIDIOC_ENUM_FMT
    Type: Video Capture

    [0]: 'MJPG' (Motion-JPEG, compressed)
        Size: Discrete 1600x1200
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 3264x2448
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 2592x1944
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 2048x1536
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 1280x960
            Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 1024x768
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 800x600
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 640x480
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 320x240
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 1600x1200
            Interval: Discrete 0.067s (15.000 fps)
            Interval: Discrete 0.067s (15.000 fps)
    [1]: 'YUYV' (YUYV 4:2:2)
        Size: Discrete 1600x1200
            Interval: Discrete 0.100s (10.000 fps)
            Interval: Discrete 0.100s (10.000 fps)
        Size: Discrete 3264x2448
            Interval: Discrete 0.500s (2.000 fps)
        Size: Discrete 2592x1944
            Interval: Discrete 0.333s (3.000 fps)
        Size: Discrete 2048x1536
            Interval: Discrete 0.333s (3.000 fps)
        Size: Discrete 1280x960
            Interval: Discrete 0.100s (10.000 fps)
        Size: Discrete 1024x768
            Interval: Discrete 0.100s (10.000 fps)
        Size: Discrete 800x600
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 640x480
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 320x240
            Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 1600x1200
            Interval: Discrete 0.100s (10.000 fps)
            Interval: Discrete 0.100s (10.000 fps)

I feel like I've tried every possible version of GStreamer/FFMPEG command to achieve this and have had varying success, but more often than not I come up against "av_interleaved_write_frame(): Cannot allocate memory" or some issue with converting M-JPEG to rawvideo for the loopback device...

What command should I be using? I'm at the end of my tether!

I've tried:

gst-launch-1.0 v4l2src device=/dev/video0 ! "image/jpeg,width=3264,height=2448,framerate=15/1" ! avdec_mjpeg ! "video/x-raw,format=YUY2,width=3264,height=2448,framerate=15/1" ! v4l2sink device=/dev/video99

and

ffmpeg -f v4l2 -video_size 3264x2448 -i /dev/video0 -vcodec rawvideo -pix_fmt yuyv422 -r 1 -f v4l2 /dev/video99

(EDIT) gives:

ffmpeg version 4.1.6-1~deb10u1+rpt2 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 8 (Raspbian 8.3.0-6+rpi1)
  configuration: --prefix=/usr --extra-version='1~deb10u1+rpt2' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-vout-drm --enable-v4l2-request --enable-libudev --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared --libdir=/usr/lib/arm-linux-gnueabihf --cpu=arm1176jzf-s --arch=arm
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 14226.770484, bitrate: 255688 kb/s
    Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 3264x2448, 255688 kb/s, 2 fps, 2 tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> rawvideo (native))
Press [q] to stop, [?] for help
Output #0, video4linux2,v4l2, to '/dev/video99':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 3264x2448, q=2-31, 127844 kb/s, 1 fps, 1 tbn, 1 tbc
    Metadata:
      encoder         : Lavc58.35.100 rawvideo
av_interleaved_write_frame(): Cannot allocate memory
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:01.00 bitrate=N/A speed=15.8x    
video:15606kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Conversion failed!

and

ffmpeg -f v4l2 -video_size 3264x2448 -i /dev/video0 -vcodec rawvideo -pix_fmt yuv420p -r 15 -f v4l2 /dev/video99

(EDIT) gives:

ffmpeg version 4.1.6-1~deb10u1+rpt2 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 8 (Raspbian 8.3.0-6+rpi1)
  configuration: --prefix=/usr --extra-version='1~deb10u1+rpt2' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-vout-drm --enable-v4l2-request --enable-libudev --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared --libdir=/usr/lib/arm-linux-gnueabihf --cpu=arm1176jzf-s --arch=arm
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 13704.103283, bitrate: 255688 kb/s
    Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 3264x2448, 255688 kb/s, 2 fps, 2 tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> rawvideo (native))
Press [q] to stop, [?] for help
Output #0, video4linux2,v4l2, to '/dev/video99':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 3264x2448, q=2-31, 1438248 kb/s, 15 fps, 15 tbn, 15 tbc
    Metadata:
      encoder         : Lavc58.35.100 rawvideo
av_interleaved_write_frame(): Cannot allocate memory
    Last message repeated 7 times
frame=    8 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.53 bitrate=N/A dup=7 drop=0 speed=1.37x    
video:93636kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Conversion failed!

and

ffmpeg -f v4l2 -input_format mjpeg -i /dev/video0 -vcodec copy -f v4l2 /dev/video99

(EDIT) gives:

ffmpeg version 4.1.6-1~deb10u1+rpt2 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 8 (Raspbian 8.3.0-6+rpi1)
  configuration: --prefix=/usr --extra-version='1~deb10u1+rpt2' --toolchain=hardened --incdir=/usr/include/arm-linux-gnueabihf --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-omx-rpi --enable-mmal --enable-neon --enable-rpi --enable-vout-drm --enable-v4l2-request --enable-libudev --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared --libdir=/usr/lib/arm-linux-gnueabihf --cpu=arm1176jzf-s --arch=arm
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
[mjpeg @ 0xdb5240] EOI missing, emulating
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 12511.425271, bitrate: N/A
    Stream #0:0: Video: mjpeg, yuvj422p(pc, bt470bg/unknown/unknown), 3264x2448, 15 fps, 15 tbr, 1000k tbn, 1000k tbc
[video4linux2,v4l2 @ 0xdb7f10] V4L2 output device supports only a single raw video stream
Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
    Last message repeated 1 times

I'm not sure what to troubleshoot next - looking at available memory, there's no problem, so what am I doing wrong? This copy of the feed is the last piece of the puzzle...

Chris White
  • 21
  • 3
  • Copy & paste all text from the log from your last ffmpeg command. – llogan Jul 27 '21 at 21:19
  • Thanks man, I was hoping you'd respond! I've followed a LOT of your posts thus far... – Chris White Jul 27 '21 at 21:24
  • I've added the output from the previous two before - they both give the memory allocation error. As they are not the MJPEG input, I suspect they are only capable of 2fps at that resolution, which is really not ideal! Thanks for the help so far! – Chris White Jul 27 '21 at 21:56
  • Does the problem only occur on the Raspberry Pi? Can you duplicate it on a linux desktop? 4.1 is old. Try git master, or 4.4 at least. – llogan Jul 28 '21 at 00:35
  • I only have a Raspberry PI and that's also what the final version will be running on... so if I can't get it to work on a Pi, then there's no help... It looks like I have the wrong syntax for the MJPEG stream, so I really want to make sure that's right before I have to resort to the raw video stream. Do you have any suggestions on what I could change for that line? Building 4.4 from source may be a little beyond my capabilities! – Chris White Jul 28 '21 at 09:08
  • Some further discoveries... I can stream the mjpeg stream from the camera to an mjpeg file using ```ffmpeg -f v4l2 -input_format mjpeg -framerate 15 -video_size 3264x2448 -i /dev/video0 -c copy -t 10 output1.mjpeg``` but it only seems to manage 2-3fps from the output. I can also play this mjpeg file using ```ffplay -i output1.mjpeg``` - so why can't I send it to the loopback device? – Chris White Jul 28 '21 at 15:09

1 Answers1

0

I don't have enough points to comment, so I'll write what I've got so far. I'm having a similar issue, but using an RPi3 and am trying to stream from a webcam at 1440p to two loopback devices. One will be used for live streaming and the other for computer vision.

ffmpeg

I've got it to work, but the stream is 6-12 FPS, with a speed of 0.2-0.4x (too slow for what I want). I don't know how to fix this. Note the below is for 1 stream, but can easily be duplicated for two streams.

ffmpeg -hide_banner -vcodec mjpeg -s 2560x1440 -i /dev/video0 -map 0:v -vcodec rawvideo -vf format=yuv420p -f v4l2 /dev/video2

hide_banner hides a lot of the text -vcodec rawvideo is already assumed, so isn't necessary -vf format=yuv420p is used to convert pixel fmts ot smth v4l2 supports. I think this is the issue, because pixels are being converted, when what I (we) actually want is to send the compressed mjpeg stream directly to the loopback device.

Note that I didn't specify -r as it seemed to increase performance.

gst

This gives me corrupted images while streaming, but is very fast. By corrupted, I mean the colours are funky and sometimes the image is "boxy".

gst-launch-1.0 v4l2src device=/dev/video0 ! tee name=t ! queue ! v4l2sink device=/dev/videoX t. ! queue ! v4l2sink device=/dev/videoY

Source: comment by Olivier in duplicate webcam stream in a virtual device created with v4l2loopback

Note: I wrote this from my phone and don't know how to set the formatting, so I apologise for that.

Akash Gupta
  • 11
  • 2