GStreamer: Multiple RTMP sources, Picture in Picture to mux on a Jetson Nano, then to be used with RTMP pipeline with Belabox

Question

My goal is to have (2) RTMP sources in a Picture in Picture composition, encoding it into h265 mpegts, muxing audio from only the cam1 rtmp source, then sending it to the appsink;

This is how I see it in my mind, but I'm probably wrong:

[Confirmed] Working (On Device) Picture in Picture Pipeline:

Devices used: Camlink 4k (Sony Action Cam FDR-x3000) and Logitech c920

v4l2src device=/dev/video0 ! nvvidconv ! queue ! comp.sink_0
v4l2src device=/dev/video1 ! video/x-raw, width=800, height=448, framerate=30/1, format=YUY2 !
videobox left=-4 right=-4 top=-4 bottom=-4 ! nvvidconv ! queue ! comp.sink_1
 
nvcompositor name=comp sink_0::width=1920 sink_0::height=1080 sink_1::width=640 sink_1::height=360 sink_1::xpos=1266 sink_1::ypos=706 !
queue ! identity name=v_delay signal-handoffs=TRUE ! nvvidconv interpolation-method=5 !
nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps !
h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 !
mpegtsmux name=mux ! appsink name=appsink
 
alsasrc device=hw:2 ! identity name=a_delay signal-handoffs=TRUE ! volume volume=1.0 !
audioconvert ! opusenc bitrate=320000 ! opusparse ! queue ! mux.

[Confirmed] Working RTMP Pipeline:

Device used: Samsung s10e using Larix Broadcaster to stream x264 via RTMP

rtmpsrc location=rtmp://127.0.0.1/live/cam1 ! 
flvdemux name=demux 

demux.video ! identity name=v_delay signal-handoffs=TRUE ! h264parse ! nvv4l2decoder ! nvvidconv ! 
textoverlay text='' valignment=top halignment=right font-desc="Sans, 10" name=overlay ! queue ! 
videorate ! video/x-raw,framerate=60/1 !
nvvidconv interpolation-method=5 ! 
nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! 
h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! mux. 

demux.audio ! aacparse ! avdec_aac ! identity name=a_delay signal-handoffs=TRUE ! volume volume=1.0 ! 
audioconvert ! opusenc bitrate=128000 ! opusparse ! queue max-size-time=10000000000 max-size-buffers=1000 ! mux. 

mpegtsmux name=mux ! 
appsink name=appsink

All my attempts have failed;

These are My Attempts:

Attempt 1:

rtmpsrc name=cam1 location=rtmp://127.0.0.1/live/cam1 ! flvdemux name=demux0 ! queue ! demux0.video ! identity name=v_delay signal-handoffs=TRUE ! h264parse ! nvv4l2decoder ! nvvidconv ! queue ! comp.sink_0 
rtmpsrc name=cam2 location=rtmp://127.0.0.1/live/cam2 ! flvdemux name=demux1 ! queue ! demux1.video ! identity signal-handoffs=TRUE ! h264parse ! nvv4l2decoder ! nvvidconv ! queue ! comp.sink_1 

nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_1::xpos=0 sink_1::ypos=240 sink_1::width=320 sink_1::height=240 ! 
videorate ! video/x-raw,framerate=60/1 ! 
nvvidconv interpolation-method=5 ! 

nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! 
h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! mux. 

demux0. ! queue ! audio/mpeg ! decodebin ! audioconvert ! audioresample ! autoaudiosink 

mpegtsmux name=mux ! 
appsink name=appsink

Attempt 2:

rtmpsrc name=cam1 location=rtmp://127.0.0.1/live/cam1 ! 
flvdemux name=demux0
demux0.video ! identity name=v_delay0 signal-handoffs=TRUE ! h264parse ! nvv4l2decoder ! nvvidconv ! queue ! comp.sink_0

rtmpsrc name=cam2 location=rtmp://127.0.0.1/live/cam2 ! 
flvdemux name=demux1
demux1.video ! identity name=v_delay1 signal-handoffs=TRUE ! h264parse ! nvv4l2decoder ! videobox left=-4 right=-4 top=-4 bottom=-4 ! nvvidconv ! queue ! comp.sink_1

nvcompositor name=comp sink_0::width=1920 sink_0::height=1080 sink_1::width=640 sink_1::height=360 sink_1::xpos=10 sink_1::ypos=10 ! 
queue ! identity name=v_delay0 signal-handoffs=TRUE ! nvvidconv interpolation-method=5 ! 
queue ! identity name=v_delay1 signal-handoffs=TRUE ! nvvidconv interpolation-method=5 ! 
nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! 
h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! 
mpegtsmux name=mux ! appsink name=appsink

demux0.audio ! aacparse ! avdec_aac ! identity name=a_delay signal-handoffs=TRUE ! volume volume=1.0 ! 
audioconvert ! opusenc bitrate=320000 ! opusparse ! queue max-size-time=10000000000 max-size-buffers=1000 ! mux.

Current GStreamer Configuration:

Update 1: I tried @SeB's Solution but it did not work:

Here are some screenshots showing the process:

videotestsrc on port 4953:

videotestsrc on port 4954:

full test pipeline:

Update 2: The Solution:

By Utilizing @SeB's answer and tinkering with it a bit, I was able to take two rtmpsrc's and compose them together, then send it to that same rtmp server under a different key, and use the rtmp pipeline that ships with the belacoder.

During my testing this only works if you follow the belabox tutorial, and not with the pre-made image.

Here is the pipeline that I used:

gst-launch-1.0 -v \
 rtmpsrc location=rtmp://127.0.0.1/live/cam1 ! flvdemux name=demux0 \
   demux0. ! queue ! video/x-h264 ! h264parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,width=1920,height=1080,pixel-aspect-ratio=1/1' ! identity ! queue ! comp.sink_0 \
   demux0. ! queue ! audio/mpeg ! mux. \
 rtmpsrc location=rtmp://127.0.0.1/live/cam2 ! flvdemux name=demux1 \
   demux1. ! queue ! video/x-h264 ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=YUY2,width=800,height=448,pixel-aspect-ratio=1/1 ! videobox left=-4 right=-4 top=-4 bottom=-4 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! identity ! queue ! comp.sink_1 \
 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_0::zorder=1 sink_1::xpos=0 sink_1::ypos=0 sink_1::width=808,sink_1::height=456 sink_1::zorder=2 ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' \
   ! nvv4l2h264enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! h264parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! mux. \
 flvmux name=mux ! rtmpsink location='location=rtmp://127.0.0.1/live/cam3 live=1'

Then I just edited the rtmp pipeline that comes with belacoder to pull from /cam3.

Here it is working in OBS Studio using belaUI + belacoder via SRTLA:

This is the pipeline I used in belaUI/belacoder:

rtmpsrc location=rtmp://127.0.0.1/live/cam3 ! 
flvdemux name=demux 

demux.video ! identity name=v_delay signal-handoffs=TRUE ! h264parse ! nvv4l2decoder ! nvvidconv ! 
textoverlay text='' valignment=top halignment=right font-desc="Sans, 10" name=overlay ! queue ! 
nvvidconv interpolation-method=5 ! 
nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! 
h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! mux. 

demux.audio ! aacparse ! avdec_aac ! identity name=a_delay signal-handoffs=TRUE ! volume volume=1.0 ! 
audioconvert ! voaacenc bitrate=128000 ! aacparse ! queue max-size-time=10000000000 max-size-buffers=1000 ! mux. 

mpegtsmux name=mux ! 
appsink name=appsink

My settings are unique to the rtmp server I have running on my belabox (Jetson-nano) so keep that in mind.

Here is the final pipeline selected in the belaUI:

Once you have it selected all you have to do is hit start and you can utilize all of the internet connections that are connected to the belabox:

Please keep in mind this is really finicky, if one of your rtmps sources crap out it ruins the whole pipeline, so this works best when all rtmp sources are in a local environment, and you have the gts-launch pipeline running as a service.

If you want more information about the open-source DIY project belabox or would like to contact me, check out my profile links @ https://stackoverflow.com/users/3331416/b3ck

SeB · Accepted Answer · 2022-04-04T17:02:05.740

Just tried simulating your sources with (I don't have a RTMP server, but should be straight forward to try adapting):

# Cam 1 1920x1080@30fps with audio
gst-launch-1.0 -e videotestsrc ! video/x-raw,format=NV12,width=320,height=240,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=1920,height=1080,pixel-aspect-ratio=1/1' ! nvv4l2h264enc ! h264parse ! queue ! flvmux name=mux    audiotestsrc ! audioconvert ! voaacenc ! queue ! mux.   mux. ! tcpserversink port=4953

# Cam2 with 800x448@30fps
gst-launch-1.0 -e videotestsrc pattern=ball ! video/x-raw,format=NV12,width=320,height=240,framerate=30/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12,width=800,height=448,pixel-aspect-ratio=1/1' ! nvv4l2h264enc ! h264parse ! queue ! flvmux ! tcpserversink port=4954

Then, this should output video and audio:

gst-launch-1.0 -v \
 tcpclientsrc port=4953 ! flvdemux name=demux0 ! h264parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,width=1920,height=1080,pixel-aspect-ratio=1/1' ! identity ! queue ! comp.sink_0 \
 tcpclientsrc port=4954 ! flvdemux name=demux1 ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=YUY2,width=800,height=448,pixel-aspect-ratio=1/1 !  nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! identity ! queue ! comp.sink_1 \
 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_0::zorder=1 sink_1::xpos=0 sink_1::ypos=0 sink_1::width=800,sink_1::height=448 sink_1::zorder=2 ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! nvvidconv ! autovideosink \
 demux0. ! queue ! audio/mpeg ! decodebin ! audioconvert ! audioresample ! autoaudiosink

If ok, you can H265 encode composed video (note that here adding videobox the second frame will now have size 808x456) and forward mpeg audio with:

gst-launch-1.0 -v \
 tcpclientsrc port=4953 ! flvdemux name=demux0 \
   demux0. ! queue ! video/x-h264 ! h264parse ! nvv4l2decoder ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,width=1920,height=1080,pixel-aspect-ratio=1/1' ! identity ! queue ! comp.sink_0 \
   demux0. ! queue ! audio/mpeg ! tsmux. \
 tcpclientsrc port=4954 ! flvdemux name=demux1 \
   demux1. ! queue ! video/x-h264 ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=YUY2,width=800,height=448,pixel-aspect-ratio=1/1 ! videobox left=-4 right=-4 top=-4 bottom=-4 ! nvvidconv ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! identity ! queue ! comp.sink_1 \
 nvcompositor name=comp sink_0::xpos=0 sink_0::ypos=0 sink_0::width=1920 sink_0::height=1080 sink_0::zorder=1 sink_1::xpos=0 sink_1::ypos=0 sink_1::width=808,sink_1::height=456 sink_1::zorder=2 ! 'video/x-raw(memory:NVMM),format=RGBA,pixel-aspect-ratio=1/1' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=NV12' \
   ! nvv4l2h265enc control-rate=1 qp-range="28,50:0,38:0,50" iframeinterval=60 preset-level=4 maxperf-enable=true EnableTwopassCBR=true insert-sps-pps=true name=venc_bps ! h265parse config-interval=-1 ! queue max-size-time=10000000000 max-size-buffers=1000 max-size-bytes=41943040 ! tsmux. \
 mpegtsmux name=tsmux ! appsink name=appsink

Thank you so much, I tried out your solution and this was the error my app was getting, the log is to large to post here, please view this pastebin: pastebin.com/TwJe2pYY I have also added my work in my original answer to show what I tried and the errors that followed. — b3ck, Apr 03 '22 at 22:46
Hmm....I don't see in my pipelines where uri property is involved...Did you try my exact pipelines ? I can confirm these work on Jetson AGX Xavier with L4T R32.6.1. — SeB, Apr 04 '22 at 21:00
I have no Nano and cannot check, but I would expect this to work on Nano as well (although Nano has a single encoder engine). Please post commands with output logs, for first terminal running videotestsrc to tcp, second terminal running the second source, and then third terminal reading and composing. I suspect a typo makes gstreamer parsing wrong. — SeB, Apr 05 '22 at 19:22
Here are my logs: https://pastebin.com/HXEr2uWM , https://pastebin.com/gc1SankX , https://pastebin.com/ByMF3muu , https://pastebin.com/vYbEQg4D — b3ck, Apr 06 '22 at 09:05
And ultimately these are the GStreamer errors I get while using your pipeline with my actual RTMP Sources: https://pastebin.com/2GZcE7X9 — b3ck, Apr 06 '22 at 09:15
I'm trying to get this pipeline to work with https://github.com/BELABOX/belacoder maybe that will help clear some things. — b3ck, Apr 06 '22 at 10:27
The test receiver pipeline looks running. First, be aware that it may take about 10s to connect, decode and display. Second, this expects a local login and monitor connected to Nano. If you log in from remote: if having a monitor and sound card attached to Nano, you would have to set DISPLAY such as `export DISPLAY=:0` (or `:1`) before running the receiver. Otherwise, just check outputting to file replacing the end : `appsink name=appsink` by `filesink location=test.ts` and adding -e flag so that it will be properly finished when you stop receiver with Ctrl-C (running for 30s should be ok). — SeB, Apr 06 '22 at 11:01
okay how about taking an easier approach, how about I just send the composition of the two rtmp sources back to my local rtmp server using h264, then I just use my regular rtmp pipeline that I know works? how would I do that? — b3ck, Apr 06 '22 at 11:19