Use case: I have two gaming PCs running a VR game in two separate rooms and a laptop in the living room connected to a projector. The VR headsets are wireless and will be in the same room as the projector with two people playing multiplayer together and the spectator screens of each gaming PC displayed side by side on the projector for others to watch the action. The game is a rhythm game (Beat Saber) so low latency is extremely important. On the other hand I can sacrifice on video quality because each desktop will only display on a 960x540 portion of the 1080p projector screen. Also audio is already taken care of and doesn't need to be transmitted to the laptop.
I have written a program with WPF and C# which displays two webpages side by side with a black bar at the top and bottom. The idea was to log into a low latency screen sharing webpage (for example parsec.app) and connect one of the PCs on each side. This actually works, but the problem is that after connecting the second computer, they both become very laggy and low fps when viewing content with a lot of movement. I really need both video streams to be smooth and with low latency. (<150ms) So using a third party service for the sharing/streaming of the screens seems to be out of the question. So I have to find a way to send the desktop streams directly over the LAN and then display them side by side with my program. I would like to have my own program display the streams so that I have authority over the layout and can add thematic pictures to the unused space on the screen. Maybe even make it customizable with up to four streams simultaneously in the future.
The laptop and one of the gaming PCs are only connected to the LAN via WiFi and the other gaming PC is connected via Ethernet. I have done some research and ffmpg or NDI seem to be the lowest latency ways to send video through a network, only I have no idea how to use them and don't have any experience programming network applications. I tried streaming my screen from one PC to the other with VLC using UDP but couldn't even get that working..
Here's a link to my visual studio project so far: https://drive.google.com/file/d/1W7khWBvKZ1zMvreH9nyfAHPVQ6BDKN5Z/view?usp=share_link
Here's a video showing my program in action: https://drive.google.com/file/d/1db3EHHV23mvdky36fcox9--crlrbZXK4/view?usp=share_link
Does anyone have any insights or ideas on how to solve my problem? Of course I'm willing to do some studying too if someone can point me to the resources where I can learn the required skills to make this work.
