1

What I am attempting to do is this: My smartphone sends its orientation (fusion of Gyro, Accel and Magnetic) to the controller which further processes the data to make it accessible by SteamVr which in turn should send me the frames which I want to send back to my smartphone.

What I achieved so far: I implemented a basic android-program which already captures its orientation, packs it into an UDP-Datagram and sends it to a given IP-adress.

I wrote a small Java program to verify its functionality and it already works.

Now I want to head to step 2: give this data to SteamVr. Right now I don't care for streaming back the video to android.

Assuming I would like to use a higher language like C# or Java, could someone provide me some help? Could I use OpenVr for that? And if yes, how? (This is not that well documented).

Edit:

I have a temporary solution. I'll use FreePie to send orientation to OpenTrack via UDP (I have modified FreePie.apk to smooth out the result a bit more). OpenTrack sends this via UDP to localhost.

I have to modify SteamVr to use a Null-Driver in order to accept a non existant HMD.

HeadTracking and Stream via Moonlight works great but now there is another Problem: As SteamVR starts a FullScreen HMD-Display-Window I can't use usual Input for any games.

Games usually start up in a separate window which accepts the Input while the HMD displays the VR-Experience. Now, the VR-Window has a big Problem. When not in focus, it neither displays nor takes Input from OpenTrack. This means No Video, No Headtracking. Some games start themselves in VR mode so I could use this window to get KeyboardInput AND Video, but Headtracking is disabled then, as stated before.

So in short: - FreePie -> OpenTrack -> SteamVR NULL-Driver - Moonlight -> Android Works for everything that don't need input

ONLY SOLUTION here is: 2nd Display. -Set 2nd Display as Main. -Move everything to the 2nd extended Display - start steamVR (which will open in 2nd Display), - start the game. Now when focus is switched to VR-Window it suddenly appears at the PCs primary plugged in display (only tested it with Laptop + TV). So Moonlight will capture Video while 2nd (current Main) Screen focus on GameWindow to capture Input.

My new question is: How to work around the need for a 2nd Display?

Dennis Ziganow
  • 49
  • 1
  • 7

1 Answers1

0

All right. Here is my solution and guide:

How to use your Android Smartphone as a VR-HMD for SteamVr

What your need:

Hardware:

  • 2 Smartphones (1 Smartphone for VR, one that can be cheap, tablet does it too)
  • Nvidia GameStream capable PC with sufficient power for VR

Software:

  • SteamVr
  • OpenVr-OpenTrack (https://github.com/r57zone/OpenVR-OpenTrack)
  • OpenTrack 2.3-rc20 (I choose this because only extractable zip works for me)
  • FreePIE IMU apk for Android (Its already in the OpenTrack folder)
  • Spacedesk for Windows and Android
  • Moonlight for Android

How to:

  • Install Steam and SteamVR (and any game you want to play)
  • Install OpenVR-OpenTrack (follow Guide on the page)
  • Install OpenTrack
  • Install FreePIE IMU on Android Phone for VR
  • Install Moonlight on Android Phone for VR

Now make sure the VR-Phone and your PC are in the same Network. Setup in OpenTrack:

Tracker -> FreePIE UDP Receiver with Pitch +180. Set the Port to the same as in the FreePIE Android App

Protocol -> UDP Receiver IP: 127.0.0.1 Port: 4242

If you start FreePIE App and OpenTrack you should see it working.

Next Steps:

  • Install Spacedesk on Windows and the cheap phone
  • Connect both.
  • On PC set to extended display.

  • Set 2nd display as main-display

  • Now start SteamVR and your game (game should be in windowed mode).

  • Move the game-window to 2nd (main) display (if it is not already).
  • Focus on SteamVR-Window (the one which should be red/black by now)
  • Click into the screen.

The Window should now be displayed on your PCs Display while the game itself remains on the Android-Display.

  • Click to focus on the Game-Window.
  • Start Moonlight on your VR Phone and start desktop (For tutorial how to stream desktop, use google)

You are ready to Play. As your PCs monitor displays the VR-Window, it will show the content and accept Android-Orientations. As the extended (main)-Monitor displays the Game-Window, the game will accept controller/keyboard/mouse-input.

Yes it is a freaking loooong work around and prolly will stress your (W)lan. You can also do this with a physical 2nd monitor if you have one spare which will reduce traffic on your router.

Tried this with WarThunder and Project CARS Pagani Edition and it works!

Dennis Ziganow
  • 49
  • 1
  • 7