0

Is there a way to share each player's raycast pointer/cursor with the other players in realtime? I'm using PUN2 and MRTK2 and Hololens 2.

Approaches I've tried so far:

  1. I tried a naïve approach of modifying the MRTK provided ShellHandRayPointer to contain a PhotonView and then using that on the MixedRealityToolkit object within the scene, but that seems to have no effect.
  2. I've tried creating a cursor prefab based off of MRTK's CursorFocus in which I add a Surface Magnetism component (tracking the hands) and then instantiate this prefab for each player in PUN2's OnJoinedRoom callback. After the instantiation call, I add the object to a non-rendered layer for the player with the goal of hiding it for the local player but allowing it show up for other players. This seems to hide the object as expected when only one player is in the room, but when a second player joins, the first player then sees a cursor show up that tracks with their hand movement, which seems unexpected to me (of note is that I'm using one Hololens 2 headset with a computer acting as the second player). Though perhaps this "crossed" behavior is due to the Surface Magnetism component?

Thanks!

Step-by-step images of how I modified the ShellHandRayPointer with a PhotonView and then reattached to the MRTK system:

scene: scene

MRTK system: MRTK system part 1

MRTK system part 2: reference to cloned ShellHandRayPointer: MRTK system part 2: reference to cloned ShellHandRayPointer

my cloned ShellHandRayPointer part 1: my cloned ShellHandRayPointer part 1

PhotonView components expanded on the cloned ShellHandRayPointer: PhotonView components expanded on the cloned ShellHandRayPointer

John Cast
  • 1,771
  • 3
  • 18
  • 40
  • Do you refer to the example in https://learn.microsoft.com/en-ca/windows/mixed-reality/develop/unity/tutorials/mr-learning-sharing-01 to achieve multi-user sharing? ShellHandRayPointer is controlled by the Input system, you may need a custom prefab or simply clone and place it manually. – Seth DU - MSFT Aug 23 '22 at 10:33
  • @SethDU-MSFT could you elaborate more on each of those options? I've tried cloning the prefab and adding a PhotonView and then tying this to the MRTK system but that didn't have any effect that I'm aware of. I'm not clear on how to place it manually or if your suggestion of creating a custom prefab is any different than what I just explained. – John Cast Aug 23 '22 at 15:13
  • Would you share more details about how do you modified the ShellHandRayPointer? I am not clear about the 2nd question. Are you creating a cursor that only shows to other players? Step-by-step actions or screenshots may help. – Seth DU - MSFT Aug 24 '22 at 01:33
  • @SethDU-MSFT See my updated question for images of how I modified the `ShellHandRayPointer` and tried to attach it to MRTK within the scene. My expectation was that each player's ray/cursor would be viewable by every other player within the same game room. Thanks again – John Cast Aug 24 '22 at 15:30
  • 1
    Thanks for the update. I will dig into it and try to reproduce this issue. – Seth DU - MSFT Aug 25 '22 at 09:05

1 Answers1

4

Regarding how to share objects in Photon in real time, as far as I know, the real-time shared objects in Photon need to be instantiated by PhotonNetwork.Instantiate, but ShellHandRayPointer in MRTK is instantiated by Input system.

You can customize a copy of ShellHandRayPointer, map the position and rotation of ShellHandRayPointer to the copy at runtime and share this copy in Photon in real time.

The position and rotation of ShellHandRayPointer can be obtained in MixedRealityToolkit.InputSystem.DetectedInputSources. Or you can use Unity's methods to get this Game Object directly.

For the cursor, you can use the same method above to create a copy of the cursor and map its position and rotation.

Zuocheng Wang - MSFT
  • 746
  • 1
  • 2
  • 6