1

The new spectator view seems to have changed the way it syncs states. It doesn't use UNet anymore but each component has a state synchronization service.

I can see how this synchronizes changes in objects that were in the scene from the start, but how does it work if I want to instantiate a prefab during runtime?

In the old UNet implementation I would use the NetworkManager to instantiate it on all connected devices, and the prefab would be created everywhere with all its referenes to child components intact. What is the new way of doing this?

jkdev
  • 11,360
  • 15
  • 54
  • 77
Krawlak
  • 13
  • 2

1 Answers1

4

State synchronization logic for Spectator View should automatically assess game object creation on the HoloLens/Broadcaster device and relay creation events to spectator/Observer devices. Spectator View logic does not support an object created on the spectator device being recreated on the HoloLens device. If this aligns with your needs, the flow of logic is the following:

  1. When a game object (flagged for synchronizing) is declared in the HoloLens's scene, it gets a TransformBroadcaster component added. This TransformBroadcaster monitors child hierarchy changes, so when a child game object of this TransformBroadcaster's game object is created, the parent TransformBroadcaster adds a TransformBroadcaster component to the child game object: https://github.com/microsoft/MixedReality-SpectatorView/blob/73414f4429048c6f0eed154fb8bbdea93b4caa81/src/SpectatorView.Unity/Assets/SpectatorView/Scripts/StateSynchronization/TransformBroadcaster.cs#L576

  2. Each frame update, the TransformBroadcaster also observes changes to its game object. If new components have been added to the game object that have associated ComponentBroadcasterServices, it adds ComponentBroadcasters to the game object: https://github.com/microsoft/MixedReality-SpectatorView/blob/73414f4429048c6f0eed154fb8bbdea93b4caa81/src/SpectatorView.Unity/Assets/SpectatorView/Scripts/StateSynchronization/TransformBroadcaster.cs#L502

  3. Once a ComponentBroadcaster (including the TransformBroadcaster) is added to a game object, it will assess all spectators on frame updates. If the ComponentBroadcaster detects that there is a spectator that does not know of its existence, it will send said spectator a broadcaster creation event: https://github.com/microsoft/MixedReality-SpectatorView/blob/73414f4429048c6f0eed154fb8bbdea93b4caa81/src/SpectatorView.Unity/Assets/SpectatorView/Scripts/StateSynchronization/ComponentBroadcaster.cs#L177

  4. The creation event will be received by the spectator device in the StateSynchronizationSceneManager. When the StateSynchronizationSceneManager sees this creation event, it obtains or creates a mirror game object (essentially a copy of the game object running on the HoloLens device). It then adds the corresponding ComponentObserver to the mirror game object using the associated IComponentBroadcasterService: https://github.com/microsoft/MixedReality-SpectatorView/blob/73414f4429048c6f0eed154fb8bbdea93b4caa81/src/SpectatorView.Unity/Assets/SpectatorView/Scripts/StateSynchronization/StateSynchronizationSceneManager.cs#L241

  5. When a ComponentObserver is added to a game object, it creates the component it is observing: https://github.com/microsoft/MixedReality-SpectatorView/blob/73414f4429048c6f0eed154fb8bbdea93b4caa81/src/SpectatorView.Unity/Assets/SpectatorView/Scripts/StateSynchronization/ComponentObserver.cs#L27 Then with each payload update, the ComponentObserver updates this attached component to mimic what is being displayed on the HoloLens device.

Once all ComponentBroadcasters register as created with the spectator device, you should have your newly created game object in the HoloLens scene recreated in the spectator scene; however, there a few other things to call out:

  1. Game objects need to be flagged in some manner as candidates for synchronization. This can be achieved through two manners. You can declare that all objects should be synchronized in your scene through the Spectator View settings: https://github.com/microsoft/MixedReality-SpectatorView/blob/master/doc/SpectatorView.Setup.md#hololens-2--hololens Or you can add a GameObjectHierarchyBroadcaster to your HoloLens scene (All child game objects under a GameObjectHierarchyBroadcaster component will be synchronized).

  2. If you have custom Unity types, you may need to create your own IComponentBroadcasterService, IComponentBroadcaster and IComponentObserver. Most of the default Unity components (transforms, meshes, etc) already have component broadcaster services, but your own custom types may require additional work.

  3. You need to have the same Asset Caches compiled into the HoloLens and spectator applications. When ComponentBroadcasters are created on spectator devices, they are often handed assetIds that are used to instantiate mirror game object components. For example, when a new MeshFilter is created it uses an assetId to look up the MeshFilter through the AssetService to know what mesh to add to the mirror game object: https://github.com/microsoft/MixedReality-SpectatorView/blob/73414f4429048c6f0eed154fb8bbdea93b4caa81/src/SpectatorView.Unity/Assets/SpectatorView/Scripts/StateSynchronization/NetworkedComponents/MeshFilter/MeshFilterObserver.cs#L15 This means that dynamically creating a mesh compared to relying on a mesh defined in one of your Assets will not work correctly.

chrisfromwork
  • 181
  • 7
  • Thanks, very informative! What I still don't understand though is how to handle a complex gameobject that has references to child components and maybe other aspects that don't map very well to this approach. The complex references are just set up once and the object is saved as a whole as a prefab (for example, a menu object with references to text components in its children). I wouldn't want to have to create broadcaster components for each and every custom script and all its parameters, because most parameters don't change during runtime and don't have to be synced. – Krawlak Aug 06 '19 at 13:26
  • Methods like GetOrCreateMirror are internal, I don't see an "official" way for the host to instantiate a prefab of a complex object, then this prefab is also instantiated on the viewers. The references are the same because it's from the same prefab, and I would only have to write custom broadcasters for components whose params change during runtime. – Krawlak Aug 06 '19 at 13:26
  • GetOrCreateMirror creates an empty game object that is intended to mimic a game object in the host scene. This will occur for the parent as well as all child game objects in the prefab to build the same game object hierarchy in the spectator device's scene. ComponentBroadcasterServices are then responsible for recreating the components you need on these mirror game objects. Most of the time, you only need some Unity types (Transforms, meshes, materials, etc) created and added as componens to these game objects because they are the only components that contribute to whats visually happening. – chrisfromwork Aug 06 '19 at 19:58
  • Although it may seem more natural to instantiate an entire prefab on the spectator device, the spectator device will actually recreate the prefab created on the host device based on all of the components it sees in the scene hierarchy. If some of these components don't register correctly, you may need to define custom ComponentBroadcasterServices. – chrisfromwork Aug 06 '19 at 20:04
  • It may also be worth calling out that a component broadcaster won't be created for your custom scripts. It will however be created for each unity type that has an associated ComponentBroadcasterService. These component broadcasters may increase the number of MonoBehaviours in your scene, but they will only send network messages when they assess that their component has been updated. – chrisfromwork Aug 06 '19 at 20:08
  • Thanks, I see the new approach is to mainly rely on "visual shells" being built up automatically on spectator side, without much underlying logic and references in object hierarchies. – Krawlak Aug 07 '19 at 12:41
  • Yeah thats a great way of phrasing it. If there are fundamental things that you need to work differently to unblock your scenarios, let us know/file issues against the repo. But, this "visual shells" approach is how the state synchronization logic was initially designed. Long term, there are some other mixed reality workstreams spinning up more robust state machines/synchronization logic to enable multiple users to act on a scene. Spectator views state synchronization logic was built primarily to enable filming. – chrisfromwork Aug 07 '19 at 21:26
  • I have a question to this topic. I created a "In-App-Browser" using an vnc-server and the nativ render plugin. Now I added the spectator view prefab. Everything works fine, except the rendered texture, so I cant see my web content. Do I need to write own IComponentBroadcasterService, IComponentBroadcaster and IComponentObserver for that? – Perazim Sep 16 '19 at 15:00