This document is about: FUSION 1

SWITCH TO

FUSION 1 FUSION 2

VR Shared

Overview

Fusion VR Shared demonstrates a quick and easy approach to start multiplayer games or applications with VR.

The choice between Shared or Host/Server topologies must be driven by your game specificities. In this sample, the Shared mode is used.

The purpose of this sample is to clarify how to handle VR rig and provide a basic teleport and grab example.

Before You start

The project has been developed with Unity 2021.3.29f1 and Fusion 1.1.8
To run the sample, first create a Fusion AppId in the PhotonEngine Dashboard and paste it into the App Id Fusion field in Real Time Settings (reachable from the Fusion menu). Then load the Launch scene and press Play.

Download

Version	Release Date	Download
1.1.8	Sep 21, 2023	Fusion VR Shared 1.1.8 Build 276

Handling Input

Meta Quest

Teleport : press A, B, X, Y, or any stick to display a pointer. You will teleport on any accepted target on release
Grab : first put your hand over the object and grab it using controller grab button

Mouse

A basic desktop rig is included in the project. It means that you have basic interaction using the mouse.

Move : left click with your mouse to display a pointer. You will teleport on any accepted target on release
Rotate : keep the right mouse button pressed and move the mouse to rotate the point of view
Grab : left click with your mouse on an object to grab it.

Connection Manager

The NetworkRunner is installed on the Connection Manager game object. The Connection Manager is in charge of configuring the game settings and starting the connection

private async void Start()
{
    // Launch the connection at start
    if (connectOnStart) await Connect();
}

public async Task Connect()
{
    // Create the scene manager if it does not exist
    if (sceneManager == null) sceneManager = gameObject.AddComponent<NetworkSceneManagerDefault>();
    if (onWillConnect != null) onWillConnect.Invoke();

    // Start or join (depends on gamemode) a session with a specific name
    var args = new StartGameArgs()
    {
        GameMode = GameMode.Shared,
        Scene = SceneManager.GetActiveScene().buildIndex,
        SceneManager = sceneManager
    };

    // Connexion criteria (note: actual project code contains alternative options)
    args.SessionName = roomName;

    await runner.StartGame(args);
}

Implementing INetworkRunnerCallbacks will allow Fusion NetworkRunner to interact with the Connection Manager class. In this sample, the OnPlayerJoined call back is used to spawn the local user prefab when the player joins the session.

public void OnPlayerJoined(NetworkRunner runner, PlayerRef player)
{
    if (player == runner.LocalPlayer && userPrefab != null)
    {
        // Spawn the user prefab for the local user
        NetworkObject networkPlayerObject = runner.Spawn(userPrefab, position: transform.position, rotation: transform.rotation, player, (runner, obj) => {
        });
    }
}

To ensure that the created player object is destroyed upon disconnection, please ensure that your prefab's NetworkObject has "Shared Mode Settings > Destroy when state authority leaves" checked.

Rigs

Overview

In an immersive application, the rig describes all the mobile parts that are required to represent an user, usually both hands, an head, and the play area (it is the personal space that can be moved, when an user teleports for instance),

While in a networked session, every user is represented by a networked rig, whose various parts positions are synchronized over the network.

Several architectures are possible, and valids, regarding how the rig parts are organized and synchronized. Here, an user is represented by a single NetworkObject, with several nested NetworkTransforms, one for each rig parts.

Regarding the specific case of the network rig representing the local user, this rig has to be driven by the hardware inputs. To simplify this process, a separate, non networked, rig has been created, called the “Hardware rig”. It uses Unity InputDevice API to collect the hardware inputs.

Details

Rig

All the parameters driving the rig (its position in space and the pose of the hands) are included in the RigState structure.

public struct RigState
{
    public Vector3 playAreaPosition;
    public Quaternion playAreaRotation;
    public Vector3 leftHandPosition;
    public Quaternion leftHandRotation;
    public Vector3 rightHandPosition;
    public Quaternion rightHandRotation;
    public Vector3 headsetPosition;
    public Quaternion headsetRotation;
    public HandCommand leftHandCommand;
    public HandCommand rightHandCommand;
}

The HardwareRig class updates the structure when requested. To do so, it collects input parameters from the various hardware rig parts.

 RigState _rigState = default;
        
public virtual RigState RigState
{
    get
    {
        _rigState.playAreaPosition = transform.position;
        _rigState.playAreaRotation = transform.rotation;
        _rigState.leftHandPosition = leftHand.transform.position;
        _rigState.leftHandRotation = leftHand.transform.rotation;
        _rigState.rightHandPosition = rightHand.transform.position;
        _rigState.rightHandRotation = rightHand.transform.rotation;
        _rigState.headsetPosition = headset.transform.position;
        _rigState.headsetRotation = headset.transform.rotation;
        _rigState.leftHandCommand = leftHand.handCommand;
        _rigState.rightHandCommand = rightHand.handCommand;
        return _rigState;
    }
}

It is the NetworkRig component, located on the user prefab, that request these inputs if it is associated with the local user, and then configures every networked rig parts to simply follow the input data coming from the matching hardware rig parts.

This is only done on the local user NetworkRig, the state authority. To ensure that those changes are replicated on proxies, (the instances of this player object on other players applications), other things have to be done:

for the rig parts position and rotation, those rig parts have NetworkTransform components, which already handle this synchronization when the Transform position or rotation are updated
for the application specific data, like the hand poses, networked vars (attributes with the [Networked] tag) are set, and callbacks are triggered on their value changes, to process their new values.

// As we are in shared topology, having the StateAuthority means we are the local user
public bool IsLocalNetworkRig => Object && Object.HasStateAuthority;

public override void Spawned()
{
    base.Spawned();
    if (IsLocalNetworkRig)
    {
        hardwareRig = FindObjectOfType<HardwareRig>();
        if (hardwareRig == null) Debug.LogError("Missing HardwareRig in the scene");
    }
}

public override void FixedUpdateNetwork()
{
    base.FixedUpdateNetwork();

    // Update the rig at each network tick for local player. The NetworkTransform will forward this to other players
    if (IsLocalNetworkRig && hardwareRig)
    {
        RigState rigState = hardwareRig.RigState;
        ApplyLocalStateToRigParts(rigState);
        ApplyLocalStateToHandPoses(rigState);
    }
}

protected virtual void ApplyLocalStateToRigParts(RigState rigState)
{
    transform.position = rigState.playAreaPosition;
    transform.rotation = rigState.playAreaRotation;
    leftHand.transform.position = rigState.leftHandPosition;
    leftHand.transform.rotation = rigState.leftHandRotation;
    rightHand.transform.position = rigState.rightHandPosition;
    rightHand.transform.rotation = rigState.rightHandRotation;
    headset.transform.position = rigState.headsetPosition;
    headset.transform.rotation = rigState.headsetRotation;
}

protected virtual void ApplyLocalStateToHandPoses(RigState rigState)
{
    // we update the hand pose info. It will trigger on network hands OnHandCommandChange on all clients, and update the hand representation accordingly
    leftHand.HandCommand = rigState.leftHandCommand;
    rightHand.HandCommand = rigState.rightHandCommand;
}

Aside from moving the network rig parts position during the FixedUpdateNetwork(), the NetworkRig component also handles the local extrapolation: during the Render(), for the local user having the authority on this object, the interpolation target, which handle the graphical representation of the various rig parts’ NetworkTransforms, is moved using the most recent local hardware rig part data.

It ensures that the local user always has the most up-to-date possible positions for their own hands (to avoid potential unease), even if the screen refresh rate is higher than the network tick rate.

The [OrderAfter] tag before the class ensure that the NetworkRig Render() will be called after the NetworkTransform methods, so that NetworkRig can override NetworkTransform own handling of the interpolation target.

public override void Render()
{
    base.Render();
    if (IsLocalNetworkRig)
    {
        // Extrapolate for local user :
        // we want to have the visual at the good position as soon as possible, so we force the visuals to follow the most fresh hardware positions
        // To update the visual object, and not the actual networked position, we move the interpolation targets
        RigState rigState = hardwareRig.RigState;

        networkTransform.InterpolationTarget.position = rigState.playAreaPosition;
        networkTransform.InterpolationTarget.rotation = rigState.playAreaRotation;
        leftHand.networkTransform.InterpolationTarget.position = rigState.leftHandPosition;
        leftHand.networkTransform.InterpolationTarget.rotation = rigState.leftHandRotation;
        rightHand.networkTransform.InterpolationTarget.position = rigState.rightHandPosition;
        rightHand.networkTransform.InterpolationTarget.rotation = rigState.rightHandRotation;
        headset.networkTransform.InterpolationTarget.position = rigState.headsetPosition;
        headset.networkTransform.InterpolationTarget.rotation = rigState.headsetRotation;
    }

Headset

The NetworkHeadset class is very simple : it provides an access to the headset NetworkTransform for the NetworkRig class

public class NetworkHeadset : NetworkBehaviour
{
    [HideInInspector]
    public NetworkTransform networkTransform;
    private void Awake()
    {
        if (networkTransform == null) networkTransform = GetComponent<NetworkTransform>();
    }
}

Hands

Like the NetworkHeadset class, the NetworkHand class provides access to the hand Network Transform for the NetworkRig class.

To synchronize the hand pose, a network structure called HandCommand has been created in the HardwareHand class.

// Structure representing the inputs driving a hand pose 
[System.Serializable]
public struct HandCommand : INetworkStruct
{
    public float thumbTouchedCommand;
    public float indexTouchedCommand;
    public float gripCommand;
    public float triggerCommand;
    // Optionnal commands
    public int poseCommand;
    public float pinchCommand;// Can be computed from triggerCommand by default
}

This HandCommand structure is used into the IHandRepresentation interface which set various hand properties including the hand pose. The NetworkHand can have a child IHandRepresentation, to which it will forward hand pose data.

    public interface IHandRepresentation
    {
        public void SetHandCommand(HandCommand command);
        public GameObject gameObject { get; }
        public void SetHandColor(Color color);
        public void SetHandMaterial(Material material);
        public void DisplayMesh(bool shouldDisplay);
        public bool IsMeshDisplayed { get; }
    }

The OSFHandRepresentation class, located on each hand, implements this interface in order to modify the fingers position thanks to the provided hand animator (ApplyCommand(HandCommand command) function).

Now, let's see how it is synchronized.

The HandCommand structure is updated with the fingers’ positions into the Update() of the HardwareHand

protected virtual void Update()
{
    // update hand pose
    handCommand.thumbTouchedCommand = thumbAction.action.ReadValue<float>();
    handCommand.indexTouchedCommand = indexAction.action.ReadValue<float>();
    handCommand.gripCommand = gripAction.action.ReadValue<float>();
    handCommand.triggerCommand = triggerAction.action.ReadValue<float>();
    handCommand.poseCommand = handPose;
    handCommand.pinchCommand = 0;
    // update hand interaction
    isGrabbing = grabAction.action.ReadValue<float>() > grabThreshold;
}

At each NetworkRig FixedUpdateNetwork(), the hand pose datas are updated on the local user, along with the others rig parts.

public override void FixedUpdateNetwork()
{
    base.FixedUpdateNetwork();

    // Update the rig at each network tick for local player. The NetworkTransform will forward this to other players
    if (IsLocalNetworkRig && hardwareRig)
    {
        RigState rigState = hardwareRig.RigState;
        ApplyLocalStateToHandPoses(rigState);
        ApplyLocalStateToRigParts(rigState);
    }
}


protected virtual void ApplyLocalStateToHandPoses(RigState rigState)
{
    // we update the hand pose info. It will trigger on network hands OnHandCommandChange on all clients, and update the hand representation accordingly
    leftHand.HandCommand = rigState.leftHandCommand;
    rightHand.HandCommand = rigState.rightHandCommand;
}

The NetworkHand component, located on each hand of the user prefab, manages the hand representation update.

To do so, the class contains a HandCommand networked structure.

[Networked(OnChanged = nameof(OnHandCommandChange))]
public HandCommand HandCommand { get; set; }

As the HandCommand is a networked var, the call back OnHandCommandChange() is called, on all clients, every time the networked structure changes, and updates the hand representation accordingly.

public static void OnHandCommandChange(Changed<NetworkHand> changed)
{
    // Will be called on all clients when the local user change the hand pose structure
    // We trigger here the actual animation update
    changed.Behaviour.UpdateHandRepresentationWithNetworkState();
}

void UpdateHandRepresentationWithNetworkState()
{
    if (handRepresentation != null) handRepresentation.SetHandCommand(HandCommand);
}

Similarly to what NetworkRig does for the rig part positions, during the Render(), NetworkHand also handles the extrapolation and update of the hand pose, using the local hardware hands.

public override void Render()
{
    base.Render();
    if (IsLocalNetworkRig)
    {
        // Extrapolate for local user : we want to have the visual at the good position as soon as possible, so we force the visuals to follow the most fresh hand pose
        UpdateRepresentationWithLocalHardwareState();
    }
}

void UpdateRepresentationWithLocalHardwareState()
{
    if (handRepresentation != null) handRepresentation.SetHandCommand(LocalHardwareHand.handCommand);
}

Teleport & locomotion

The RayBeamer class located on each hardware rig hand is in charge of displaying a ray when the user pushes a button. When the user releases the button, if the ray target is valid, then an event is triggered.

   if (onRelease != null) onRelease.Invoke(lastHitCollider, lastHit);

This event is listened to by the Rig Locomotion class located on the hardware rig.

       beamer.onRelease.AddListener(OnBeamRelease);

Then, it calls the rig teleport coroutine…

protected virtual void OnBeamRelease(Collider lastHitCollider, Vector3 position)
{
[...]
    if (ValidLocomotionSurface(lastHitCollider))
    {
        StartCoroutine(rig.FadedTeleport(position));
    }
}

Which updates the hardware rig position, and ask a Fader component available on the hardware headset to fade in and out the view during the teleport (to avoid cybersickness).

public virtual IEnumerator FadedTeleport(Vector3 position)
{
    if (headset.fader) yield return headset.fader.FadeIn();
    Teleport(position);
    if (headset.fader) yield return headset.fader.WaitBlinkDuration();
    if (headset.fader) yield return headset.fader.FadeOut();
}


public virtual void Teleport(Vector3 position)
{
    Vector3 headsetOffet = headset.transform.position - transform.position;
    headsetOffet.y = 0;
    Vector3 previousPosition = transform.position;
    transform.position = position - headsetOffet;
    if (onTeleport != null) onTeleport.Invoke(previousPosition, transform.position);
}

As seen previously, this modification on the hardware rig position will be synchronized over the network during the NetworkRig FixedUpdateNetwork().

The same strategy applies for the rig rotation where CheckSnapTurn() triggers a rig modification.

IEnumerator Rotate(float angle)
{
    timeStarted = Time.time;
    rotating = true;
    yield return rig.FadedRotate(angle);
    rotating = false;
}

public virtual IEnumerator FadedRotate(float angle)
{
    if (headset.fader) yield return headset.fader.FadeIn();
    Rotate(angle);
    if (headset.fader) yield return headset.fader.WaitBlinkDuration();
    if (headset.fader) yield return headset.fader.FadeOut();
}

public virtual void Rotate(float angle)
{
    transform.RotateAround(headset.transform.position, transform.up, angle);
}

Grabbing

Overview

The grabbing logic here is based on two networked components, NetworkHandColliderGrabber and NetworkHandColliderGrabbable:

the NetworkHandColliderGrabber triggers the grab and ungrab when the hardware hand has triggered a grab action over a grabbable object
the NetworkHandColliderGrabbable synchronizes other the network the grabbing info with network vars, so that the grabbable object follows its grabber on each players applications.

Note: While the rig parts positions and hand pose handling is very similar to what would be done in a host or server topology, the way the grabbing is handled here is very specific to the shared topology, to make it as simple to read as possible

This current page describes a very simple and easy to implement grabbing system, strongly tied to the network rig. An alternative implementation, relying more on the hardware rig, can be found here: VR Shared - Local rig grabbing

Details

Grabbing

The HardwareHand class, located on each hand, updates the isGrabbing bool at each update : the bool is true when the user presses the grip button.
Please note that, the updateGrabWithAction bool is used to support the deskop rig, a version of the rig that can be driven with the mouse and keyboard (this bool must be set to False for desktop mode, True for VR mode)

 protected virtual void Update()
 {
    // update hand pose
    handCommand.thumbTouchedCommand = thumbAction.action.ReadValue<float>();
    handCommand.indexTouchedCommand = indexAction.action.ReadValue<float>();
    handCommand.gripCommand = gripAction.action.ReadValue<float>();
    handCommand.triggerCommand = triggerAction.action.ReadValue<float>();
    handCommand.poseCommand = handPose;
    handCommand.pinchCommand = 0;
    // update hand interaction
    if(updateGrabWithAction) isGrabbing = grabAction.action.ReadValue<float>() > grabThreshold;
}

To detect collisions with grabbable objects, a simple box collider is located on each network hand.

In order to synchronize the grab action on the network, a NetworkHandColliderGrabber class is added on each network hand. When a collision occurs, the method OnTriggerStay(Collider other) is called.

First, because the collider is located on each network hand, it is required to restrict to collisions that concern the local hand, and not another player.

// We only trigger grabbing for our local hands
if (!hand.IsLocalNetworkRig || !hand.LocalHardwareHand) return;

Then, it checks if an object is already grabbed. For simplification, multiple grabbing is not allowed in this sample.

// Exit if an object is already grabbed
if (GrabbedObject != null)
{
    // It is already the grabbed object or another, but we don't allow shared grabbing here
    return;
}

Then it checks that :

the collided object can be grabbed (it has a NetworkHandColliderGrabbable component)
the user presses the grip button

If these conditions are met, the grabbed object is asked to follow the hand ((1) in the diagram above) thanks to the NetworkHandColliderGrabbable Grab method

NetworkHandColliderGrabbable grabbable;
if (lastCheckedCollider == other)
{
    grabbable = lastCheckColliderGrabbable;
} 
else
{
    grabbable = other.GetComponentInParent<NetworkHandColliderGrabbable>();
}

// To limit the number of GetComponent calls, we cache the latest checked collider grabbable result
lastCheckedCollider = other;
lastCheckColliderGrabbable = grabbable;
if (grabbable != null)
{
    if (hand.LocalHardwareHand.isGrabbing) Grab(grabbable);
}

Grabbing synchronization

Because the sample is using the shared mode, it is possible for every player to request the state authority on the object and change the network vars describing the grabbing state. Thus, it is possible that the player doesn’t have the authority on the grabbed object when he tries to grab it. So, the NetworkHandColliderGrabbable Grab method first requests the state authority before storing the current grabber (and the grab point offsets). Following the object position is active when IsGrabbed is true, that is when the CurrentGrabber is set.

public async void Grab(NetworkHandColliderGrabber newGrabber)
{
    if (onWillGrab != null) onWillGrab.Invoke(newGrabber);

    // Find grabbable position/rotation in grabber referential
    localPositionOffsetWhileTakingAuthority = newGrabber.transform.InverseTransformPoint(transform.position);
    localRotationOffsetWhileTakingAuthority = Quaternion.Inverse(newGrabber.transform.rotation) * transform.rotation;
    grabberWhileTakingAuthority = newGrabber;

    // Ask and wait to receive the stateAuthority to move the object
    isTakingAuthority = true;
    await Object.WaitForStateAuthority();
    isTakingAuthority = false;

    // We waited to have the state authority before setting Networked vars
    LocalPositionOffset = localPositionOffsetWhileTakingAuthority;
    LocalRotationOffset = localRotationOffsetWhileTakingAuthority;

    // Update the CurrentGrabber in order to start following position in the FixedUpdateNetwork
    CurrentGrabber = grabberWhileTakingAuthority;
}

Please note that CurrentGrabber, LocalPositionOffset and LocalRotationOffset are declared as a networked variable. It means that all players will receive their new values as soon as they are updated ((2) in the diagram above), and that the on change callbacks will allow all players to configure the object on grab and ungrab events (to edit/restore its kinematic state mainly, when needed ((3) in the diagram above) ).

Note: WaitForStateAuthority is an helper extension method

public static async Task<bool> WaitForStateAuthority(this NetworkObject o, float maxWaitTime = 8)
{
    float waitStartTime = Time.time;
    o.RequestStateAuthority();
    while (!o.HasStateAuthority && (Time.time - waitStartTime) < maxWaitTime)
    {
        await System.Threading.Tasks.Task.Delay(1);
    }
    return o.HasStateAuthority;
}

Follow

In the NetworkHandColliderGrabbable FixedUpdateNetwork(), the object position is updated to follow the grabbing hand when the player has the object authority and is grabbing the object ((4) in the diagram above). The NetworkTransform component on it will then ensure that the position is synched for all players.

public override void FixedUpdateNetwork()
{
    // We only update the object position if we have the state authority
    if (!Object.HasStateAuthority) return;

    if (!IsGrabbed) return;
    // Follow grabber, adding position/rotation offsets
    Follow(followingtransform: transform, followedTransform: CurrentGrabber.transform, LocalPositionOffset, LocalRotationOffset);
}

void Follow(Transform followingtransform, Transform followedTransform, Vector3 localPositionOffsetToFollowed, Quaternion localRotationOffsetTofollowed)
{
    followingtransform.position = followedTransform.TransformPoint(localPositionOffsetToFollowed);
    followingtransform.rotation = followedTransform.rotation * localRotationOffsetTofollowed;
}

Render

Similarly to NetworkRig & NetworkHand, NetworkHandColliderGrabbable handles the extrapolation and updates the position of the visual of the grabbed object during the Render() at the most recent position, even between network ticks ((5) in the diagram above). The various [OrderAfter] tag in the classes ensure that the NetworkGrabbble Render() will be called after the NetworkTransform methods, to override the NetworkTransform's handling of the interpolation target in those classes.

This extrapolation has however 2 specificities when compared to the previous ones:

First, the extrapolation is not restricted to the local user. When an object is grabbed, every user "knows" that it should follow the grabbing hand (thanks to the networked vars describing the grabbing): even if the network positions of the grabbed object and grabber might be a bit out of sync, the visuals have to match (to avoid having the object floating slightly around the hand on proxies).
Secondly, an option (enabled by default) has been added to extrapolate while taking the authority, in order to provide the best user experience: it avoids having the grabbed object staying still until the authority is received (even if it is a very short duration, users could perceive it slightly in VR).
So, while requesting the authority, the grabber and grabbing point positions are stored in temporary local vars, to have a specific extrapolation using those data.

public override void Render()
{
    if (isTakingAuthority && extrapolateWhileTakingAuthority)
    {
        // If we are currently taking the authority on the object due to a grab, the network info are still not set
        //  but we will extrapolate anyway (if the option extrapolateWhileTakingAuthority is true) to avoid having the grabbed object staying still until we receive the authority
        ExtrapolateWhileTakingAuthority();
        return;
    }

    // No need to extrapolate if the object is not grabbed
    if (!IsGrabbed) return;

    // Extrapolation: Make visual representation follow grabber, adding position/rotation offsets
    // We extrapolate for all users: we know that the grabbed object should follow accuratly the grabber, even if the network position might be a bit out of sync
    Follow(followingtransform: networkTransform.InterpolationTarget.transform, followedTransform: CurrentGrabber.hand.networkTransform.InterpolationTarget.transform, LocalPositionOffset, LocalRotationOffset);
}

void ExtrapolateWhileTakingAuthority()
{
    // No need to extrapolate if the object is not really grabbed
    if (grabberWhileTakingAuthority == null) return;

    // Extrapolation: Make visual representation follow grabber, adding position/rotation offsets
    // We use grabberWhileTakingAuthority instead of CurrentGrabber as we are currently waiting for the authority transfer: the network vars are not already set, so we use the temporary versions
    Follow(followingtransform: networkTransform.InterpolationTarget.transform, followedTransform: grabberWhileTakingAuthority.hand.networkTransform.InterpolationTarget.transform, localPositionOffsetWhileTakingAuthority, localRotationOffsetWhileTakingAuthority);
}

Third party

Hands
- Oculus Sample Framework hands
Sounds
- obsydianx-interface-sfx-pack-1

Here are some suggestions for modification or improvement that you can practice doing on this project :

Display the local teleport ray to others players (by using networked variable or INetworkInput )
Add voice capability. For more details on Photon Voice integration with Fusion, see this page: https://doc.photonengine.com/en-us/voice/current/getting-started/voice-for-fusion
Create a button to spawn additionnal networked object at runtime