VR Shared - Local rig grabbing

This page describes an alternative implementation of a grabbing system for the VR Shared technical sample. To understand how the rig is setup, or any other details regarding this sample, please refer to the main VR Shared page first.

Differences with the basic VR Shared sample grabbing logic

This grabbing system, while very similar to the one described in the VR Shared technical sample, has some differences that can be useful in some situations:

the collider detecting grabbable objects is localized on the hardware rig instead of the network rig
this way, it can be used even when the network rig is not yet spawned (like full offline situations)
besides, it can be used in situation where on purpose we place the network rig at a position differing form the hardware rig (if the network rig position is smoothed during teleportation, ...)

Downloads

This sample is included in the VR Shared page download. The scene demonstrating the local rig grabbing is located in the Scenes/AlternativeHardwareBasedGrabbingDemo folder.

Overview

The grabbing logic here is separated in 2 parts:

the local part, non networked, that detected the actual grabbing/ungrabbing when the hardware hand has triggered a grab/ungrab action over a grabbable object (Grabber and Grabbable classes)
the networked part, that ensure that all players are aware of the grabbing status, and which manages the actual position change to follow the grabbing hand (NetworkGrabber and NetworkGrabbable classes).

Fusion VR Shared Local rig grabbing Logic

Note: the code contains a few lines to allow the local part to manage the following itself when used offline, for instance in use cases where the same components are used for an offline lobby. But this document will focus on the actual networked usages.

Details

Grabbing

The HardwareHand class, located on each hand, updates the isGrabbing bool at each update : the bool is true when the user presses the grip button.
Please note that, the updateGrabWithAction bool is used to support the deskop rig, a version of the rig that can be driven with the mouse and keyboard (this bool must be set to False for desktop mode, True for VR mode)

 protected virtual void Update()
 {
    // update hand pose
    // (...)

    // update hand interaction
    if(updateGrabWithAction) isGrabbing = grabAction.action.ReadValue<float>() > grabThreshold;
}

To detect collisions with grabbable objects, a simple box collider is located on each hardware hand, used by a Grabber component placed on this hand: when a collision occurs, the method OnTriggerStay(Collider other) is called.

First, it checks if an object is already grabbed. For simplification, multiple grabbing is not allowed in this sample.

// Exit if an object is already grabbed
if (grabbedObject != null)
{
    // It is already the grabbed object or another, but we don't allow shared grabbing here
    return;
}

Then it checks that :

the collided object can be grabbed (it has a Grabbable component)
the user presses the grip button

If these conditions are met, the grabbed object is asked to follow the hand ((1) in the diagram above) thanks to the Grabbable Grab method

Grabbable grabbable;

if (lastCheckedCollider == other)
{
    grabbable = lastCheckColliderGrabbable;
}
else
{
    grabbable = other.GetComponentInParent<Grabbable>();
}
// To limit the number of GetComponent calls, we cache the latest checked collider grabbable result
lastCheckedCollider = other;
lastCheckColliderGrabbable = grabbable;
if (grabbable != null)
{
    if (hand.isGrabbing) Grab(grabbable);
}

Grabbing synchronization

The Grabbable Grab() method stores the grabbing position offset.
It also tells the NetworkGrabbable associated component that a grab by the local user occured ((2) in the diagram above), through the networkGrabbable.LocalGrab() call

public virtual void Grab(Grabber newGrabber, Transform grabPointTransform = null)
{
    if (onWillGrab != null) onWillGrab.Invoke(newGrabber);

    // Find grabbable position/rotation in grabber referential
    localPositionOffset = newGrabber.transform.InverseTransformPoint(transform.position);
    localRotationOffset = Quaternion.Inverse(newGrabber.transform.rotation) * transform.rotation;
    currentGrabber = newGrabber;

    if (networkGrabbable)
    {
        networkGrabbable.LocalGrab();
    }
    else
    {
        // We handle the following if we are not online (online, the DidGrab will be called by the NetworkGrabbable DidGrab, itself called on all clients by HandleGrabberChange when the grabber networked var has changed)
        DidGrab();
    }
}

On the NetworkGrabbable, the LocalGrab() call :

first request the state authority on the object: this way, it will be possible to store [Networked] attributes values
once the state authority is received, it stores in those [Networked] attributes the details describing the grabbing (the grabber and the offset)

public async virtual void LocalGrab()
{
    // Ask and wait to receive the stateAuthority to move the object
    isTakingAuthority = true;
    await Object.WaitForStateAuthority();
    isTakingAuthority = false;

    // We waited to have the state authority before setting Networked vars
    LocalPositionOffset = grabbable.localPositionOffset;
    LocalRotationOffset = grabbable.localRotationOffset;

    // Update the CurrentGrabber in order to start following position in the FixedUpdateNetwork
    CurrentGrabber = grabbable.currentGrabber.networkGrabber;
}

Note: WaitForStateAuthority is an helper extension method

public static async Task<bool> WaitForStateAuthority(this NetworkObject o, float maxWaitTime = 8)
{
    float waitStartTime = Time.time;
    o.RequestStateAuthority();
    while (!o.HasStateAuthority && (Time.time - waitStartTime) < maxWaitTime)
    {
        await System.Threading.Tasks.Task.Delay(1);
    }
    return o.HasStateAuthority;
}

The [Networked] var changes will trigger OnGrabberChanged on all clients ((3) in the diagram above):

[Networked(OnChanged = nameof(OnGrabberChanged))]
public NetworkGrabber CurrentGrabber { get; set; }

In it, the LoadOld() and LoadNew() calls allow to compare the CurrentGrabber value to the previous one:

// Callback that will be called on all clients on grabber change (grabbing/ungrabbing)
public static void OnGrabberChanged(Changed<NetworkGrabbable> changed)
{
    // We load the previous state to find what was the grabber before
    changed.LoadOld();
    NetworkGrabber previousGrabber = null;
    if (changed.Behaviour.CurrentGrabber != null)
    {
        previousGrabber = changed.Behaviour.CurrentGrabber;
    }
    // We reload the current state to see the current grabber
    changed.LoadNew();

    changed.Behaviour.HandleGrabberChange(previousGrabber);
}

This way, every client can call DidGrab() and DidUngrab() when relevant, both method forwarding to the Grabbable DidGrab() and DidUngrab() calls ((4) in the diagram above):

protected virtual void HandleGrabberChange(NetworkGrabber previousGrabber)
{
    if (previousGrabber)
    {
        DidUngrab();
    }
    if (CurrentGrabber)
    {
        DidGrab();
    }
}

protected virtual void DidGrab()
{
    grabbable.DidGrab();
    if (onDidGrab != null) onDidGrab.Invoke(CurrentGrabber);
}

protected virtual void DidUngrab()
{
    grabbable.DidUngrab();
    if (onDidUngrab != null) onDidUngrab.Invoke();
}

This way, the Grabbable component can properly set the isKinematic value (the object physics is disabled when grabbed), and apply release velocity

public virtual void DidGrab()
{
    // While grabbed, we disable physics forces on the object, to force a position based tracking
    if (rb) rb.isKinematic = true;
}

public virtual void DidUngrab()
{
    // We restore the default isKinematic state if needed
    if (rb) rb.isKinematic = expectedIsKinematic;

    // We apply release velocity if needed
    if (rb && rb.isKinematic == false && applyVelocityOnRelease)
    {
        rb.velocity = Velocity;
        rb.angularVelocity = AngularVelocity;
    }

    ResetVelocityTracking();
}

Follow

As the object physics is disabled while grabbed, following the current grabber is simply teleporting to its actual position:

public virtual void Follow(Transform followingtransform, Transform followedTransform, Vector3 localPositionOffsetToFollowed, Quaternion localRotationOffsetTofollowed)
{
    followingtransform.position = followedTransform.TransformPoint(localPositionOffsetToFollowed);
    followingtransform.rotation = followedTransform.rotation * localRotationOffsetTofollowed;
}

When online, the following code is called during FixedUpdateNetwork calls ((5) in the diagram above):

public override void FixedUpdateNetwork()
{
    // We only update the object position if we have the state authority
    if (!Object.HasStateAuthority) return;

    if (!IsGrabbed) return;
    // Follow grabber, adding position/rotation offsets
    grabbable.Follow(followingtransform: transform, followedTransform: CurrentGrabber.transform, LocalPositionOffset, LocalRotationOffset);
}

The position change is only done on the grabbing user client (the state authority), and then the NetworkTransform ensures that all players receive the position updates.

Render

Regarding the extrapolation ((6) in the diagram above), made during the Render() (the NetworkGrabbable class has a OrderAfter directive to override the NetworkTransform interpolation if needed ), 2 cases have to be handled here :

extrapolation while the object is grabbed, for all client: the object expected position is known, it should be on the hand position. So the grabbable visual (ie. NetworkTransform's interpolation target) has to be on the position of the hand visual.
extrapolation while the object authority is requested, for the grabbing client: during a few frames, CurrentGrabber is not yet set has the authority request is still pending ([Networked] var can only be set while having the state authority). So IsGrabbed won't yet return true, and the actual NetworkGrabber to follow has to be found through the local Grabbable and Grabber components.

public override void Render()
{
    if (isTakingAuthority && extrapolateWhileTakingAuthority)
    {
        // If we are currently taking the authority on the object due to a grab, the network info are still not set
        //  but we will extrapolate anyway (if the option extrapolateWhileTakingAuthority is true) to avoid having the grabbed object staying still until we receive the authority
        ExtrapolateWhileTakingAuthority();
        return;
    }

    // No need to extrapolate if the object is not grabbed
    if (!IsGrabbed) return;

    // Extrapolation: Make visual representation follow grabber, adding position/rotation offsets
    // We extrapolate for all users: we know that the grabbed object should follow accuratly the grabber, even if the network position might be a bit out of sync
    grabbable.Follow(followingtransform: networkTransform.InterpolationTarget.transform, followedTransform: CurrentGrabber.hand.networkTransform.InterpolationTarget.transform, LocalPositionOffset, LocalRotationOffset);
}

protected virtual void ExtrapolateWhileTakingAuthority()
{
    // No need to extrapolate if the object is not really grabbed
    if (grabbable.currentGrabber == null) return;
    NetworkGrabber networkGrabber = grabbable.currentGrabber.networkGrabber;

    // Extrapolation: Make visual representation follow grabber, adding position/rotation offsets
    // We use grabberWhileTakingAuthority instead of CurrentGrabber as we are currently waiting for the authority transfer: the network vars are not already set, so we use the temporary versions
    grabbable.Follow(followingtransform: networkTransform.InterpolationTarget.transform, followedTransform: networkGrabber.hand.networkTransform.InterpolationTarget.transform, grabbable.localPositionOffset, grabbable.localRotationOffset);
}

Differences with the basic VR Shared sample grabbing logic
Downloads
Overview
Details

Grabbing
Grabbing synchronization
Follow
Render