Implementing interaction in an imported animation (FBX)

Hi Unity-community,

I am fairly new here, and I have started playing around in making my own VR-environments for my Oculus Quest 2 through Unity 6. I am sort of surprised how far I’ve come by just using ChatGPT without any knowledge of C# or Unity a week ago. But now I have encountered issues as ChatGPT seems to be trained on outdated versions of Unity and SDKs.

I have exported an animation with moving objects through Blender in an FBX-file and I have managed to import this, and get an animation going through the timeline (via an Animator to the entire imported FBX-scene and a Playable Director). All is working well and I can step into my animated VR-scene of moving objects in my Oculus-set.

Now I would like to add interaction to my app, i.e. I want to be able to grab and throw around the moving objects in my scene and thereby stopping them from being animated.

Disabling the Animator of the imported FBX-scene, I am able to grab and throw objects by assigning a RigidBody, Box Collider, a Grab Interactable and a Grabbable script to each object. But I have not managed to get it working with the Animator active as it seems to override all interactions.

I understand that I somehow probably need to assign some kind of script to the object such that it falls out of the hierarchy of the animated scene upon grabbing the object. But ChatGPT has not been able to solve this for me.

I am currently using the Meta XR All-in-one SDK (using the OVRCameraRigInteraction) and the XR Plug-in.

Can someone here assist me with this issue?

That’s exactly what you need to do… but know that when you put it BACK into that animated hierarchy, it will immediately snap to wherever it would have been if you had left it.

If you want it to keep moving relatively from wherever you hand-moved it to, that’s a bit trickier.

One approach is to animate only proxy objects, then have a script that every frame uses their animated offset and moves the connected object.

That proxy script would do so until such time as you grab one of those objects, then you take it over.

When you let the grabbed object go, it could resume being “driven” by remote movements from the animated proxy objects.

Hi Kurt-Dekker,

and thank you so much for the quick response!

I am actually completely fine with the objects falling out of the animation and never coming back to the animated state unless the app is restarted.

Do you know how I can implement this?

Generally, you just have to call the .SetParent() method on the Transform of the object you grabbed.

// removes parenting, sets the GrabbedObject to be at the root of the scene:
     GrabbedObject.transform.SetParent(null);

Pass in the Transform reference you want to parent it to if you want to actually put it in some other hierarchy.

In related news, the easiest cheesiest way to put something in your hand is to call .SetParent() and pass in the Transform of your hand or controller.

Hi Kurt-Dekker,

Thank you, that sounds easy enough :slight_smile:

However, I am sorry for being on a very Beginner-level here. I am wondering where I place this line of code? Do I create a new component/script to the object or do I add that somewhere in the Grabbable or GrabInteractable scripts?

For reference, ChatGPTs solution is to create a new script called “DetachOnGrab”. But, I tried this and it does not work. The object is still not grabbable when the Animator is active.

using UnityEngine;

public class DetachOnGrab : MonoBehaviour
{
    private Rigidbody rb;
    private Animator animator;

    void Start()
    {
        // Cache references
        rb = GetComponent<Rigidbody>();
        animator = GetComponent<Animator>();

        if (rb == null)
        {
            Debug.LogError($"No Rigidbody found on {gameObject.name}. Please add one!");
        }

        if (animator == null)
        {
            Debug.LogWarning($"No Animator found on {gameObject.name}. Grabbing will work without animations.");
        }
    }

    public void OnGrabStart()
    {
        Debug.Log($"{gameObject.name} grabbed. Permanently detaching from parent.");

        // Detach the object from its parent (permanently)
        transform.SetParent(null);

        // Enable physics for grabbing
        if (rb != null)
        {
            rb.isKinematic = false;
        }

        // Disable the Animator to stop animations
        if (animator != null)
        {
            animator.enabled = false;
        }
    }

    public void OnGrabEnd()
    {
        Debug.Log($"{gameObject.name} released. Physics remains active.");

        // Physics remains active after release
        if (rb != null)
        {
            rb.isKinematic = false; // Keeps physics enabled
        }

        // Animator remains disabled permanently
        if (animator != null)
        {
            animator.enabled = false;
        }
    }
}

Looks reasonable (see line 30 where it does the deparenting?), but is anything calling those methods?

Find out by debugging… debugging is how you take your learning about code to the next level.

By debugging you can find out exactly what your program is doing so you can fix it.

Use the above techniques to get the information you need in order to reason about what the problem is.

You can also use Debug.Log(...); statements to find out if any of your code is even running. Don’t assume it is.

Once you understand what the problem is, you may begin to reason about a solution to the problem.

Remember with Unity the code (aka “scripts”) is only a tiny fraction of the problem space. Everything asset- and scene- wise must also be set up correctly to match the associated code and its assumptions.

Thanks, it will take me while though to learn how to do proper debugging given my current skill-level.

Ok, let’s say that there is something that I’ve done in my “problem space”. Given a complete blank project and an animated FBX. How would you (or anyone else in the community seeing this thread) setup this app?

If there is a video tutorial or related thread, that would be great.

I’ll attach a screenshot of the settings if that would be helpful to see what I might have done wrong.

There’s really no point in learning how to code if you’re not going to learn also how to set stuff up in Unity.

Code is great, code is fine, if it’s not connected PERFECTLY it is worthless.

Apps are not set up by dogma, otherwise programmers would no longer be necessary.

Apps are set up by proper engineering, which involves understanding the problem and selecting a solution and implementing it.

Wow, that was kind of rude…

I thought this was a trivial problem for people with vast experience in VR app-development and I just wanted to reach out to the experts for help. I thought implementations of grabbing objects was basic knowledge for you guys, so I thought there was a quick fix for this particular issue. But it is ok to say that what I want to do is more complicated. If so, I will leave this little fun endeavor into the VR world. I never intended to be a VR-developer, I was just playing around.

If the answer is “sit down, learn to code C# and educate yourself to be an app-developer, then you will find a solution” it is not very helpful for me. Unity is a quite complicated program to learn from scratch.

I guess this forum was only for experts and not beginners like me …

For a someone who has gone so far with ChatGPT, it really disappoints me to see you write this.

I’m simply pointing out that not ALL can be learned from ChatGPT.

There’s no way around that.

If you want to take that point and instead twist it into this:

and this:

Then I’m afraid I have failed to reach you, which is again disappointing because you seem so enthusiastic about ChatGPT.

Best of luck in your journeys.

I’m sorry if I hit a nerve by writing that ChatGPT got me to this state. I am wondering if I would have received different advice if I kept that part to myself and just asked for help to setup the problem from scratch.

The ironic thing is that the reason I am even here is because of the issues you’re pointing out and I am well-aware of. I am an engineer myself, which is why I probably get sensitive getting a lecture about engineering late at night. Sorry for that and for the tone in my previous response. I am now getting used to utilizing ChatGPT for a variety of engineering tasks and seeing how far I can go. I know that ChatGPT can provide me with tools and building blocks, but I know that there is a limit to what it can build. It has a hard time building a proper boat, but something that will float and that will draw you quite far out in the sea. So when it fails, you are quite helpless and ill-equipped in getting further.

I reached out here EXACTLY for that particular reason. I reached a dead-end with a seemingly trivial task, and ChatGPT could not solve it as it keeps getting giving me solutions using depreciated functions with erroneous attributes. Or perhaps I am asking the wrong questions. I wanted to see if there was a way through the expert-community to solve this, but I understand that this feels like explaining quantum physics to a toddler.

It still does not take away the fact that ChatGPT is there now and can help people like me to get over the hurdle that is, for example, learning a new software. I would not have dreamed before of being able to create something that works for simple visualizations in my Oculus headset. I am 100% sure that tools like ChatGPT will be very useful for engineers in the future. It will allow engineers to be much better equipped to take on complex tasks that would otherwise be overwhelming. Engineers will need to evolve to this reality, but I do not think new technological tools will replace engineers for all the aforementioned reasons.

With that said, I am sorry if my approach has offended anyone here. Using the previous analogy, I understand that it is frustrating trying to explain boat-building to someone drowning far out at sea because they built a half-ass floating device. I probably would react the same way with people coming into my area of expertise using ChatGPT.

I will continue my prompt-engineering adventure with ChatGPT to find a workaround and debug the program whenever I find the time. If I find the solution, I’ll make sure to post it here to help others with similar issues.

Thank you anyway for quickly responding to my queries!

So, I managed to solve it after a lot of back-and-forth-debugging using ChatGPT. For anyone interested in a similar problem, here is my solution from scratch:

  1. Add the Meta → Tools → Building Blocks → Grab Interaction to create a camera rig and controllers with built-in grabbing functionality.

  2. Add your FBX-scene (unpack if necessary such that particles can be moved up in the hierarchy). Make sure it is imported with animations. Add an Animator with a custom FBX-controller (not entirely sure if this controller was necessary).

  3. Create an empty GameObject, add a Playable Director and create a timeline file in the project folder. In the timeline-window, drag the FBX-object from the hierarchy to the timeline window and click Add Animation track and choose the Scene from the FBX.

  4. Choose an object within the FBX, right-click and choose Interaction SDK → Add Grab Interaction. This will create a child to the object including the Grabbable, Grab Interactable and Hand Grab Interactable scripts to the object. It will also auto-generate a Collider for the object and a Rigidbody.

  5. To the object, add Component → New Script and create two new scripts (I will post them below):

  • Detach On Grab
  • Grab Interaction Handler

Run the scene. Objects are now running in the animation, but upon grabbing they fall out of the animation and become regular grabbable objects in the scene.

To most in the forum, this would probably be a trivial thing to setup, but I want to help any beginners of Unity that might stumble upon this post.

Custom scripts:

using UnityEngine;
using Oculus.Interaction;

public class GrabInteractionHandler : MonoBehaviour
{
    private GrabInteractable grabInteractable; // For detecting grab interactions
    private DetachOnGrab detachOnGrab;        // Reference to DetachOnGrab script
    private Transform parentObject;           // The parent object with Rigidbody and Collider
    private bool isGrabbed = false;           // Tracks grab state

    private void Start()
    {
        // Get the GrabInteractable component
        grabInteractable = GetComponent<GrabInteractable>();
        if (grabInteractable == null)
        {
            Debug.LogError("[GrabInteractionHandler] GrabInteractable component missing on this GameObject.");
            return;
        }

        // Get the DetachOnGrab component
        detachOnGrab = GetComponent<DetachOnGrab>();
        if (detachOnGrab == null)
        {
            Debug.LogError("[GrabInteractionHandler] DetachOnGrab script missing on this GameObject.");
            return;
        }

        // Find the parent object with Rigidbody and Collider
        parentObject = transform.parent;
        if (parentObject == null)
        {
            Debug.LogError("[GrabInteractionHandler] No parent object found! Ensure this GameObject is a child.");
            return;
        }

        // Subscribe to grab and release events
        grabInteractable.WhenPointerEventRaised += HandlePointerEvent;
    }

    private void OnDestroy()
    {
        // Unsubscribe from grab and release events
        if (grabInteractable != null)
        {
            grabInteractable.WhenPointerEventRaised -= HandlePointerEvent;
        }
    }

    private void HandlePointerEvent(PointerEvent pointerEvent)
    {
        if (pointerEvent.Type == PointerEventType.Select)
        {
            HandleGrabStart();
        }
        else if (pointerEvent.Type == PointerEventType.Unselect)
        {
            HandleGrabEnd();
        }
    }

    private void HandleGrabStart()
    {
        Debug.Log("[GrabInteractionHandler] Object grabbed.");
        detachOnGrab?.OnGrabStart();
    }

    private void HandleGrabEnd()
    {
        Debug.Log("[GrabInteractionHandler] Object released.");
        detachOnGrab?.OnGrabEnd();
    }
}

using UnityEngine;
using Oculus.Interaction;

public class DetachOnGrab : MonoBehaviour
{
    private Transform parentObject;       // The parent object with Rigidbody and Collider
    private Transform originalParent;    // Original parent in the FBX hierarchy
    private Rigidbody rb;                // Reference to Rigidbody
    private bool isDetached = false;     // Ensure detachment happens only once
    private bool isGrabbed = false;      // Tracks if the object is currently grabbed

    private void Start()
    {
        // Reference the parent object
        parentObject = transform.parent;
        if (parentObject == null)
        {
            Debug.LogError("[DetachOnGrab] Parent object is null! Ensure this script is placed on the child with grab components.");
            return;
        }

        // Store the original parent (FBX hierarchy)
        originalParent = parentObject.parent;

        // Get the Rigidbody component
        rb = parentObject.GetComponent<Rigidbody>();
        if (rb == null)
        {
            Debug.LogError("[DetachOnGrab] Rigidbody missing on parent object.");
        }
    }

    public void OnGrabStart()
    {
        Debug.Log("[DetachOnGrab] OnGrabStart called.");

        if (!isDetached)
        {
            DetachParent();
        }

        // Ensure the object follows the controller
        isGrabbed = true;

        // Disable Animator control for this object
        RemoveFromAnimator();

        // Disable physics while grabbed
        if (rb != null)
        {
            rb.isKinematic = true;
            rb.linearVelocity = Vector3.zero;
            rb.angularVelocity = Vector3.zero;
        }

        Debug.Log("[DetachOnGrab] Object grabbed and follows controller.");
    }

    public void OnGrabEnd()
    {
        Debug.Log("[DetachOnGrab] OnGrabEnd called.");

        // Stop following the controller
        isGrabbed = false;

        // Ensure the object stays in place
        if (rb != null)
        {
            rb.isKinematic = false; // Re-enable physics
        }

        Debug.Log("[DetachOnGrab] Object stays in place after release.");
    }

    private void DetachParent()
    {
        if (parentObject != null)
        {
            parentObject.parent = null; // Detach from FBX hierarchy
            isDetached = true;

            Debug.Log($"[DetachOnGrab] Parent object detached: {parentObject.name}");
        }
    }

    private void RemoveFromAnimator()
    {
        // Ensure the Animator no longer controls this object
        var animator = originalParent.GetComponent<Animator>();
        if (animator != null)
        {
            animator.Rebind(); // Forces the Animator to recalculate hierarchy bindings
        }

        Debug.Log("[DetachOnGrab] Removed object from Animator control.");
    }

    private void LateUpdate()
    {
        // Ensure the object follows the controller while grabbed
        if (isGrabbed && rb.isKinematic)
        {
            parentObject.position = transform.position; // Match the child object's transform
            parentObject.rotation = transform.rotation;
        }
    }
}

A few screenshots of my setup: