Human Body Tracking Model

Unity Engine
,
1

Hi, does anyone successfully replaced the default robot model on AR Foundation 4.x?

I’ve been trying to download the biped-robot.zip provided by Apple from this page Rigging a Model for Motion Capture | Apple Developer Documentation
Then I compared .fbx file into the prefab on AR Foundation.

I tried directly importing the .fbx file without any changes on it.
Load the BoneController component into it, but it produce weird view.

I read on some post and articles that I must account the rotation and position of my rig with the prefab’s.
Has anyone got successfully made changes to the default robot?

As I read post and comments from here Example Rig for 3D Human Skeleton @distastee has made successful rigging. I tried his code but I cant seem to make it work.
Maybe I am missing something?

Please I need your advice.

Thank you so much.

2

Have you taken a look at the AR Foundation sample for 3D human tracking? Please be aware that the model used here is not a MechAnim model. It has a bone and joint structure that adheres to Apple’s spec.

https://github.com/Unity-Technologies/arfoundation-samples#humanbodytracking3d

3

Yes I already looked and analyzed it.
From what I observed, the prefab model in the AR Foundation (right) has different joint directions from the fbx file released by the apple (https://developer.apple.com/documentation/arkit/rigging_a_model_for_motion_capture , left)

As far as what I understand from his post, he said that i must compare my T-pose to T-pose of default prefab and use the offset when updating the transform later.

Here is my code.

using UnityEngine.XR.ARFoundation;
using UnityEngine;
using UnityEngine.XR.ARSubsystems;
using System.Collections.Generic;
using System;

namespace UnityEngine.XR.ARFoundation.Samples
{
    public class CustomBoneController : MonoBehaviour
    {
        // 3D joint skeleton
        enum JointIndices
        {
            Invalid = -1,
            root = 0, // parent: <none> [-1]
            hips_joint = 1, // parent: Root [0]
            left_upLeg_joint = 2, // parent: Hips [1]
            left_leg_joint = 3, // parent: LeftUpLeg [2]
            left_foot_joint = 4, // parent: LeftLeg [3]
            left_toes_joint = 5, // parent: LeftFoot [4]
            left_toesEnd_joint = 6, // parent: LeftToes [5]
            right_upLeg_joint = 7, // parent: Hips [1]
            right_leg_joint = 8, // parent: RightUpLeg [7]
            right_foot_joint = 9, // parent: RightLeg [8]
            right_toes_joint = 10, // parent: RightFoot [9]
            right_toesEnd_joint = 11, // parent: RightToes [10]
            spine_1_joint = 12, // parent: Hips [1]
            spine_2_joint = 13, // parent: Spine1 [12]
            spine_3_joint = 14, // parent: Spine2 [13]
            spine_4_joint = 15, // parent: Spine3 [14]
            spine_5_joint = 16, // parent: Spine4 [15]
            spine_6_joint = 17, // parent: Spine5 [16]
            spine_7_joint = 18, // parent: Spine6 [17]
            left_shoulder_1_joint = 19, // parent: Spine7 [18]
            left_arm_joint = 20, // parent: LeftShoulder1 [19]
            left_forearm_joint = 21, // parent: LeftArm [20]
            left_hand_joint = 22, // parent: LeftForearm [21]
            left_handIndexStart_joint = 23, // parent: LeftHand [22]
            left_handIndex_1_joint = 24, // parent: LeftHandIndexStart [23]
            left_handIndex_2_joint = 25, // parent: LeftHandIndex1 [24]
            left_handIndex_3_joint = 26, // parent: LeftHandIndex2 [25]
            left_handIndexEnd_joint = 27, // parent: LeftHandIndex3 [26]
            left_handMidStart_joint = 28, // parent: LeftHand [22]
            left_handMid_1_joint = 29, // parent: LeftHandMidStart [28]
            left_handMid_2_joint = 30, // parent: LeftHandMid1 [29]
            left_handMid_3_joint = 31, // parent: LeftHandMid2 [30]
            left_handMidEnd_joint = 32, // parent: LeftHandMid3 [31]
            left_handPinkyStart_joint = 33, // parent: LeftHand [22]
            left_handPinky_1_joint = 34, // parent: LeftHandPinkyStart [33]
            left_handPinky_2_joint = 35, // parent: LeftHandPinky1 [34]
            left_handPinky_3_joint = 36, // parent: LeftHandPinky2 [35]
            left_handPinkyEnd_joint = 37, // parent: LeftHandPinky3 [36]
            left_handRingStart_joint = 38, // parent: LeftHand [22]
            left_handRing_1_joint = 39, // parent: LeftHandRingStart [38]
            left_handRing_2_joint = 40, // parent: LeftHandRing1 [39]
            left_handRing_3_joint = 41, // parent: LeftHandRing2 [40]
            left_handRingEnd_joint = 42, // parent: LeftHandRing3 [41]
            left_handThumbStart_joint = 43, // parent: LeftHand [22]
            left_handThumb_1_joint = 44, // parent: LeftHandThumbStart [43]
            left_handThumb_2_joint = 45, // parent: LeftHandThumb1 [44]
            left_handThumbEnd_joint = 46, // parent: LeftHandThumb2 [45]
            neck_1_joint = 47, // parent: Spine7 [18]
            neck_2_joint = 48, // parent: Neck1 [47]
            neck_3_joint = 49, // parent: Neck2 [48]
            neck_4_joint = 50, // parent: Neck3 [49]
            head_joint = 51, // parent: Neck4 [50]
            jaw_joint = 52, // parent: Head [51]
            chin_joint = 53, // parent: Jaw [52]
            left_eye_joint = 54, // parent: Head [51]
            left_eyeLowerLid_joint = 55, // parent: LeftEye [54]
            left_eyeUpperLid_joint = 56, // parent: LeftEye [54]
            left_eyeball_joint = 57, // parent: LeftEye [54]
            nose_joint = 58, // parent: Head [51]
            right_eye_joint = 59, // parent: Head [51]
            right_eyeLowerLid_joint = 60, // parent: RightEye [59]
            right_eyeUpperLid_joint = 61, // parent: RightEye [59]
            right_eyeball_joint = 62, // parent: RightEye [59]
            right_shoulder_1_joint = 63, // parent: Spine7 [18]
            right_arm_joint = 64, // parent: RightShoulder1 [63]
            right_forearm_joint = 65, // parent: RightArm [64]
            right_hand_joint = 66, // parent: RightForearm [65]
            right_handIndexStart_joint = 67, // parent: RightHand [66]
            right_handIndex_1_joint = 68, // parent: RightHandIndexStart [67]
            right_handIndex_2_joint = 69, // parent: RightHandIndex1 [68]
            right_handIndex_3_joint = 70, // parent: RightHandIndex2 [69]
            right_handIndexEnd_joint = 71, // parent: RightHandIndex3 [70]
            right_handMidStart_joint = 72, // parent: RightHand [66]
            right_handMid_1_joint = 73, // parent: RightHandMidStart [72]
            right_handMid_2_joint = 74, // parent: RightHandMid1 [73]
            right_handMid_3_joint = 75, // parent: RightHandMid2 [74]
            right_handMidEnd_joint = 76, // parent: RightHandMid3 [75]
            right_handPinkyStart_joint = 77, // parent: RightHand [66]
            right_handPinky_1_joint = 78, // parent: RightHandPinkyStart [77]
            right_handPinky_2_joint = 79, // parent: RightHandPinky1 [78]
            right_handPinky_3_joint = 80, // parent: RightHandPinky2 [79]
            right_handPinkyEnd_joint = 81, // parent: RightHandPinky3 [80]
            right_handRingStart_joint = 82, // parent: RightHand [66]
            right_handRing_1_joint = 83, // parent: RightHandRingStart [82]
            right_handRing_2_joint = 84, // parent: RightHandRing1 [83]
            right_handRing_3_joint = 85, // parent: RightHandRing2 [84]
            right_handRingEnd_joint = 86, // parent: RightHandRing3 [85]
            right_handThumbStart_joint = 87, // parent: RightHand [66]
            right_handThumb_1_joint = 88, // parent: RightHandThumbStart [87]
            right_handThumb_2_joint = 89, // parent: RightHandThumb1 [88]
            right_handThumbEnd_joint = 90, // parent: RightHandThumb2 [89]
        }
        const int k_NumSkeletonJoints = 91;

        [SerializeField]
        Transform[] m_BoneMapping = new Transform[k_NumSkeletonJoints];

        [SerializeField]
        private const float jointScaleModifier = 1f;

        [SerializeField]
        private Dictionary<JointIndices, Quaternion> ROBOTPOSE = new Dictionary<JointIndices, Quaternion>
        {
            //rotation
           
            { JointIndices.root, new Quaternion(0f, 0f, 0f, 1f) },
            { JointIndices.hips_joint, new Quaternion(0f, 0f, 0f, 1f) },
            { JointIndices.left_upLeg_joint, new Quaternion(-0.5251914f, 0.4729119f, 0.5257655f, -0.4733909f) },
            { JointIndices.left_leg_joint, new Quaternion(-0.4589739f, 0.5374146f, 0.4594831f, -0.5379627f) },
            { JointIndices.left_foot_joint, new Quaternion(-0.6927235f, 0.1400152f, 0.6934642f, -0.1401326f) },
            { JointIndices.left_toes_joint, new Quaternion(-0.7067051f, 0.006181896f, 0.7074544f, -0.006156165f) },
            { JointIndices.left_toesEnd_joint, new Quaternion(-0.7067052f, 0.006181717f, 0.7074544f, -0.006156351f) },
            { JointIndices.right_upLeg_joint, new Quaternion(-0.4731499f, -0.5254809f, 0.4731455f, 0.5254829f) },
            { JointIndices.right_leg_joint, new Quaternion(-0.5376937f, -0.4592238f, 0.5376896f, 0.4592263f) },
            { JointIndices.right_foot_joint, new Quaternion(-0.1400807f, -0.6930928f, 0.1400763f, 0.6930935f) },
            { JointIndices.right_toes_joint, new Quaternion(-0.006176293f, -0.7070798f, 0.00617099f, 0.7070799f) },
            { JointIndices.right_toesEnd_joint, new Quaternion(-0.00617677f, -0.7070798f, 0.006170507f, 0.7070799f) },
            { JointIndices.spine_1_joint, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },
            { JointIndices.spine_2_joint, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },
            { JointIndices.spine_3_joint, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },
            { JointIndices.spine_4_joint, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },
            { JointIndices.spine_5_joint, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },
            { JointIndices.spine_6_joint, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },
            { JointIndices.spine_7_joint, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },
            { JointIndices.left_shoulder_1_joint, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },
            { JointIndices.left_arm_joint, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },
            { JointIndices.left_forearm_joint, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },
            { JointIndices.left_hand_joint, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },
            { JointIndices.left_handIndexStart_joint, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },
            { JointIndices.left_handIndex_1_joint, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },
            { JointIndices.left_handIndex_2_joint, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },
            { JointIndices.left_handIndex_3_joint, new Quaternion(-0.5000001f, -0.5000001f, 0.5000001f, 0.5000001f) },
            { JointIndices.left_handIndexEnd_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
            { JointIndices.left_handMidStart_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
            { JointIndices.left_handMid_1_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
            { JointIndices.left_handMid_2_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
            { JointIndices.left_handMid_3_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
            { JointIndices.left_handMidEnd_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
            { JointIndices.left_handPinkyStart_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
            { JointIndices.left_handPinky_1_joint, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },
            { JointIndices.left_handPinky_2_joint, new Quaternion(-0.7004818f, -0.08645806f, 0.09656784f, 0.7018013f) },
            { JointIndices.left_handPinky_3_joint, new Quaternion(-0.7072248f, -0.0002385676f, -0.0009994805f, 0.706988f) },
            { JointIndices.left_handPinkyEnd_joint, new Quaternion(-0.7065828f, -0.03009519f, -0.03084728f, 0.7063168f) },
            { JointIndices.left_handRingStart_joint, new Quaternion(-0.0001880527f, -0.04309286f, -0.0005318075f, 0.9990709f) },
            { JointIndices.left_handRing_1_joint, new Quaternion(0.002059653f, 0.06050844f, 0.03457659f, 0.9975666f) },
            { JointIndices.left_handRing_2_joint, new Quaternion(0.002059653f, 0.06050844f, 0.03457659f, 0.9975666f) },
            { JointIndices.left_handRing_3_joint, new Quaternion(0.002059653f, 0.06050844f, 0.03457659f, 0.9975666f) },
            { JointIndices.left_handRingEnd_joint, new Quaternion(0.002059653f, 0.06050844f, 0.03457659f, 0.9975666f) },
            { JointIndices.left_handThumbStart_joint, new Quaternion(0.002059653f, 0.06050844f, 0.03457659f, 0.9975666f) },
            { JointIndices.left_handThumb_1_joint, new Quaternion(7.659197E-05f, 0.008220375f, 0.01719604f, 0.9998186f) },
            { JointIndices.left_handThumb_2_joint, new Quaternion(7.659197E-05f, 0.008220375f, 0.01719604f, 0.9998186f) },
            { JointIndices.left_handThumbEnd_joint, new Quaternion(7.659197E-05f, 0.008220375f, 0.01719604f, 0.9998186f) },
            { JointIndices.neck_1_joint, new Quaternion(7.659197E-05f, 0.008220375f, 0.01719604f, 0.9998186f) },
            { JointIndices.neck_2_joint, new Quaternion(7.659197E-05f, 0.008220375f, 0.01719604f, 0.9998186f) },
            { JointIndices.neck_3_joint, new Quaternion(-8.016825E-05f, -0.04412498f, -0.0002519935f, 0.9990261f) },
            { JointIndices.neck_4_joint, new Quaternion(-8.016825E-05f, -0.04412498f, -0.0002519935f, 0.9990261f) },
            { JointIndices.head_joint, new Quaternion(-8.016825E-05f, -0.04412498f, -0.0002519935f, 0.9990261f) },
            { JointIndices.jaw_joint, new Quaternion(-8.016825E-05f, -0.04412498f, -0.0002519935f, 0.9990261f) },
            { JointIndices.chin_joint, new Quaternion(-8.016825E-05f, -0.04412498f, -0.0002519935f, 0.9990261f) },
            { JointIndices.left_eye_joint, new Quaternion(-0.0001313686f, -0.111419f, 0.0001013279f, 0.9937735f) },
            { JointIndices.left_eyeball_joint, new Quaternion(-0.0001313686f, -0.111419f, 0.0001013279f, 0.9937735f) },
            { JointIndices.left_eyeLowerLid_joint, new Quaternion(-0.0001313686f, -0.111419f, 0.0001013279f, 0.9937735f) },
            { JointIndices.left_eyeUpperLid_joint, new Quaternion(-0.0001313686f, -0.111419f, 0.0001013279f, 0.9937735f) },
            { JointIndices.nose_joint, new Quaternion(-0.0001313686f, -0.111419f, 0.0001013279f, 0.9937735f) },
            { JointIndices.right_eye_joint, new Quaternion(-0.6163417f, -0.3677837f, -0.1319845f, 0.6836949f) },
            { JointIndices.right_eyeball_joint, new Quaternion(-0.6163417f, -0.3677837f, -0.1319845f, 0.6836949f) },
            { JointIndices.right_eyeLowerLid_joint, new Quaternion(-0.6163417f, -0.3677837f, -0.1319845f, 0.6836949f) },
            { JointIndices.right_eyeUpperLid_joint, new Quaternion(-0.6163418f, -0.3677837f, -0.1319845f, 0.6836948f) },
            { JointIndices.right_shoulder_1_joint, new Quaternion(-0.7017999f, 0.09655666f, 0.08646959f, -0.7004833f) },
            { JointIndices.right_arm_joint, new Quaternion(-0.7069917f, -0.0005104197f, -0.0002469169f, -0.7072219f) },
            { JointIndices.right_forearm_joint, new Quaternion(-0.7062967f, -0.03134356f, 0.0305951f, -0.7065598f) },
            { JointIndices.right_hand_joint, new Quaternion(-0.9990404f, -0.0005291998f, 0.04379734f, -0.0001842082f) },
            { JointIndices.right_handIndexStart_joint, new Quaternion(-0.9973709f, 0.03453366f, -0.06367449f, 0.002123177f) },
            { JointIndices.right_handIndex_1_joint, new Quaternion(-0.9973709f, 0.03453366f, -0.06367449f, 0.002123177f) },
            { JointIndices.right_handIndex_2_joint, new Quaternion(-0.9973709f, 0.03453366f, -0.06367449f, 0.002123177f) },
            { JointIndices.right_handIndex_3_joint, new Quaternion(-0.9973709f, 0.03453366f, -0.06367449f, 0.002123177f) },
            { JointIndices.right_handIndexEnd_joint, new Quaternion(-0.9973709f, 0.03453366f, -0.06367449f, 0.002123177f) },
            { JointIndices.right_handMidStart_joint, new Quaternion(-0.999788f, 0.01716189f, -0.01139313f, 8.320808E-05f) },
            { JointIndices.right_handMid_1_joint, new Quaternion(-0.999788f, 0.01716189f, -0.01139313f, 8.320808E-05f) },
            { JointIndices.right_handMid_2_joint, new Quaternion(-0.999788f, 0.01716189f, -0.01139313f, 8.320808E-05f) },
            { JointIndices.right_handMid_3_joint, new Quaternion(-0.999788f, 0.01716189f, -0.01139313f, 8.320808E-05f) },
            { JointIndices.right_handMidEnd_joint, new Quaternion(-0.999788f, 0.01716189f, -0.01139313f, 8.320808E-05f) },
            { JointIndices.right_handPinkyStart_joint, new Quaternion(-0.999161f, -0.0002829432f, 0.04095532f, -0.000130713f) },
            { JointIndices.right_handPinky_1_joint, new Quaternion(-0.999161f, -0.0002829432f, 0.04095532f, -0.000130713f) },
            { JointIndices.right_handPinky_2_joint, new Quaternion(-0.999161f, -0.0002829432f, 0.04095532f, -0.000130713f) },
            { JointIndices.right_handPinky_3_joint, new Quaternion(-0.999161f, -0.0002829432f, 0.04095532f, -0.000130713f) },
            { JointIndices.right_handPinkyEnd_joint, new Quaternion(-0.999161f, -0.0002829432f, 0.04095532f, -0.000130713f) },
            { JointIndices.right_handRingStart_joint, new Quaternion(-0.9941221f, 7.390976E-05f, 0.1082657f, -0.0001827627f) },
            { JointIndices.right_handRing_1_joint, new Quaternion(-0.9941221f, 7.390976E-05f, 0.1082657f, -0.0001827627f) },
            { JointIndices.right_handRing_2_joint, new Quaternion(-0.9941221f, 7.390976E-05f, 0.1082657f, -0.0001827627f) },
            { JointIndices.right_handRing_3_joint, new Quaternion(-0.9941221f, 7.390976E-05f, 0.1082657f, -0.0001827627f) },
            { JointIndices.right_handRingEnd_joint, new Quaternion(-0.9941221f, 7.390976E-05f, 0.1082657f, -0.0001827627f) },
            { JointIndices.right_handThumbStart_joint, new Quaternion(-0.6848241f, -0.1300335f, 0.3655986f, -0.6168025f) },
            { JointIndices.right_handThumb_1_joint, new Quaternion(-0.6848241f, -0.1300335f, 0.3655986f, -0.6168025f) },
            { JointIndices.right_handThumb_2_joint, new Quaternion(-0.6848241f, -0.1300335f, 0.3655986f, -0.6168026f) },
            { JointIndices.right_handThumbEnd_joint, new Quaternion(-0.684824f, -0.1300335f, 0.3655986f, -0.6168026f) }
           
        };

        [SerializeField]
        Dictionary<JointIndices, Transform> MyRigJoints = new Dictionary<JointIndices, Transform>
        {
            {JointIndices.root, null},
            {JointIndices.hips_joint, null},
            {JointIndices.left_upLeg_joint, null},
            {JointIndices.left_leg_joint, null},
            {JointIndices.left_foot_joint, null},
            {JointIndices.left_toes_joint, null},
            {JointIndices.left_toesEnd_joint, null},
            {JointIndices.right_upLeg_joint, null},
            {JointIndices.right_leg_joint, null},
            {JointIndices.right_foot_joint, null},
            {JointIndices.right_toes_joint, null},
            {JointIndices.right_toesEnd_joint, null},
            {JointIndices.spine_1_joint, null},
            {JointIndices.spine_2_joint, null},
            {JointIndices.spine_3_joint, null},
            {JointIndices.spine_4_joint, null},
            {JointIndices.spine_5_joint, null},
            {JointIndices.spine_6_joint, null},
            {JointIndices.spine_7_joint, null},
            {JointIndices.left_shoulder_1_joint, null},
            {JointIndices.left_arm_joint, null},
            {JointIndices.left_forearm_joint, null},
            {JointIndices.left_hand_joint, null},
            {JointIndices.left_handIndexStart_joint, null},
            {JointIndices.left_handIndex_1_joint, null},
            {JointIndices.left_handIndex_2_joint, null},
            {JointIndices.left_handIndex_3_joint, null},
            {JointIndices.left_handIndexEnd_joint, null},
            {JointIndices.left_handMidStart_joint, null},
            {JointIndices.left_handMid_1_joint, null},
            {JointIndices.left_handMid_2_joint, null},
            {JointIndices.left_handMid_3_joint, null},
            {JointIndices.left_handMidEnd_joint, null},
            {JointIndices.left_handPinkyStart_joint, null},
            {JointIndices.left_handPinky_1_joint, null},
            {JointIndices.left_handPinky_2_joint, null},
            {JointIndices.left_handPinky_3_joint, null},
            {JointIndices.left_handPinkyEnd_joint, null},
            {JointIndices.left_handRingStart_joint, null},
            {JointIndices.left_handRing_1_joint, null},
            {JointIndices.left_handRing_2_joint, null},
            {JointIndices.left_handRing_3_joint, null},
            {JointIndices.left_handRingEnd_joint, null},
            {JointIndices.left_handThumbStart_joint, null},
            {JointIndices.left_handThumb_1_joint, null},
            {JointIndices.left_handThumb_2_joint, null},
            {JointIndices.left_handThumbEnd_joint, null},
            {JointIndices.neck_1_joint, null},
            {JointIndices.neck_2_joint, null},
            {JointIndices.neck_3_joint, null},
            {JointIndices.neck_4_joint, null},
            {JointIndices.head_joint, null},
            {JointIndices.jaw_joint, null},
            {JointIndices.chin_joint, null},
            {JointIndices.left_eye_joint, null},
            {JointIndices.left_eyeball_joint, null},
            {JointIndices.left_eyeLowerLid_joint, null},
            {JointIndices.left_eyeUpperLid_joint, null},
            {JointIndices.nose_joint, null},
            {JointIndices.right_eye_joint, null},
            {JointIndices.right_eyeball_joint, null},
            {JointIndices.right_eyeLowerLid_joint, null},
            {JointIndices.right_eyeUpperLid_joint, null},
            {JointIndices.right_shoulder_1_joint, null},
            {JointIndices.right_arm_joint, null},
            {JointIndices.right_forearm_joint, null},
            {JointIndices.right_hand_joint, null},
            {JointIndices.right_handIndexStart_joint, null},
            {JointIndices.right_handIndex_1_joint, null},
            {JointIndices.right_handIndex_2_joint, null},
            {JointIndices.right_handIndex_3_joint, null},
            {JointIndices.right_handIndexEnd_joint, null},
            {JointIndices.right_handMidStart_joint, null},
            {JointIndices.right_handMid_1_joint, null},
            {JointIndices.right_handMid_2_joint, null},
            {JointIndices.right_handMid_3_joint, null},
            {JointIndices.right_handMidEnd_joint, null},
            {JointIndices.right_handPinkyStart_joint, null},
            {JointIndices.right_handPinky_1_joint, null},
            {JointIndices.right_handPinky_2_joint, null},
            {JointIndices.right_handPinky_3_joint, null},
            {JointIndices.right_handPinkyEnd_joint, null},
            {JointIndices.right_handRingStart_joint, null},
            {JointIndices.right_handRing_1_joint, null},
            {JointIndices.right_handRing_2_joint, null},
            {JointIndices.right_handRing_3_joint, null},
            {JointIndices.right_handRingEnd_joint, null},
            {JointIndices.right_handThumbStart_joint, null},
            {JointIndices.right_handThumb_1_joint, null},
            {JointIndices.right_handThumb_2_joint, null},
            {JointIndices.right_handThumbEnd_joint, null}
        };

        [SerializeField]
        private Quaternion[] RotationCompensations = new Quaternion[k_NumSkeletonJoints];
    

        [SerializeField]
        [Tooltip("The root bone of the skeleton.")]
        Transform m_SkeletonRoot;

        /// <summary>
        /// Get/Set the root bone of the skeleton.
        /// </summary>
        public Transform skeletonRoot
        {
            get
            {
                return m_SkeletonRoot;
            }
            set
            {
                m_SkeletonRoot = value;
            }
        }

        void Start()
        {
            //InitializeSkeletonJoints();
        }
       

        //Breadth-first search
        Transform FindDeepChild(Transform aParent, string aName)
        {
            Queue<Transform> queue = new Queue<Transform>();
            queue.Enqueue(aParent);
            while (queue.Count > 0)
            {
                var c = queue.Dequeue();
                if (c.name == aName)
                {
                    return c;
                }
                foreach (Transform t in c)
                    queue.Enqueue(t);
            }
            return null;
        }

        public void InitializeSkeletonJoints()
        {
            // Walk through all the child joints in the skeleton and
            // store the skeleton joints at the corresponding index in the m_BoneMapping array.
            // This assumes that the bones in the skeleton are named as per the
            // JointIndices enum above.
            Queue<Transform> nodes = new Queue<Transform>();
            nodes.Enqueue(m_SkeletonRoot);

            while (nodes.Count > 0)
            {
                Transform next = nodes.Dequeue();
                for (int i = 0; i < next.childCount; ++i)
                {
                    nodes.Enqueue(next.GetChild(i));
                }
                ProcessJoint(next);
            }

           
            foreach (var jointKeyValuePair in ROBOTPOSE)
            {
                MyRigJoints[jointKeyValuePair.Key] = FindDeepChild(m_SkeletonRoot, jointKeyValuePair.Key.ToString());
            }

            int j = 0;
            foreach (var jointKeyValuePair in MyRigJoints)
            {
                //current world rotation of the joint in my rig
                Quaternion startingRot = jointKeyValuePair.Value.rotation;
                //arkit t pose
                Quaternion robotJointRotation = Quaternion.identity;
                JointIndices index = jointKeyValuePair.Key;
                if (!ROBOTPOSE.TryGetValue(index, out robotJointRotation))
                {
                    Debug.LogError("key missing is " + jointKeyValuePair.Key.ToString());
                    continue;
                }
                //conversionRotation can convert ARKit rotations to somewthing we can apply to our rig
                var conversionRotation = Quaternion.Inverse(robotJointRotation) * startingRot;

                RotationCompensations[j] = conversionRotation;
                j++;
            }

        }

        public void ApplyBodyPose(ARHumanBody body)
        {
            var joints = body.joints;
            if (!joints.IsCreated)
                return;

            for (int i = 0; i < k_NumSkeletonJoints; ++i)
            {
                XRHumanBodyJoint joint = joints[i];
                var bone = m_BoneMapping[i];
                if (bone != null)
                {
                    bone.transform.localPosition = joint.localPose.position;
                    bone.transform.rotation = joint.anchorPose.rotation * RotationCompensations[i];
                }
            }
        }

        void ProcessJoint(Transform joint)
        {
            int index = GetJointIndex(joint.name);
            if (index >= 0 && index < k_NumSkeletonJoints)
            {
                m_BoneMapping[index] = joint;
            }
            else
            {
                Debug.LogWarning($"{joint.name} was not found.");
            }
        }

        // Returns the integer value corresponding to the JointIndices enum value
        // passed in as a string.
        int GetJointIndex(string jointName)
        {
            JointIndices val;
            if (Enum.TryParse(jointName, out val))
            {
                return (int)val;
            }
            return -1;
        }
    }
}

But I cant seem to make it work.
It just produce weird view of the model when running the app.

I hope someone can help me, im willing to pay for the tutorial/source code just to make this work.
Thank you so much.

1 Like
4

I’ve been tracking these issues for over 9 months now… The issues are still not solved. I mean KEY feature of ARFoundation + ARKit is UNUSABLE “as is”, so many inconsistencies that it amazes me.

2 Likes
5

do you have a working solution for this? im willing to pay some bucks for the solution

2 Likes
6

Has anyone found a fix for this?

1 Like
7

Well, kinda… Funny thing, the robot rig provided in APPLE documentation for this purpose is WRONG! DO NOT USE IT. You should export robot rig from the arfoundation unity3D project, that’s right, export it, because the original model is hidden. It seems there is no shurtcuts at the moment - you need to take that rig from that robot and apply it to your outfit/character model.

8

How to export the robot rig (one from arfoundation unity3D project) from unity so that it can be imported in e.g. blender?

9

I was able to do this, sort of. I did not replace the ControlledRobot, instead I use it to control a separate model but hide the ControlledRobot as needed.

Images are from mapping the ControlledRobot to a separate avatar with a fairly standard rig (e.g. 3 spine joints, 1 neck vs the ControlledRobot with 7 spine joints and 4 neck).

The solution I came up with is a bit complicated, so I’m sure there is a better way to do this, but the essentials are:

a) I map each joint from the new model to the closest corresponding ControlledRobot joint, ignoring extras (something like avatar:Spine1 => robot:Spine, avatar:Spine2 => robot:Spine3, avatar:Spine3 => robot:Spine7, etc…).

b) I record the initial rotation for each joint on both models

c) I get and store the initial delta between their rotations:

robotLocalRotation = controlledRobotJoint.bone.worldToLocalMatrix.rotation
avatarLocalRotation = avatarJoint.bone.worldToLocalMatrix.rotation

avatarJoint.originalLookRotation = avatarLocalRotation* Quaternion.Inverse(robotLocalRotation);

d) On each LateUpdate I loop through all the joints of the avatar and find the mapped controlledRobot joint. I rotate the avatar joint to point to the same currentLookRotation (so same forward and up) as the matched controlledRobot joint. Than I rotate that back by the starting delta rotation (the originalLookRotation above).


controlledRobotJoint.currentLookRotation = Quaternion.LookRotation(bone.transform.forward, bone.transform.up);

avatarBone.rotation = controlledRobotJoint.currentLookRotation * Quaternion.Inverse(avatarJoint.originalLookRotation)

It tracks pretty well though I should be doing some additional trig to improve the rotations when there are intervening joints, but it has been decent for my purposes.

7211050--866539--robot1.jpg
7211050--866539--robot1.jpg1173×994 101 KB

7211050--866542--robot2.jpg
7211050--866542--robot2.jpg1324×981 98.7 KB

1 Like
10

thanks for sharing! sir can you please make a short tutorial about this ?

11

Here is a GitHub repo sample project. This uses AR Foundation body tracking (with the ControlledRobot from the Unity arfoundation-samples project) to puppeteer a second avatar with a different armature.

https://github.com/genereddick/ARBodyTrackingAndPuppeteering

1 Like
12

I appreciate you a lot for sharing your code. I was having a problem with this, as my character looks totally different from the ControlledRobot model. It was making weird stretch distortion to match the bone position/ location in ARkit. I’m planning to play with different numbers and equations to make this more precise. Also looking forward to your updated version.

13

How to export a prefab?

14

Hello there, i am still facing the problem with a custom Avatar for the Human Body tracking. I export the ControlledRobot with the FBX Exporter to Maya, unbind the skin, rebind my custom mesh to it, export as FBX and the whole model is broken (kind of rotated wrong?!). Are there any hints on this?!

@floatingfactory the example scene is also not working for me :frowning:

15

Is this already solved? I need a fix this also.

I tried exporting the prefab to FBX but when I import it in the blender, the model is messed up.

@floatingfactory same with @Sheeks_1 it’s not working for me when I tried it, Is there a specific way for this to work?