Example Rig for 3D Human Skeleton

Unity Engine
1

Hello,

I’m attempting to use the new ARKit 3 “motion capture” feature for realtime 3D human skeleton creation and retargeting. I couldn’t find any documentation on this, and the ARFoundation sample project on git only superficially touched on it by mapping a sphere onto a human “head” joint.

Based on my sleuthing, however, I was able to create a system that should theoretically map a the AR rig to a character rig from Maya. Unfortunately, this approach quite literally blew up in my face, with random triangles morphing everywhere in a cacophony of geometry. Something is clearly wrong here.

Is there by chance an example rig or prefab I can take a look at, so our artist and I can make the necessary adjustments to get our rig to work?

Thanks!
David

1 Like
2

Dude. Came here to post exactly this. By using position only I can at least get the joints into the right location - but the mesh is all torn up (because obviously rigs are built for rotation only). Using the rotations directly…is a bad idea.

I’ve been thinking that using the world space forward vector of each joint translated into model space might be the best idea. That way if you have a different rig hierarchy (or move the model around) you won’t have weird issues.

Still trying to get it to work - but hoping to maybe crowdsource some solutions.

Edit: totally doesn’t work. It gives you the same value as the world space quaternion. Makes sense but I was really hoping that there was a bug somewhere in the rotations.

3

Thanks for posting this! Good to know we have some help :slight_smile:

We created a flat hierarchy (i.e. sibling relationship) of cubes for each joint and placed them under the root joint. Then we updated both the local positions and rotations of each cube to match the joint from ARFoundation.

What we discovered surprised us. The rotations of the cubes looked correct, down to the finger joints. This means that the data we get from ARFoundation assumes our rig joints are siblings. This is pretty non standard for a rig, I know.

Tomorrow we’re going to make a very simple rig made of cylinders to see if we can at least get a humanoid-looking figure that way. We suspect we simply aren’t correctly weighting the joints to the skinned mesh (seriously, what are you supposed to do with 7 spine joints??). That’s why having an example from Apple or ARFoundation would be just peachy :wink:

4

That’s super surprising. Yes please let me know how that cylinder test goes.

I’d been working under the assumption that there was a bug in rotations. As such - I tested out having the joints “look at” the location of the next point in the rig. This causes a ton of issues if your rig isn’t the same size as the recording of the human - and I was in the middle of trying to fix that part too.

If the rotations coming out of ARFoundation are correct - then what could be the root problem is that our joints aren’t calibrated to the same “zero” rotation. Essentially we would have to convert the ARFoundation rotation into “joint space” and then do the rotation.

What would be super helpful in your test is coloring one end of the cylinder to ensure that the joints aren’t flipping around 180 degrees or something weird like that…as that would also explain the busted behavior I’ve been seeing.

5

Here’s a script that can generate the neutral rig used by ARKit 3. I got the information from a native iOS app. Apple provided an API to access “neturalPose” but Unity’s ARFoundation didn’t implement it.

Please drag a small sphere as the prefab to the “Sphere Prefab” slot.

https://paste.ubuntu.com/p/nvQ7QyQbhS/

1 Like
6

This is probably because you are using
ARHumanBody.joints[index].anchorPose instead of ARHumanBody.joints[index].localPose
The former transform is relative the root, the later is the local transform

7

Thanks for posting that Drew. I used that to figure out that what they consider “neutral pose” and what my rig considers “neutral pose” are wildly different. It looks like those values are in world space?

The closest I’ve gotten to something that looks correct is creating a rotation per-joint that converts ARKit neutral pose to my rig’s neutral pose:
Quaternion conversionRotation = Quaternion.Inverse(arkitNeutralRot) * myRigNeutralRot;

I then take the ARKit world rotation and apply the conversionRotation to it:
tform.rotation = ARHumanBody.joints[index].anchorPose * conversionRotation;

Unfortunately - the root of each new hierarchy (arms, legs, spine_0) looks like it has a 180 turn applied to it in the y direction.

I’m starting to wonder if there’s an issue with coordinate systems or something going on. Though honestly - it feels near impossible to fix this without access to source or a working example.

8

Yeah I’m having the same issue. The neutral rig I got from Apple’s API and the tracking rig information I got from Unity doesn’t match exactly. Some joints has a 180 turn and some don’t.
Either Apple made a mistake in their API, or Unity did some remapping under the hood.
I spent a day to manually change their rotation to make the mapping barely work. It’s not worth it. I suggest to wait for updates from Apple/Unity.

1 Like
9

Thanks guys, this has been super helpful.

@drewhong , we used the rig generated from the code you provided to create our own rig. We then used

jointTransform.localRotation = joint.localPose.rotation;

…iteratively on each joint in the rig.

We now have a skinned mesh that looks mostly humanoid in AR, made out of cylinders and a half-sphere for the head. However, there is still a lot of twisting at the joints so we’re looking into what’s going on there.

10

Forgot to mention, we are totally ignoring positional data for simplicity. Also, we discovered that localPose.position was returning a zero-vector anyway. The only way for us to apply positions was to use anchoredPose.position and then re-convert it into the correct coordinate space. It didn’t look much better on our rig anyway, so we left it off.

11

Happy (but sad) to hear I’m not alone on this one. Sounds like bug territory if we’re all running into exactly the same thing.

I went digging around in the package source - but its basically a thin shim to a native dll…so there wasn’t much to learn. Maybe we can get @jimmya for some wisdom?

1 Like
12

Had a convo with Jimmy yesterday. He poked some people aaaaaaaaaaaand GitHub - Unity-Technologies/arfoundation-samples: Example content for Unity projects based on AR Foundation was updated with a 3d rig example. It looks pretty thin though. I’ll be digging into it today.

13

The HumanBodyTracking3D scene, in arfoundation-samples, has been updated to control the ARKit3 Robot model’s skeleton, using the ARHumanBody pose detected by ARFoundation.
The HumanBodyTracker MonoBehavior takes care of detecting human bodies and passing on the detected body pose to the Robot’s skeleton, via BoneController. The BoneController is a component on the Robot model, that maps the detected Human body pose to the Robot’s skeleton bones.

3 Likes
14

Thanks for publishing and posting a response! I think this may have helped me figure out the problem.

Background: To quickly iterate - I had recorded some data from my iPhone and hacked up a quick body-tracking simulator. For simplicity I set world rotation of my rig to equal the world rotation of the pre-recorded body data.

When hooking this up to my character - I get total garbage. When I hooked it up to the robot rig - it worked perfectly. WTF. So clearly there’s some assumption baked into the rigs that’s different (which I THOUGHT that world rotation would overcome. Seems not). I’ve attached two images showing the knee joint selected (and the move gizmo set to local space).
4669310--439274--knee.JPG 4669310--439277--knee2.JPG

I THINK this is the root of the problem. The robot rig has the X axis pointing to the next joint. My rig has negative-z pointing to the next joint. I get very different results when setting the same rotation on both rigs. I’ll be honest - I wasn’t expecting that result when I’m setting the world rotation of the joint…but I don’t have extensive experience with character rigs so maybe this should have been obvious.

In any case - I think I was close with the idea of “defining a rest pose - calibrating against it to create an offset - applying that offset when reading in tracking data” I just need to figure out why my initial attempt failed.

15

GOT IT

Turns out the neutral pose that drew pasted has some bad rotations in it. This is all super hacked together but is a good starting point. Using this index map:

    enum JointIndices
    {
        Invalid = -1,
        Root = 0, // parent: Invalid
        Hips = 1, // parent: Root
        LeftUpLeg = 2, // parent: Hips
        LeftLeg = 3, // parent: LeftUpLeg
        LeftFoot = 4, // parent: LeftLeg
        LeftToes = 5, // parent: LeftFoot
        LeftToesEnd = 6, // parent: LeftToes
        RightUpLeg = 7, // parent: Hips
        RightLeg = 8, // parent: RightUpLeg
        RightFoot = 9, // parent: RightLeg
        RightToes = 10, // parent: RightFoot
        RightToesEnd = 11, // parent: RightToes
        Spine1 = 12, // parent: Hips
        Spine2 = 13, // parent: Spine1
        Spine3 = 14, // parent: Spine2
        Spine4 = 15, // parent: Spine3
        Spine5 = 16, // parent: Spine4
        Spine6 = 17, // parent: Spine5
        Spine7 = 18, // parent: Spine6
        RightShoulder1 = 19, // parent: Spine7
        RightShoulder2 = 20, // parent: RightShoulder1
        RightArm = 21, // parent: RightShoulder2
        RightForearm = 22, // parent: RightArm
        RightHand = 23, // parent: RightForearm
        RightHandThumbStart = 24, // parent: RightHand
        RightHandThumb1 = 25, // parent: RightHandThumbStart
        RightHandThumb2 = 26, // parent: RightHandThumb1
        RightHandThumbEnd = 27, // parent: RightHandThumb2
        RightHandIndexStart = 28, // parent: RightHand
        RightHandIndex1 = 29, // parent: RightHandIndexStart
        RightHandIndex2 = 30, // parent: RightHandIndex1
        RightHandIndex3 = 31, // parent: RightHandIndex2
        RightHandIndexEnd = 32, // parent: RightHandIndex3
        RightHandMidStart = 33, // parent: RightHand
        RightHandMid1 = 34, // parent: RightHandMidStart
        RightHandMid2 = 35, // parent: RightHandMid1
        RightHandMid3 = 36, // parent: RightHandMid2
        RightHandMidEnd = 37, // parent: RightHandMid3
        RightHandRingStart = 38, // parent: RightHand
        RightHandRing1 = 39, // parent: RightHandRingStart
        RightHandRing2 = 40, // parent: RightHandRing1
        RightHandRing3 = 41, // parent: RightHandRing2
        RightHandRingEnd = 42, // parent: RightHandRing3
        RightHandPinkyStart = 43, // parent: RightHand
        RightHandPinky1 = 44, // parent: RightHandPinkyStart
        RightHandPinky2 = 45, // parent: RightHandPinky1
        RightHandPinky3 = 46, // parent: RightHandPinky2
        RightHandPinkyEnd = 47, // parent: RightHandPinky3
        LeftShoulder1 = 48, // parent: Spine7
        LeftShoulder2 = 49, // parent: LeftShoulder1
        LeftArm = 50, // parent: LeftShoulder2
        LeftForearm = 51, // parent: LeftArm
        LeftHand = 52, // parent: LeftForearm
        LeftHandThumbStart = 53, // parent: LeftHand
        LeftHandThumb1 = 54, // parent: LeftHandThumbStart
        LeftHandThumb2 = 55, // parent: LeftHandThumb1
        LeftHandThumbEnd = 56, // parent: LeftHandThumb2
        LeftHandIndexStart = 57, // parent: LeftHand
        LeftHandIndex1 = 58, // parent: LeftHandIndexStart
        LeftHandIndex2 = 59, // parent: LeftHandIndex1
        LeftHandIndex3 = 60, // parent: LeftHandIndex2
        LeftHandIndexEnd = 61, // parent: LeftHandIndex3
        LeftHandMidStart = 62, // parent: LeftHand
        LeftHandMid1 = 63, // parent: LeftHandMidStart
        LeftHandMid2 = 64, // parent: LeftHandMid1
        LeftHandMid3 = 65, // parent: LeftHandMid2
        LeftHandMidEnd = 66, // parent: LeftHandMid3
        LeftHandRingStart = 67, // parent: LeftHand
        LeftHandRing1 = 68, // parent: LeftHandRingStart
        LeftHandRing2 = 69, // parent: LeftHandRing1
        LeftHandRing3 = 70, // parent: LeftHandRing2
        LeftHandRingEnd = 71, // parent: LeftHandRing3
        LeftHandPinkyStart = 72, // parent: LeftHand
        LeftHandPinky1 = 73, // parent: LeftHandPinkyStart
        LeftHandPinky2 = 74, // parent: LeftHandPinky1
        LeftHandPinky3 = 75, // parent: LeftHandPinky2
        LeftHandPinkyEnd = 76, // parent: LeftHandPinky3
        Neck1 = 77, // parent: Spine7
        Neck2 = 78, // parent: Neck1
        Neck3 = 79, // parent: Neck2
        Neck4 = 80, // parent: Neck3
        Head = 81, // parent: Neck4
        Jaw = 82, // parent: Head
        Chin = 83, // parent: Jaw
        Nose = 84, // parent: Head
        RightEye = 85, // parent: Head
        RightEyeUpperLid = 86, // parent: RightEye
        RightEyeLowerLid = 87, // parent: RightEye
        RightEyeBall = 88, // parent: RightEye
        LeftEye = 89, // parent: Head
        LeftEyeUpperLid = 90, // parent: LeftEye
        LeftEyeLowerLid = 91, // parent: LeftEye
        LeftEyeBall = 92, // parent: LeftEye
    }

I dumped the T pose of the robot into a dictionary IN WORLD COORDINATES:

        private Dictionary<JointIndices, Quaternion> ROBOTPOSE = new Dictionary<JointIndices, Quaternion>
        {
            { JointIndices.Root, new Quaternion(0f, 0f, 0f, 1f) },{ JointIndices.Hips, new Quaternion(0f, 0f, 0f, 1f) },{ JointIndices.LeftUpLeg, new Quaternion(-0.5251914f, 0.4729119f, 0.5257655f, -0.4733909f) },{ JointIndices.RightUpLeg, new Quaternion(-0.4731499f, -0.5254809f, 0.4731455f, 0.5254829f) },{ JointIndices.Spine1, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },{ JointIndices.LeftLeg, new Quaternion(-0.4589739f, 0.5374146f, 0.4594831f, -0.5379627f) },{ JointIndices.RightLeg, new Quaternion(-0.5376937f, -0.4592238f, 0.5376896f, 0.4592263f) },{ JointIndices.Spine2, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },{ JointIndices.LeftFoot, new Quaternion(-0.6927235f, 0.1400152f, 0.6934642f, -0.1401326f) },{ JointIndices.RightFoot, new Quaternion(-0.1400807f, -0.6930928f, 0.1400763f, 0.6930935f) },{ JointIndices.Spine3, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },{ JointIndices.LeftToes, new Quaternion(-0.7067051f, 0.006181896f, 0.7074544f, -0.006156165f) },{ JointIndices.RightToes, new Quaternion(-0.006176293f, -0.7070798f, 0.006170991f, 0.7070799f) },{ JointIndices.Spine4, new Quaternion(-0.5f, -0.5f, 0.5f, 0.5f) },{ JointIndices.LeftToesEnd, new Quaternion(-0.7067052f, 0.006181717f, 0.7074544f, -0.006156351f) },{ JointIndices.RightToesEnd, new Quaternion(-0.00617677f, -0.7070798f, 0.006170507f, 0.7070799f) },{ JointIndices.Spine5, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },{ JointIndices.Spine6, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },{ JointIndices.Spine7, new Quaternion(-0.4925276f, -0.5073624f, 0.4925276f, 0.5073624f) },{ JointIndices.RightShoulder1, new Quaternion(-0.7004818f, -0.08645806f, 0.09656784f, 0.7018013f) },{ JointIndices.LeftShoulder1, new Quaternion(-0.7017999f, 0.09655666f, 0.08646959f, -0.7004833f) },{ JointIndices.Neck1, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },{ JointIndices.RightShoulder2, new Quaternion(-0.7004818f, -0.08645806f, 0.09656784f, 0.7018013f) },{ JointIndices.LeftShoulder2, new Quaternion(-0.7017999f, 0.09655666f, 0.08646959f, -0.7004833f) },{ JointIndices.Neck2, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },{ JointIndices.RightArm, new Quaternion(-0.7072251f, -4.330277E-05f, -0.0008040965f, 0.706988f) },{ JointIndices.LeftArm, new Quaternion(-0.7069915f, -0.0008285469f, 7.144874E-05f, -0.7072218f) },{ JointIndices.Neck3, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },{ JointIndices.RightForearm, new Quaternion(-0.706572f, -0.03035438f, -0.03110623f, 0.7063052f) },{ JointIndices.LeftForearm, new Quaternion(-0.7063079f, -0.03109138f, 0.03034304f, -0.7065705f) },{ JointIndices.Neck4, new Quaternion(-0.5572078f, -0.4353384f, 0.5572078f, 0.4353384f) },{ JointIndices.RightHand, new Quaternion(-0.0001886487f, -0.04345921f, -0.0005316734f, 0.9990551f) },{ JointIndices.LeftHand, new Quaternion(-0.999056f, -0.0005290508f, 0.04344079f, -0.0001838654f) },{ JointIndices.Head, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },{ JointIndices.RightHandThumbStart, new Quaternion(-0.563176f, -0.450308f, -0.1651125f, 0.6728995f) },{ JointIndices.RightHandIndexStart, new Quaternion(-0.001846135f, -0.1037805f, -0.0003555566f, 0.9945986f) },{ JointIndices.RightHandMidStart, new Quaternion(-0.001781464f, -0.04345801f, -0.0004611611f, 0.9990537f) },{ JointIndices.RightHandRingStart, new Quaternion(-0.001758099f, 0.006566688f, -0.0005474091f, 0.9999769f) },{ JointIndices.RightHandPinkyStart, new Quaternion(-0.00176309f, 0.05727234f, -0.0006380975f, 0.9983569f) },{ JointIndices.LeftHandThumbStart, new Quaternion(-0.6729128f, -0.16509f, 0.4503065f, -0.5631677f) },{ JointIndices.LeftHandIndexStart, new Quaternion(-0.9946009f, -0.0003358126f, 0.1037586f, -0.00184305f) },{ JointIndices.LeftHandMidStart, new Quaternion(-0.9990543f, -0.000755012f, 0.04344012f, -0.001763821f) },{ JointIndices.LeftHandRingStart, new Quaternion(-0.9999767f, -0.0005270988f, -0.006585166f, -0.001753271f) },{ JointIndices.LeftHandPinkyStart, new Quaternion(-0.9983559f, -0.0006175488f, -0.05729078f, -0.001757562f) },{ JointIndices.Jaw, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },{ JointIndices.Nose, new Quaternion(-0.5000001f, -0.5000001f, 0.5000001f, 0.5000001f) },{ JointIndices.RightEye, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.LeftEye, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.RightHandThumb1, new Quaternion(-0.6383249f, -0.3353892f, -0.0345653f, 0.691998f) },{ JointIndices.RightHandIndex1, new Quaternion(-0.141001f, -0.06067762f, 0.1492173f, 0.976817f) },{ JointIndices.RightHandMid1, new Quaternion(-0.001823545f, -0.01739544f, 0.08677565f, 0.9960744f) },{ JointIndices.RightHandRing1, new Quaternion(0.0863688f, 0.043634f, 0.0978535f, 0.9904854f) },{ JointIndices.RightHandPinky1, new Quaternion(0.1280032f, 0.08536577f, 0.1233153f, 0.980368f) },{ JointIndices.LeftHandThumb1, new Quaternion(-0.6920068f, -0.03454062f, 0.3353892f, -0.6383165f) },{ JointIndices.LeftHandIndex1, new Quaternion(-0.9768156f, 0.1492398f, 0.06065904f, -0.1409955f) },{ JointIndices.LeftHandMid1, new Quaternion(-0.9961001f, 0.08648346f, 0.01738015f, -0.001796693f) },{ JointIndices.LeftHandRing1, new Quaternion(-0.9904823f, 0.09787226f, -0.04365361f, 0.08637467f) },{ JointIndices.LeftHandPinky1, new Quaternion(-0.980363f, 0.1233332f, -0.08538568f, 0.1280103f) },{ JointIndices.Chin, new Quaternion(-0.4956177f, -0.5043442f, 0.4956177f, 0.5043442f) },{ JointIndices.RightEyeUpperLid, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.RightEyeLowerLid, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.RightEyeBall, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.LeftEyeUpperLid, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.LeftEyeLowerLid, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.LeftEyeBall, new Quaternion(-0.4999974f, -0.5000026f, 0.4999974f, 0.5000026f) },{ JointIndices.RightHandThumb2, new Quaternion(-0.6677005f, -0.2722507f, 0.03234765f, 0.692105f) },{ JointIndices.RightHandIndex2, new Quaternion(-0.1477825f, -0.0415144f, 0.2769809f, 0.9485351f) },{ JointIndices.RightHandMid2, new Quaternion(-0.004873842f, -0.01679799f, 0.2617181f, 0.964986f) },{ JointIndices.RightHandRing2, new Quaternion(0.0938583f, 0.02353925f, 0.3136097f, 0.9446088f) },{ JointIndices.RightHandPinky2, new Quaternion(0.1357787f, 0.07236205f, 0.2191423f, 0.9634857f) },{ JointIndices.LeftHandThumb2, new Quaternion(-0.6921116f, 0.03237311f, 0.2722515f, -0.6676922f) },{ JointIndices.LeftHandIndex2, new Quaternion(-0.9485307f, 0.277003f, 0.04149672f, -0.1477744f) },{ JointIndices.LeftHandMid2, new Quaternion(-0.965063f, 0.2614351f, 0.01678771f, -0.004844725f) },{ JointIndices.LeftHandRing2, new Quaternion(-0.9446015f, 0.3136272f, -0.0235571f, 0.09386835f) },{ JointIndices.LeftHandPinky2, new Quaternion(-0.963479f, 0.2191598f, -0.07238111f, 0.1357878f) },{ JointIndices.RightHandThumbEnd, new Quaternion(-0.6677005f, -0.2722507f, 0.03234762f, 0.692105f) },{ JointIndices.RightHandIndex3, new Quaternion(-0.1494033f, -0.03523868f, 0.3167771f, 0.9359966f) },{ JointIndices.RightHandMid3, new Quaternion(-0.005977005f, -0.01643786f, 0.325129f, 0.945508f) },{ JointIndices.RightHandRing3, new Quaternion(0.09519702f, 0.01735026f, 0.3747187f, 0.9220752f) },{ JointIndices.RightHandPinky3, new Quaternion(0.1402616f, 0.06323615f, 0.2822996f, 0.9469081f) },{ JointIndices.LeftHandThumbEnd, new Quaternion(-0.6921116f, 0.03237309f, 0.2722515f, -0.6676922f) },{ JointIndices.LeftHandIndex3, new Quaternion(-0.9358417f, 0.3172399f, 0.03515086f, -0.1494111f) },{ JointIndices.LeftHandMid3, new Quaternion(-0.9456036f, 0.3248518f, 0.01642954f, -0.005947292f) },{ JointIndices.LeftHandRing3, new Quaternion(-0.9446015f, 0.3136272f, -0.0235571f, 0.09386835f) },{ JointIndices.LeftHandPinky3, new Quaternion(-0.9469005f, 0.2823165f, -0.06325462f, 0.1402719f) },{ JointIndices.RightHandIndexEnd, new Quaternion(-0.1494033f, -0.03523868f, 0.3167771f, 0.9359965f) },{ JointIndices.RightHandMidEnd, new Quaternion(-0.005977005f, -0.01643786f, 0.325129f, 0.945508f) },{ JointIndices.RightHandRingEnd, new Quaternion(0.09519702f, 0.01735026f, 0.3747187f, 0.9220752f) },{ JointIndices.RightHandPinkyEnd, new Quaternion(0.1402616f, 0.06323615f, 0.2822996f, 0.9469081f) },{ JointIndices.LeftHandIndexEnd, new Quaternion(-0.9358417f, 0.3172399f, 0.03515095f, -0.1494111f) },{ JointIndices.LeftHandMidEnd, new Quaternion(-0.9456036f, 0.3248518f, 0.01642954f, -0.005947292f) },{ JointIndices.LeftHandRingEnd, new Quaternion(-0.9446015f, 0.3136272f, -0.0235571f, 0.09386835f) },{ JointIndices.LeftHandPinkyEnd, new Quaternion(-0.9469005f, 0.2823165f, -0.06325462f, 0.1402719f) },

        };

At initialization - I compare my character’s T pose to the robot’s T pose - create a rotation that will convert from “robot joint space” to “my joint’s space” - and store that value for later:

            foreach (var jointKeyValuePair in MyRigJoints)
            {
                //current world rotation of the joint in my rig
                Quaternion startingRot = jointKeyValuePair.Value.rotation;
                //arkit t pose
                Quaternion robotJointRotation = Quaternion.identity;
                JointIndices index = jointKeyValuePair.Key;
                if (!ROBOTPOSE.TryGetValue(index, out robotJointRotation))
                {
                    Debug.LogError("key missing is " + jointKeyValuePair.Key.ToString());
                    continue;
                }
                //conversionRotation can convert ARKit rotations to somewthing we can apply to our rig
                var conversionRotation = Quaternion.Inverse(robotJointRotation) * startingRot;

                RotationCompensations[index] = compensate;
            }

Then later when you get the updated human body pose from ARKit, you apply that stored compensation to the arkit rotations:

myjointTransform.rotation = ARHumanBody.joints[index].anchorPose.rotation * RotationCompensations[index];

The closer your character’s T pose is to the reference T pose - the more accurate your animations will be.

Since all of this is in world space - you’ll still need to account for any parent transforms. Eventually I’d like to get this working in local space…but really I just want to move on at this point.

Hope that helps and fixes everyone else’s issues. Thanks again for all the help folks.

1 Like
16

Sorry for the bad rotations… I really have no idea where it went wrong. I’ll look in to it. Sorry for the confusion.

17

No worries. It was impossible to tell since nothing at all was working. In all honesty - it still helped me significantly. Once I had correct data I was able to quickly fashion a solution. Wouldn’t have been able to do that without your post.

1 Like
18

@distastee Don’t suppose you’ve made one of these for the changed mappings and robot in beta 4 yet?

19

Not yet. Just upgraded yesterday and all my shit is completely hosed. I’m working on fixing it today - which means I’ll post here once I figure out wtf went wrong.

20

I don’t think any changes are necessary. The indicies all switched place - but it was thankfully built to handle that. From what I can tell - all the rotations + tracking seem to remain accurate.