Face Capture: Jaw bone movement?

Unity Engine
,
1

I’ve just tried out Face Capture on a model I have. The model has standard blend shapes, which work well with face capture. However, my model also has a Jaw bone, which needs to be moved up/down as the mouth opens and closes. I don’t see a way to control that behavior with Face Capture.

Is there some approach to doing so? Or is jaw movement something that might come in a future update?

2

I don’t have a answer for the questions, but I saw use of bones in some Blender characters people created recently too. They used bones near each eyebrow, allowing bone animation to do some quite expressive face contortions.

Seems pretty hard to generalize though. Blendshapes really became normalized when the Apple ARKit set of shapes came out. It provided a degree of standardization, which I don’t think is there with facial bones at this stage.

3

It seems like maybe what I should be doing is get the bones to move via blend shapes as well. I’ve just been looking into this. It seems that I can associate bone transform changes to blend shape value changes. So, for my “Mouth_Open” blend shape, I can also make the jaw move. So, I’ll be looking into whether that’s the solution here.

4

@dgoyette Yes, it is easiest to create a script that moves the bones based on the blend shapes of the mesh. Ideally the script should apply the pose between LateUpdate and rendering to avoid any latency.

5

Thanks. I put together a little script to do this. There were a few blend shapes that could potentially move the jaw bone, so the script supports multiple blend shapes each having some impact on the bone. Just including it here in case it’s useful to anyone.

using System;
using System.Collections.Generic;
using System.Linq;
using UnityEngine;

[ExecuteInEditMode]
public class LeoBoneController : MonoBehaviour
{
    public SkinnedMeshRenderer BodyRenderer;
    public Transform JawBone;
    public List<BonePositionBlender> JawBlenders;
    public Vector3 JawBoneOffsetLowerLimit;
    public Vector3 JawBoneOffsetUpperLimit;

    private Dictionary<string, int> _blendShapeIndices;

    private Vector3 _initialJawLocalPosition;
    private bool _jawPositionInitialized;

    private void LateUpdate()
    {
        if (!_jawPositionInitialized)
        {
            InitializeJawPosition();
        }
        else
        {
            var stuff = JawBlenders.Select(jb => BodyRenderer.GetBlendShapeWeight(_blendShapeIndices[jb.BlendShapeName]) / 100 * jb.FullPositionOffset).ToList();
            Vector3 jawOffset = JawBlenders.Select(jb => BodyRenderer.GetBlendShapeWeight(_blendShapeIndices[jb.BlendShapeName]) / 100 * jb.FullPositionOffset)
                .Sum()
                .Clamp(JawBoneOffsetLowerLimit, JawBoneOffsetUpperLimit);

            JawBone.transform.localPosition = _initialJawLocalPosition + jawOffset;
        }
    }

    private void InitializeJawPosition()
    {
        if (BodyRenderer == null || JawBone == null)
        {
            return;
        }

        _blendShapeIndices = new Dictionary<string, int>();
        for (int blendShapeIndex = 0; blendShapeIndex < BodyRenderer.sharedMesh.blendShapeCount; blendShapeIndex++)
        {
            _blendShapeIndices[BodyRenderer.sharedMesh.GetBlendShapeName(blendShapeIndex)] = blendShapeIndex;
        }

        _initialJawLocalPosition = JawBone.localPosition;
        _jawPositionInitialized = true;
    }

    [Serializable]
    public class BonePositionBlender
    {
        public string BlendShapeName;
        public Vector3 FullPositionOffset;

    }
}

public static class Vector3Util
{
    public static Vector3 Sum(this IEnumerable<Vector3> self)
    {
        var s = self.ToList();
        Vector3 sum = Vector3.zero;
        foreach (var v3 in self)
        {
            sum += v3;
        }

        return sum;
    }

    public static Vector3 Clamp(this Vector3 self, Vector3 low, Vector3 high)
    {
        return new Vector3(Mathf.Clamp(self.x, low.x, high.x),
            Mathf.Clamp(self.y, low.y, high.y),
            Mathf.Clamp(self.z, low.z, high.z));
    }
}