Duplicating a Mesh including blendshapes to send via network

Unity Engine
1

Hello all,

I’m stuck trying to save all information of a mesh (imported from an fbx) to a serializable format so that I can send that information via network. Don’t worry, I only have to do this once, so file size is not an issue for now. Copying vertices, normals, tangents, and so on is not an issue and works just as expected.

However, when I want to copy the blendshapes, this happens:

How the original looks with blendshape “jawopen” modified:

2156×1369 126 KB

How the copy looks:

2101×1167 214 KB

As you can see, while something is modified when altering the blendshape weights, some information is lost here.

Does anyone know what the issue might be here?

For testing, I made a simple script that copies all information from the current mesh to a new one.

public class MeshHolder
{
    public string name;
    public Vector3[] vertices;
    public Vector3[] normals;
    public Vector4[] tangents;
    public Vector2[] uv;
    public int[] triangles;
    public BlendShapeHolder[] BlendShapeHolders;
    public Matrix4x4[] bindposes;
    public BoneWeight[] boneWeights;
    public Bounds bounds;
    public int blendshapeCount;
}

public class BlendShapeHolder
{
    public string name;
    public float weight;
    public int frameCount;
    public int vertexCount;
    public Vector3[][] dVertices;
    public Vector3[][] dNormals;
    public Vector3[][] dTangents;
}

public void SaveMesh(){

Mesh myMesh = CurrentMesh.sharedMesh;

            MeshHolder.normals = myMesh.normals;
            MeshHolder.vertices = myMesh.vertices;
            MeshHolder.uv = myMesh.uv;
            MeshHolder.tangents = myMesh.tangents;
            MeshHolder.triangles = myMesh.triangles;
            MeshHolder.bindposes = myMesh.bindposes;
            MeshHolder.boneWeights = myMesh.boneWeights;
            MeshHolder.bounds = myMesh.bounds;
            MeshHolder.blendshapeCount = myMesh.blendShapeCount;

            Vector3[] dVertices = new Vector3[myMesh.vertexCount];
            Vector3[] dNormals = new Vector3[myMesh.vertexCount];
            Vector3[] dTangents = new Vector3[myMesh.vertexCount];
            MeshHolder.BlendShapeHolders = new BlendShapeHolder[myMesh.blendShapeCount];
            for (int shape = 0; shape < myMesh.blendShapeCount; shape++)
            {
                BlendShapeHolder holder = new BlendShapeHolder();
                holder.dVertices = new Vector3[myMesh.blendShapeCount][];
                holder.dTangents = new Vector3[myMesh.blendShapeCount][];
                holder.dNormals = new Vector3[myMesh.blendShapeCount][];
                holder.frameCount = myMesh.GetBlendShapeFrameCount(shape);
                for (int frame = 0; frame < holder.frameCount; frame++)
                {
                   holder.name = myMesh.GetBlendShapeName(shape);
                    float frameWeight = myMesh.GetBlendShapeFrameWeight(shape, frame);
                    holder.weight = frameWeight;
                    holder.dVertices[frame] = new Vector3[myMesh.vertices.Length];
                    holder.dTangents[frame] = new Vector3[myMesh.vertices.Length];
                    holder.dNormals[frame] = new Vector3[myMesh.vertices.Length];
                    myMesh.GetBlendShapeFrameVertices(shape, frame, dVertices, dNormals, dTangents);
                    for (int k = 0; k < holder.dVertices[frame].Length; k++)
                    {
                        holder.dVertices[frame][k] = dVertices[k];
                        holder.dNormals[frame][k] = dNormals[k];
                        holder.dTangents[frame][k] = dTangents[k];
                    }
                }

                MeshHolder.BlendShapeHolders[shape] = holder;
            }
}

… and transfers it to the new one.

public void LoadMesh(){
newMesh = new Mesh();
            newMesh.vertices = MeshHolder.vertices;
            newMesh.normals = MeshHolder.normals;
            newMesh.uv = MeshHolder.uv;
            newMesh.tangents = MeshHolder.tangents;
            newMesh.triangles = MeshHolder.triangles;
            newMesh.bindposes = MeshHolder.bindposes;
            newMesh.boneWeights = MeshHolder.boneWeights;
            newMesh.bounds = MeshHolder.bounds;
            for (int shape = 0; shape < MeshHolder.blendshapeCount; shape++)
            {
                int frameCount = MeshHolder.BlendShapeHolders[shape].frameCount;
                for (int frame = 0; frame < frameCount; frame++)
                {
                    BlendShapeHolder holder = MeshHolder.BlendShapeHolders[shape];
                    string shapeName = holder.name;
                    float frameWeight = holder.weight;
                    Vector3[] dVertices = holder.dVertices[frame];
                    Vector3[] dNormals = holder.dNormals[frame];
                    Vector3[] dTangents = holder.dTangents[frame];
                    newMesh.AddBlendShapeFrame(shapeName, frameWeight, dVertices, dNormals, dTangents);
                }
}
2

First thing is to massively simplify the data. Make a quad (or even a triangle) with a blendshape and then you have a chance of being able to print out the resulting vertices. Once you have a verified working small input dataset, feed the data through your code above and study the actual output data. With 3 or 4 vertices it might become possible to reason about where things are going wrong, which would not be possible with a full face head.

3

I’ve reduced the complexity to a cube in Maya, where I only animate one vertex with a single blendshape. Importing this into Unity and copying the mesh results in the below errors:

1365×731 37 KB

1899×1343 46.2 KB

As you can see, the result is not that dissimilar but while the right original model is correctly bent, on the left the faces of the cube seem only to shift and lose their connections.

4

2039×1199 143 KB

Here is another even simpler example using only a quad, changing one of the vertices. The blue one is the copied mesh, the grey one the imported fbx mesh. Could it have something to do with how Unity imports the fbx? Because playing around with these values has a massive impact on how the blue copy looks…

Is there a way to access these settings during runtime?

5

That could be a thing… check that you have unity scaling throughout: 1,1,1 on all Transforms involved, and also make sure the import settings on the FBX are 1.0.

I am not familiar enough with blendshapes to know if they can fail if scaling is off, but it sorta looks like perhaps the scaling might be what’s going on here.

6

I’m afraid it’s definitely not that simple. Scaling and import settings are correct :confused:

7

So, I came up with a solution after all. As a reminder, my main issue was that when copying data from a mesh object to a new mesh object, I seem to lose some information that affects the animation of the blendshapes as seen in the above images.

The solution: Do not use a new mesh at all but use a prefab of a working mesh, instantiate it, whenever you need to create a new object and then switch out all the information from the instantianted mesh with the copied mesh.

As an example: I saved one avatar head with working blendshapes as a prefab. Whenever I send mesh data from another avatar head through the network, I do not use this data to create a new mesh, but I instantiate the prefab and override the prefab’s mesh data with the data sent through the network :wink: