I’m working on Editor tools that compare vertex positions in a SkinnedMesh with saved base poses. Saving the base pose vertex positions works as expected, but I’m having trouble when I try to take into account the bone transformations and weights so I can compare the save positions with the current ones.
The full system is very complex, so I’ve created a simpler example to illustrate. (This example has no practical purpose, it’s just for the purposes of this question.)
To save the base pose vertex positions I use skinnedMeshRenderer.BakeMesh() and save the results to a Vector3 List. I then loop through the vertex positions to find non-zero weight bones. Each one found is added to a List of VertexBoneWeight structs. These exist simply so I can sort them by boneIndex before saving data to the mesh UV channels. The end result is extra UV channels in the mesh:
UV1: Saved base pose vertex positions.
UV2: Vertex bone indices.
UV3: Vertex bone weights.
private void SaveVertexPositions()
{
//Create an empty mesh.
Mesh mesh = new Mesh();
//Save a snapshot of the SkinnedMeshRenderer to mesh. This is the base pose.
skinnedMeshRenderer.BakeMesh(mesh, true);
//A vector3 list to hold the mesh vertex positions.
var basePoseVertices = new List<Vector3>();
//Save mesh vertex positions to basePoseVertices.
mesh.GetVertices(basePoseVertices);
//A vector3 list to hold the mesh vertex positions after we've adjusted them.
var basePoseVerticesAdjusted = new List<Vector3>();
//Use the same vertex positions for now.
basePoseVerticesAdjusted = basePoseVertices;
//A vector4 list that will hold bone indices for each vertex.
var boneIndicesUV = new List<Vector4>();
//A vector4 list that will hold bone weights for each vertex.
var boneWeightsUV = new List<Vector4>();
//The non-zero bone weights for this mesh in vertex index order
var boneWeights = skinnedMeshRenderer.sharedMesh.GetAllBoneWeights();
//The number of non-zero bone weights per vertex.
var bonesPerVertex = skinnedMeshRenderer.sharedMesh.GetBonesPerVertex();
//Keep track of where we are in the array of BoneWeight1s as we iterate over the vertices.
var boneWeightVertexIndex = 0;
//Loop through the saved mesh vertices.
for (int vertexIndex = 0; vertexIndex < basePoseVertices.Count; vertexIndex++)
{
//Will hold the vertex position after adjusting for bone weighting etc.
Vector3 adjustedVertexPosition = Vector3.zero;
//Number of non-zero bone weights for this vertex.
var numberOfBonesForThisVertex = bonesPerVertex[vertexIndex];
//A list of VertexBoneWeights to hold the bone weight influences on this vertex.
//We build this list so we can sort it by boneIndex before saving the data to UV channels.
//This means the bone order matches the shader setup.
var thisVertexBoneWeights = new List<VertexBoneWeight>();
//Loop for numberOfBonesForThisVertex.
for (int vertexBoneIndex = 0; vertexBoneIndex < numberOfBonesForThisVertex; vertexBoneIndex++)
{
//The bone index.
int boneIndex = boneWeights[boneWeightVertexIndex].boneIndex;
//The current bone weight.
float boneWeight = boneWeights[boneWeightVertexIndex].weight;
//Create a new transformation matrix.
Matrix4x4 boneMatrix = skinnedMeshRenderer.bones[boneIndex].localToWorldMatrix * skinnedMeshRenderer.sharedMesh.bindposes[boneIndex];
//Calculate a new vertex position by transforming it by vertexMatrix, then multiplying by the bone weight at this vertex.
//Add it to adjustedVertexPosition so the influence of each bone at this vertex is taken into account.
adjustedVertexPosition += (boneMatrix.MultiplyPoint(basePoseVertices[vertexIndex]) * boneWeight);
//Add a new VertexBoneWeight to thisVertexBoneWeights.
thisVertexBoneWeights.Add(new VertexBoneWeight(boneIndex, boneWeight));
//Increment boneWeightVertexIndex.
boneWeightVertexIndex++;
}
//Vertex position.
//Update basePoseVerticesAdjusted[vertexIndex] with the adjusted vertex position.
basePoseVerticesAdjusted[vertexIndex] = adjustedVertexPosition;
//Sort thisVertexBoneWeights[] by boneIndex in ascending order. We need to do this so the data saved to the UV channel is in the correct order for the shader.
thisVertexBoneWeights.Sort(delegate(VertexBoneWeight x, VertexBoneWeight y)
{
return x.boneHandler.idx.CompareTo(y.boneHandler.idx);
});
//Create Vector4 data ready for saving to the UV channels.
Vector4 workingBoneIndices = Vector4.zero;
Vector4 workingBoneWeights = Vector4.zero;
//We need to fill the x,y,z,w components of workingBoneIndices and workingBoneWeights, but there may be less than four bones weights influencing each mesh vertex. We need to fill any gaps with appropriate blank values.
//Loop for maxBonePerVertex.
for (int i = 0; i < maxBonePerVertex; i++)
{
//If thisVertexBoneWeights *has a valid item.*
if (i < thisVertexBoneWeights.Count)
{
//Update the appropriate components with the data.
workingBoneIndices = thisVertexBoneWeights*.boneHandler.boneIndex;*
workingBoneWeights = thisVertexBoneWeights*.weight;*
}
//Else thisVertexBoneWeights has no item.
else
{
//Update the appropriate components with blank data.
workingBoneIndices = -1f;
workingBoneWeights = 0f;
}
}
//Save the bone indices to boneIndicesUV[vertexIndex].
boneIndicesUV.Add(workingBoneIndices);
//Save the bone weights to boneWeightsUV[vertexIndex].
boneWeightsUV.Add(workingBoneWeights);
}
//Save the calculated data to the mesh UV channels.
skinnedMeshRenderer.sharedMesh.SetUVs(1, basePoseVerticesAdjusted);
skinnedMeshRenderer.sharedMesh.SetUVs(2, boneIndicesUV);
skinnedMeshRenderer.sharedMesh.SetUVs(3, boneWeightsUV);
//Destroy the SkinnedMeshRenderer snapshot.
DestroyImmediate(mesh);
}
[Serializable]
public struct VertexBoneWeight
{
//The boneIndex for the bone.
[SerializeField] public int boneIndex;
//The bone weight at this vertex.
[SerializeField] public float weight;
public VertexBoneWeight(int boneIndex, float weight)
{
this.boneIndex = boneIndex;
this.weight = weight;
}
}
In the shader we have four properties to hold the current bone transform matrices:
_Bone_0_Local_Matrix
_Bone_1_Local_Matrix
_Bone_2_Local_Matrix
_Bone_3_Local_Matrix
These are kept up to date in LateUpdate():
private void LateUpdate()
{
//Get the current MaterialPropertyBlock properties and save them to propertyBlock.
skinnedMeshRenderer.GetPropertyBlock(propertyBlock);
//Set shader property for root transform matrix.
propertyBlock.SetMatrix(rootTransformMatrixPropID, rootLocalMatrix);
//Loop through bones.
//(We may have less bones than the maximum of 4. In the real system this is taken care of.)
for (int i = 0; i < bones.Length; i++)
{
//Create a local transformation matrix for the bone by multiplying meshLocalMatrix by the appropriate bindpose.
Matrix4x4 boneLocalMatrix = bones_.transform.localToWorldMatrix _
_skinnedMeshRenderer.sharedMesh.bindposes[bones.boneIndex];*_
//Set shader property for bone matrix.
propertyBlock.SetMatrix(“Bone” + i + “_Local_Matrix”, boneLocalMatrix);
}
//Apply the updated values to the renderer.
skinnedMeshRenderer.SetPropertyBlock(propertyBlock);
}
}
In the vertex stage stage of the shader I’m now able to access the following:
The saved base pose vertex position in UV1 passed as positionSavedLS.
The bone indices at this vertex in UV2 passed as vertexBoneIndices.
The bone weights at this vertex in UV3 passed as vertexBoneweights.
In the shader I multiply positionSavedLS by each bone local matrix, then multiply each result by the bone weight. Adding these together should hopefully give me positionSavedLS taking into account bone transformations. However, I’m not getting the expected result. (I know this makes no practical sense as the current vertex position in the shader would give me what I’m looking for. I’m simplifying things for this example.)
//Bone 0 weight.
if(vertexBoneweights.x != 0)
{
posWithBone0 = mul(bone0LocalMatrix, float4(positionSavedLS, 0));
positionSavedLSEdited += (posWithBone0 * vertexBoneweights.x);
}
//Bone 1 weight.
if(vertexBoneweights.y != 0)
{
posWithBone1 = mul(bone1LocalMatrix, float4(positionSavedLS, 0));
positionSavedLSEdited += (posWithBone1 * vertexBoneweights.y);
}
//Bone 2 weight.
if(vertexBoneweights.z != 0)
{
posWithBone2 = mul(bone2LocalMatrix, float4(positionSavedLS, 0));
positionSavedLSEdited += (posWithBone2 * vertexBoneweights.z);
}
//Bone 3 weight.
if(vertexBoneweights.w != 0)
{
posWithBone3 = mul(bone3LocalMatrix, float4(positionSavedLS, 0));
positionSavedLSEdited += (posWithBone3 * vertexBoneweights.w);
}
I can debug in the shader by forcing the current vertex position to use positionSavedLSEdited. When the bones are unchanged from the bind pose everything works as expected. The saved positions are where you would expect, taking into account the bones:
[198555-test-mesh-bones.jpg|198555]
However, as soon as the bones are adjusted things break:
[198556-text-mesh-bones-2.jpg*|198556]*
Clearly, something is wrong with my code to transform the saved vertex position, positionSavedLS, by the bones matrices and weights. Can anyone suggest a way to fix this? Any help is much appreciated.
*
*