get skinned vertices in real time

[benblo]

I need to get the vertices of a skinned mesh as they are deformed… problem is, the SkinnedMeshRenderer component only exposes the sharedMesh, which isn’t updated (it’s the asset itself, not the mesh instance).
Am I missing something?

The MeshFilter component exposes both the mesh and sharedMesh, but from what I see it’s not in synch with the SkinnedMeshRenderer.
I found trace of a SkinnedMeshFilter, which sounds like it would be just what I need, but I fear it’s an obsolete piece of code (doesn’t compile, SkinnedMeshFilter is unknown… I never heard of it before either).

I’d be okay with reinventing the wheel, ie manually calculating the vertices positions using the bones, if anyone knows the algorithm… some sort of barycenter of the bones’ positions balanced by their weights?

[benblo]

Okay… so apparently it is a simple barycenter or whatever it’s called, script below will draw gizmos on each vertex in real time.

If someone knows a less crazy solution, I’m still interested though… I’ll log a “gimme the mesh instance dammit” feature request when it’s not 2am.

using UnityEngine;
using System.Collections;
using System.Collections.Generic;

public class SkinnedVertices : MonoBehaviour
{
	Mesh mesh;

	class Bone
	{
		internal Transform bone;
		internal float weight;
		internal Vector3 delta;
	}
	List<List<Bone>> allBones = new List<List<Bone>>();

	void Start()
	{
		SkinnedMeshRenderer skin = GetComponent(typeof(SkinnedMeshRenderer)) as SkinnedMeshRenderer;
		mesh = skin.sharedMesh;

		Debug.Log("{0} vertices, {1} weights, {2} bones",
			mesh.vertexCount, mesh.boneWeights.Length, skin.bones.Length);

		for (int i = 0; i < mesh.vertexCount; i++)
		{
			Vector3 position = mesh.vertices[i];
			position = transform.TransformPoint(position);

			BoneWeight weights = mesh.boneWeights[i];
			int[] boneIndices = new int[] { weights.boneIndex0, weights.boneIndex1, weights.boneIndex2, weights.boneIndex3 };
			float[] boneWeights = new float[] { weights.weight0, weights.weight1, weights.weight2, weights.weight3 };

			List<Bone> bones = new List<Bone>();
			allBones.Add(bones);

			for (int j = 0; j < 4; j++)
			{
				if (boneWeights[j] > 0)
				{
					Bone bone = new Bone();
					bones.Add(bone);

					bone.bone = skin.bones[boneIndices[j]];
					bone.weight = boneWeights[j];
					bone.delta = bone.bone.InverseTransformPoint(position);
				}
			}

			//if (bones.Count > 1)
			//{
			//    string msg = string.Format("vertex {0}, {1} bones", i, bones.Count);

			//    foreach (Bone bone in bones)
			//        msg += string.Format("\n\t{0} => {1} => {2}", bone.bone.name, bone.weight, bone.delta);

			//    Debug.Log(msg);
			//}
		}
	}

	void OnDrawGizmos()
	{
		if (Application.isPlaying  enabled)
		{
			for (int i = 0; i < mesh.vertexCount; i++)
			{
				List<Bone> bones = allBones[i];

				Vector3 position = Vector3.zero;
				foreach (Bone bone in bones)
					position += bone.bone.TransformPoint(bone.delta) * bone.weight;

				int boneCount = bones.Count;
				Gizmos.color = (boneCount == 4) ? Color.red :
					(boneCount == 3) ? Color.blue :
					(boneCount == 2) ? Color.green : Color.black;

				Gizmos.DrawWireCube(position, boneCount * 0.05f * Vector3.one);

				Vector3 normal = Vector3.zero;
				foreach (Bone bone in bones)
					normal += bone.bone.TransformDirection(mesh.normals[i]) * bone.weight;

				Gizmos.DrawRay(position, normal);
			}
		}
	}
}

[benblo]

Damn… now how do I get the skinned normals?? sounds trickier…

[anon_51268168]

I think the skinned mesh data is written to a special write-only portion of memory for performance, or something. I brought this up a while ago but can’t find the thread. At any rate, it’s like how it is for a reason.

Cheers,
-Jon

[Dreamora]

especially with the higher end cards and the pure shader engine, its likely not written anywhere outside the GPU RAM, so out of your reach.

it only works if it falls back to CPU animation mode where the data are required to be on RAM (basically only Intel CPU to DVI adapters, that do not have any acceptable hardware vertex transformation performance if any support at all X3100+, the 900 / 950 don’t have it)

[StarManta]

@""

Damn… now how do I get the skinned normals?? sounds trickier…

It shouldn’t be. Normals should be exactly the same as vertices as far as the math is concerned.

[benblo]

Thanks Jon, it makes more sense… still sucks though.

@""

@""

Damn… now how do I get the skinned normals?? sounds trickier…

It shouldn’t be. Normals should be exactly the same as vertices as far as the math is concerned.

To transform the vertex I do this

Vector3 position = Vector3.zero;
foreach (Bone bone in bones)
    position += bone.bone.TransformPoint(bone.delta) * bone.weight;

I just don’t see what math to do to transform the normal… I’m probably just too tired though, it’s 4am here

Edit: just played Night of the Cephalopods, pretty cool

[StarManta]

It should be the same, just using TransformDirection instead of TransformPoint.

[llavigne]

@""

especially with the higher end cards and the pure shader engine, its likely not written anywhere outside the GPU RAM, so out of your reach.

it only works if it falls back to CPU animation mode where the data are required to be on RAM (basically only Intel CPU to DVI adapters, that do not have any acceptable hardware vertex transformation performance if any support at all X3100+, the 900 / 950 don’t have it)

Unity compute skin on the cpu only.

[benblo]

@""

It should be the same, just using TransformDirection instead of TransformPoint.

Well… that defies my sense of space (which is pretty crappy to start with), but you’re right it works, thanks a bunch!
FWIW, I edited the script above to gizmo normals.

@""

Unity compute skin on the cpu only.

So… which is it? UT, any chance to have a definitive answer on this? (David I know you’re out there, stop hiding ;))

[Aras]

@""

So… which is it? UT, any chance to have a definitive

Skinning is currently done on the CPU.CPU.

[benblo]

Thanks Aras… so why can’t I get the deformed mesh? (dammit please)
Could you explain what Jon hinted at? and more importantly, will that ever change? (after Windows dammit)

Edit: because right now it kinda works but I seem to have a small offset between the rendered mesh and some vertices I compute, and some normals are way out… plus this reinventing of the wheel is really inelegant, and I like my code pretty.

[Aras]

@""

Thanks Aras… so why can’t I get the deformed mesh? (dammit please)

Because the vertices indeed are directly written into a “dynamic vertex buffer”. That memory usually sits on AGP or PCIe bus, and writing to it is okay, but reading is slow. Possible, but slow.

and more importantly, will that ever change? (after Windows dammit)

Will it ever be implemented - probably. What we could do, I guess:

• Whenever skinned result is requested, compute skinning at that time. So each reading of the skin would compute the whole result.

• Or, the ability to say to the skinned mesh renderer “I’ll need skinned results, keep them around please”. Then it would not skin directly into AGP/PCIe memory, but instead into a temporary buffer. Which then would be copied to AGP/PCIe. And temporary buffer could be returned to your code.

Now, when that will be implemented - good question. I don’t know. If enough people shout “I need it dammit”, then it will be sooner. The problem is that people shout “I need this dammit” for thousand different things, and we have to sort them somehow

[benblo]

Thanks! So I guess both parties were right, it’s kind of in between, good to know it’s doable in theory.

BTW, any idea why I have an offset (I’m working on LateUpdate)? does my algorithm seem correct?

@Aras Pranckevicius

Now, when that will be implemented - good question. I don’t know. If enough people shout “I need it dammit”, then it will be sooner. The problem is that people shout “I need this dammit” for thousand different things, and we have to sort them somehow

Gee, this is surprising! :lol:
After all you only had 15000 bugs reported in the last month… slackers.

[Aras]

@""

BTW, any idea why I have an offset (I’m working on LateUpdate)? does my algorithm seem correct?

I’m not sure if it’s correct, specifically the “deltas” part. Maybe it is, I did not write down the math and see if it matches.

Basically, the skinning is this:

result =
  matrices[bone0]*value*weight0 +
  // ...
  matrices[boneN]*value*weightN;

Where ‘matrices’ would be a matrix array. Each matrix in the array is:

currentBoneMatrix * bindPose

The ‘bind pose’ matrices is nothing more than just an inverse matrix of the bone at mesh export time - i.e. the matrix that transforms from mesh-as-it-is to local space of the bone. Then current bone matrix takes from local space and transforms into world space.

Now, the above skinning code can of course be rewritten as:

finalMatrix =
    matrices[bone0] * weight0 +
    // ...
    matrices[boneN] * weightN;
result = finalMatrix * value;

here, matrices[bone]*weight just multiplies all components of the matrix by a number. Basically, it’s fine to “blend” the matrices first. Whichever is faster, depends on the situation.

[andeeeee]

@Aras Pranckevicius

What we could do, I guess:

• Whenever skinned result is requested, compute skinning at that time. So each reading of the skin would compute the whole result.

This would actually be useful even if it is slow. For example, you might want to emit particles from the surface of a skinned mesh object to simulate an explosion, disintegration or whatever. I would have found this useful on a couple of occasions.

[anon_5991062]

@Aras Pranckevicius

Basically, the skinning is this:

result =
  matrices[bone0]*value*weight0 +
  // ...
  matrices[boneN]*value*weightN;

This won’t work for me, seems it’s not possible to multiply a Matrix4x4 with a float?

/P

[Aras]

That was pseudocode Multiplying a matrix by a float here is multiplying all matrix elements by that float.

[anon_5991062]

thanks

Adding the matrices together is also done on the element level then?

/Patrik

[Aras]

Yes.