Visualize skining weights with a shader

[Dzhange]

Hi guys,

I’m new to unity shader and not really knowing if it is possible to do so.
Here is my question: I have a skinned mesh in unity, and I would like to visualize the skinning weights from each bone for each pixel, with a grayscale image. For example, if the skinned mesh has 8 bones, it would output 8 different grayscale images, each corresponds to a bone, and the value of each pixel represents the skinned weights.

I really don’t know if this could be accomplished with the shader(and how to implement that), could anyone give me some sketch?
If this is not possible or extremely hard, are there any alternatives?

Thanks!

[Invertex]

So you want to see a square texturemap for each bone? Or, more simply, would you just like to visualize the weighting as colors on the actual mesh surface…

The main issue regardless is that with GPU skinning enabled, Unity is simply using a vertex shader to transform the mesh data and then renders that transformed mesh structure with your material, so the vertex weights aren’t seen by your shader. I believe there are some custom GPU skinning assets you could take knowledge from to manually submit and interact with the skinning data instead, otherwise your option is to do that on CPU side or submit the weighting data to a compute shader to generate textures to view.

[Dzhange]

@Invertex

So you want to see a square texturemap for each bone? Or, more simply, would you just like to visualize the weighting as colors on the actual mesh surface…

The main issue regardless is that with GPU skinning enabled, Unity is simply using a vertex shader to transform the mesh data and then renders that transformed mesh structure with your material, so the vertex weights aren’t seen by your shader. I believe there are some custom GPU skinning assets you could take knowledge from to manually submit and interact with the skinning data instead, otherwise your option is to do that on CPU side or submit the weighting data to a compute shader to generate textures to view.

Thanks for your reply! Actually I’m considering generating a dataset with unity, so I think doing this on CPU side makes sense. Previously I’ve been thinking about submitting weighting data to the shader, but I couldn’t find any document talking submitting data other than vertex geometry (and uv) to the shader, do you know any method to do so?

Another way I’m thinking about is to create a texture of skinning weight offline and use that for rendering. but I’m stuck with retrieving the bone weights from the mesh… the GetBoneWeights function always give me an empty list, I’m trying to find out why this is happening, do you have any clue about this?

Thank you again!

[Invertex]

@Dzhange

Thanks for your reply! Actually I’m considering generating a dataset with unity, so I think doing this on CPU side makes sense. Previously I’ve been thinking about submitting weighting data to the shader, but I couldn’t find any document talking submitting data other than vertex geometry (and uv) to the shader, do you know any method to do so?

Another way I’m thinking about is to create a texture of skinning weight offline and use that for rendering. but I’m stuck with retrieving the bone weights from the mesh… the GetBoneWeights function always give me an empty list, I’m trying to find out why this is happening, do you have any clue about this?

Thank you again!

It should be working, unless perhaps you’re using the “Unlimited” weights mode, in which case you might have to use this newer method:
https://docs.unity3d.com/ScriptReference/Mesh.GetAllBoneWeights.html

As for submitting data to shader, yes you could bind a StructuredBuffer to assign the weights to, or even assign the weights as a texture which I’ve seen done for some stuff before.

[Dzhange]

@Invertex

It should be working, unless perhaps you’re using the “Unlimited” weights mode, in which case you might have to use this newer method:
https://docs.unity3d.com/ScriptReference/Mesh.GetAllBoneWeights.html

As for submitting data to shader, yes you could bind a StructuredBuffer to assign the weights to, or even assign the weights as a texture which I’ve seen done for some stuff before.

Thanks! I kind of solved the bone weight problem, it seems there is something strange that happened in the LeapMotion extension I’m using. I made a texture with bone weights, but the problem is that the texture only contains per-vertex information, and I want it to cover the whole hand, do you have any idea of doing so? I’m looking into the Structured Buffer, maybe I can pass the boneweight to shader as vertex colors.

[Invertex]

@Dzhange

Thanks! I kind of solved the bone weight problem, it seems there is something strange that happened in the LeapMotion extension I’m using. I made a texture with bone weights, but the problem is that the texture only contains per-vertex information, and I want it to cover the whole hand, do you have any idea of doing so? I’m looking into the Structured Buffer, maybe I can pass the boneweight to shader as vertex colors.

It would be more efficient to leave it as per-vertex in the texture, and then in your shader, sample the correct pixel from your texture inside of the vertex program of the shader and assign it to a float4 in your v2f (vertex to fragment) struct, this value will then get interpolated across the triangle, giving you a whole hand coloring.
You can get the ID of the vertex your vertex program is currently processing by having uint vertID : SV_VertexID; in your Input/appdata struct.

You can use tex2Dlod() to sample in the vertex program.

This way you won’t need to render a large full UV texture, just a tiny texture that only has as many pixels as there are verts. Much less data to pass around and take up memory.