Data stored in texcoord behaves differently on Mesh/Skinned Mesh

I am a beginner working on a revision version of outline/Toon shader base on this nice tutorial

Where inside a paragraph it mentioned a way to Handeling sharp edges by calculating a “Smoothed” version of Object Normal and stores it in a seperated texcoord

After several dig in the web I was able to write an Editor script that handelling normal calculation of each import, and stores the smoothed normal into texcoord1(uv2)

Everything behaves normal on a regular mesh: I get all the sharedMeshs from MeshFilter by using OnPostprocessModel(GameObject), and manual calculate and store the smoothed information by Mesh.SetUVs(1, arrayOfVertexAveragedNormals).

(I uses a custom shader to render the preview for origional normal NORMAL and smoothed normal from TEXCOORD1)

Preview for the Origional Normal (The normal of the model was intentionally modified to show a difference)


--------------------------------------------
Preview for the Smoothed Normal

--------------------------------------------

Afterward I simply grab a Unity-chan model from the Asset Store, then apply the same application on the sharedMesh of its SkinnedMeshRender, however this time the result was… different.

Preview for the Origional Normal
6070560--658119--upload_2020-7-9_15-26-21.png
--------------------------------------------
Preview for the Smoothed Normal
6070560--658116--upload_2020-7-9_15-25-7.png
--------------------------------------------

The NORMAL behaves, just the right way, however, the smoothed information stored inside the TEXCOORD1 was… somewhate not updated, notice the arm of the character.
6070560--658131--upload_2020-7-9_15-38-38.png
Here’s the code I uses for generating smoothed normal at run time and the shader I used for previewing.

using System.Collections.Generic;
using System.Linq;
using UnityEngine;
using UnityEditor;
using System;

public class ProcessNormals : AssetPostprocessor
{
    private void OnPostprocessModel(GameObject importedModel)
    {
        foreach (var item in importedModel.GetComponentsInChildren<MeshFilter>())
        {
            PreProcessNormals(item.sharedMesh);
        }
        foreach (var item in importedModel.GetComponentsInChildren<SkinnedMeshRenderer>())
        {
            PreProcessNormals(item.sharedMesh);
        }
        ModelImporter modelImporter = assetImporter as ModelImporter;
        modelImporter.isReadable = false;
    }

    private void OnPreprocessModel()
    {
        ModelImporter modelImporter = assetImporter as ModelImporter;
        modelImporter.isReadable = true;
    }

    private Mesh PreProcessNormals(Mesh toProcess)
    {
        var origionalNormal = toProcess.normals;
       
        Vector3[] meshVertices = toProcess.vertices;
        //map vertex positions to the ids of all vertices at that position
        Dictionary<Vector3, List<int>> vertexMerge = new Dictionary<Vector3, List<int>>();
        for (int i = 0; i < toProcess.vertexCount; i++)
        {
            Vector3 vectorPosition = meshVertices[i];

            if (!vertexMerge.ContainsKey(vectorPosition))
            {
                //if not already in our collection as a key, add it as a key
                vertexMerge.Add(vectorPosition, new List<int>());
            }

            //add the vertex id to our collection
            vertexMerge[vectorPosition].Add(i);
        }

        //map vertexIDs to the averaged normal
        Vector3[] meshNormals = toProcess.normals;
        Vector3[] vertexAveragedNormals = new Vector3[toProcess.vertexCount];

        foreach (List<int> duplicatedVertices in vertexMerge.Values)
        {
            //calculate average normal
            Vector3 sumOfNormals = Vector3.zero;
            foreach (int vertexIndex in duplicatedVertices)
            {
                sumOfNormals += meshNormals[vertexIndex];
            }

            Vector3 averagedNormal = (sumOfNormals /= duplicatedVertices.Count).normalized; //average is sum divided by the number of summed elements

            //write the result to our output
            foreach (int vertexIndex in duplicatedVertices)
            {
                vertexAveragedNormals[vertexIndex] = averagedNormal;
            }
        }

        //toProcess.colors = vertexAveragedNormals.Select(x => new Color(x.x,x.y,x.z)).ToArray();
        //toProcess.SetNormals(vertexAveragedNormals);
        toProcess.SetUVs(1, vertexAveragedNormals);
        var debug = origionalNormal.Zip
            (
            vertexAveragedNormals,
            (first, second) => (first - second != Vector3.zero) ? $"({Math.Round(first.x, 2)},{Math.Round(first.y, 2)},{Math.Round(first.z, 2)})" + "\t\t" + $"({Math.Round(second.x, 2)},{Math.Round(second.y, 2)},{Math.Round(second.z, 2)})" : null
            ).Where(x => x != null);
        Debug.Log("Processed" + (debug.Count() != 0 ? "\n" + debug.Aggregate((first, second) => first + "\n" + second) : " Null"));
        return toProcess;
    }
}
Shader "Custom/RenderObjectTex"
{
    Properties
    {
    }
    SubShader
    {
        Tags { "RenderType"="Opaque" }
        LOD 100

        Pass
        {
           
            Cull Back

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"


            struct appdata{
                float4 vertex : POSITION;
                //float3 normal : NORMAL;
                float3 normal : TEXCOORD1;
            };


            struct v2f
            {
                float4 pos : SV_POSITION;
                float3 color : TEXCOORD1;
            };


            v2f vert (appdata v)
            {
                v2f o;
                //o.pos = UnityObjectToClipPos (v.vertex + (v.normal) * 0.02);
                o.pos = UnityObjectToClipPos (v.vertex);
                //o.color = UnityObjectToWorldDir(v.normal).xyz;
                o.color = mul((float3x3) UNITY_MATRIX_VP, mul((float3x3) UNITY_MATRIX_M, v.normal)) * 0.5 + 0.5;
                //o.color = v.normal;
                return o;
            }
           
            half4 frag (v2f i) : COLOR
            {
                return half4 (i.color, 1);
            }
            ENDCG
        }
    }
}

Many modelling tools out there do not use the same Y up axis as Unity. Some use Z up. Skinned meshes that were created in a Z up tool will be imported with that orientation, but will appear in game properly Y up, but this is only because Unity updates the animated bone positions to match the Y up orientation, but not the original model data.

This is important because Unity’s skinning only updates the vertex position, normal, and tangent data. If you have a normal encoded in the vertex color or UV, the skinning doesn’t have any way to know how to modify that data, so it remains in the original orientation of the mesh’s base pose.

For doing what you’re doing to skinned meshes, the common approach is to encode the smoothed normal into the mesh’s tangent, but this comes at the cost of no longer allowing you to use traditional normal maps. That can be worked around by using a custom shader that uses derivatives to calculate the tangent to world space transform needed for normal maps. The other option is you can store your smoothed normal vectors as tangent space normal vectors, which would let you reconstruct the object or world space smoothed normal from the skinned mesh’s normal and tangent.

*edit: Some further information about Unity’s skinning. It’s done either on the CPU or on the GPU with a “stream out” shader, both of which are hidden from the user. Basically the shader above is being applied to a static pre-deformed mesh that no longer has any information regarding the original skinning or bones, so there’s no way to “fix” custom normal data in a shader.

6 Likes

Thank you So much for offering this support, I found very limited information on this.