So I spent some time pondering this and realize what the problem is. The Transform node using World to Tangent is wrong, and has always been wrong.
More over, so are all other examples of transforming from world space to tangent space.
So, if you have a matrix that transforms from one space to another, and you need to go the opposite way, you need the inverse matrix. The usual view of a tangent to world matrix is that it’ll be orthogonal, that is to say the normal, tangent, and bitangent are all perfectly perpendicular to each other. The nice thing with an orthogonal matrix is the inverse is the transpose matrix, and in shaders you can trivially apply a matrix as transposed by flipping the order of the matrix and the vector in the mul() function.
Basically, if the tangent to world matrix (represented here as tbn) is orthogonal, the below is true:
worldSpaceVector == mul(mul(tbn, worldSpaceVector), tbn);
The problem is … tangent space isn’t orthogonal!
Much of the time it’s close enough to not be obviously wrong, so the above examples for surface shaders and for shader graph will look correct enough on some geometry, but more extreme cases it will not.
To do this properly, you need to calculate the real inverse matrix of the tangent to world matrix.
For a Surface Shader it looks like this (here with a drop down to let you choose between modes, using stolen modified code from UnityStandardParticleInstancing.cginc):
Shader "Custom/FlatSurfaceShader" {
Properties {
_Color ("Color", Color) = (1,1,1,1)
_MainTex ("Albedo (RGB)", 2D) = "white" {}
[KeywordEnum(Approximate, Exact)] _InverseMatrix ("World To Tangent Matrix", Float) = 0.0
}
SubShader {
Tags { "RenderType"="Opaque" }
CGPROGRAM
#pragma surface surf Standard fullforwardshadows vertex:vert
#pragma target 3.0
#pragma shader_feature _ _INVERSEMATRIX_EXACT
sampler2D _MainTex;
struct Input {
float2 uv_MainTex;
float3 cameraRelativeWorldPos;
float3 worldNormal;
INTERNAL_DATA
};
half _Glossiness;
half _Metallic;
fixed4 _Color;
// pass camera relative world position from vertex to fragment
void vert(inout appdata_full v, out Input o)
{
UNITY_INITIALIZE_OUTPUT(Input,o);
o.cameraRelativeWorldPos = mul(unity_ObjectToWorld, float4(v.vertex.xyz, 1.0)) - _WorldSpaceCameraPos.xyz;
}
void surf (Input IN, inout SurfaceOutputStandard o) {
fixed4 c = tex2D (_MainTex, IN.uv_MainTex);
o.Albedo = c.rgb * _Color.rgb;
#if !defined(UNITY_PASS_META)
// flat world normal from position derivatives
half3 flatWorldNormal = normalize(cross(ddy(IN.cameraRelativeWorldPos.xyz), ddx(IN.cameraRelativeWorldPos.xyz)));
// construct world to tangent matrix
half3 worldT = WorldNormalVector(IN, half3(1,0,0));
half3 worldB = WorldNormalVector(IN, half3(0,1,0));
half3 worldN = WorldNormalVector(IN, half3(0,0,1));
#if defined(_INVERSEMATRIX_EXACT)
// inverse transform matrix
half3x3 w2tRotation;
w2tRotation[0] = worldB.yzx * worldN.zxy - worldB.zxy * worldN.yzx;
w2tRotation[1] = worldT.zxy * worldN.yzx - worldT.yzx * worldN.zxy;
w2tRotation[2] = worldT.yzx * worldB.zxy - worldT.zxy * worldB.yzx;
half det = dot(worldT.xyz, w2tRotation[0]);
w2tRotation *= rcp(det);
#else
half3x3 w2tRotation = half3x3(worldT, worldB, worldN);
#endif
// apply world to tangent transform to flat world normal
o.Normal = mul(w2tRotation, flatWorldNormal);
#endif
}
ENDCG
}
FallBack "Diffuse"
}
And for Shader Graph that’s just …
Aahhhg!
So much clutter. 