RecalculateTangents breaks existing normal map

[hippocoder]

I have a mesh which is baked by Simplygon, and it looks fantastic. The problem is that Parallax Offset doesn’t work with this mesh unless I run it through Unity’s mesh.RecalculateTangents()

While this fixes the mesh for offset and any later projected decals, it does not fix the underlying original normals! In fact it breaks those.

I am asking for advice how to convert the old tangent normal map to the new mesh tangents? It’s the last thing I need to fix in my tool chain before i can begin work, so any advice is seriously appreciated.

I have a shader I use in the thread below to bake the mesh to a texture, so I used that with various attempts to reorient the old normal map to the new mesh tangents. The shader is in the link below, but I am not sure how to do tangent to ( new ) tangent conversion, or even if that’s right in this case.

Baking Decals into Texture - incorrect normals

I am baking a decal from world space into a mesh’s texture, and this works for colour, but the normal directions are all over the place. What works: The pixel is projected into the texture from world decal space to UV space correctly. It works fine for colour. What doesn’t: But the normals are oriented incorrectly. The pixels are in the right place in the target UV map, just I’m not sure how to rotate the final normal direction by whatever tangent to account for it, it’s not just a problem of…

However my experiments (all involving somehow trying to rotate old tangents to new tangents) have all failed.

float4 fragReorientedNormal(v2f IN) : SV_Target
{
    FRAGSTART

    float3 sourceNormalRT = UnpackNormal(tex2D(_sourceNormalRT, IN.uv));
    float3x3 m = cotangent_frame(IN.normal, IN.worldPos, IN.uv);

    //float3x3 tangentMatrix = float3x3(IN.worldT, IN.worldB, IN.worldN);
    float3 normal = mul(m, sourceNormalRT);
    return float4(normalize(normal), 1);
}

^ Current clueless bumbling

Help!

[hippocoder]

Perhaps I can bake the normals out in world space from Simplygon then convert world->tangent, but in past tests that went tits up too.

[hippocoder]

That didn’t work. Basically I just need to reorient the normal map as before in the other thread, but without projection, and I’m not sure what the rotation matrix looks like for that. Any help?

[bgolus]

If you have a tangent space normal map using tangent that no longer exist on the mesh and don’t match those Unity generates … then you’re fudged. There’s no way to transform from that tangent space to anything else if you don’t know the original tangent space you’re working from.

I would absolutely recommend using “world” (really object) space normals in that case if possible, as you don’t need the original tangent space then.

As for converting from world to tangent space, see this post:

Flat lighting without separate smoothing groups?

So I spent some time pondering this and realize what the problem is. The Transform node using World to Tangent is wrong, and has always been wrong. More over, so are all other examples of transforming from world space to tangent space. So, if you have a matrix that transforms from one space to another, and you need to go the opposite way, you need the inverse matrix. The usual view of a tangent to world matrix is that it'll be orthogonal, that is to say the normal, tangent, and bitangent are…

I can’t remember if I linked to that in the previous thread as the example for doing a reverse matrix or not. However as long as you transform the world normal from the texture with that matrix, it should perfectly match the tangent space Unity uses for the built in rendering paths.

[hippocoder]

Thanks for replying. Finally a lifejacket!

Firstly, I do have the original mesh, with tangents that work fine. I am only using Unity’s mesh.RecalculateTangents() on that mesh to try and fix Parallax Offset, since the original mesh has rotated UV islands when it is baked out.

So When I call RecalculateTangents, it invalidates the original normal map directions, which I presume are based on the mesh tangent directions for the rotated UV islands.

Knowing why I’m doing this, and the fact I have access to both sets of tangents (before and after) would that change your advice, and if so what would you recommend?

It’s all for Parallax Offset in the end (plus any other directional effects like running water I might add).

[bgolus]

You’d need both sets of mesh tangents accessible to the shader at the same time. As soon as you call RecalculateTangents() the original tangent vectors are going to be overwritten, and you don’t really want to keep them around in an extra UV set since they’re worthless after you’ve got the corrected tangent space normal map.

You could conceivably render using the mesh’s original tangents and normal map to a render texture to get the world space normals, call RecalculateTangents() and then do it again with the new mesh to transfer the world normals to the new tangent space. If you go that route I’d even recommend using an RGBAHalf render texture for the world normals to avoid too much precision loss. If you can export the Simplygon normals as 16 bpc that’d help a lot too, assuming you’re using a newer version of Unity that properly handles 16 bit png files.

[bgolus]

The real question for me is … why isn’t the parallax offset working with the original tangents. I can see problems with UV islands causing seams, but if you’re not redoing the UVs I don’t fully get why recalculating the tangents “fixes” it.

[hippocoder]

Seams are no problem, but the offset going in wildly different directions is. A real puzzle.

I gave the test model/scene to the Amplify guys and they were stumped. I can send it to you as well if you like. It’s simply a mesh that Simplygon baked from high to low poly, with a single normal map of the high poly. It displays fine until you attempt offset.

Ultimately I’ll probably try the world space normal bake thing, but I’m still incredibly curious as to WHY something with apparently perfectly respectable normals/tangents would defeat parallax offset. Let me know if you’d like to look at it too.

[bgolus]

I’m super curious actually.

[hippocoder]

I’ve sent a PM but feel free to share any findings in here too, there’s no IP there just a test model I have made.

I’ve also tried the following code to draw on a render texture using the mesh to generate a new texture:

float4 fragWorldToTangent(v2f IN) : SV_Target
{
    float3 worldNormal = UnpackNormal(tex2D(_sourceNormalRT, IN.uv));
    worldNormal = normalize(worldNormal);

    float3x3 worldToTangentRotation;
    worldToTangentRotation[0] = IN.worldB.yzx * IN.worldN.zxy - IN.worldB.zxy * IN.worldN.yzx;
    worldToTangentRotation[1] = IN.worldT.zxy * IN.worldN.yzx - IN.worldT.yzx * IN.worldN.zxy;
    worldToTangentRotation[2] = IN.worldT.yzx * IN.worldB.zxy - IN.worldT.zxy * IN.worldB.yzx;

    float det = dot(IN.worldT.xyz, worldToTangentRotation[0]);
    worldToTangentRotation *= rcp(det);

    float3 normal = mul(worldToTangentRotation, worldNormal);
    normal = normalize(normal);

    return float4(normal*0.5+0.5, 1);
}

Source world space normals (simplygon’s output) - I use linear and import as a regular texture not normal (I think that’s right?):

Result of code:

(please excuse the blit to scene view to test output)
Not quite sure what I have missed to cause the lingering out of range colours there.

[bgolus]

Okay. Wow. So the problem is Simplygon is outputting Direct3D oriented normal maps (-Y), and flipping the bitangent direction to deal with that. That means they don’t actually match the UV orientation, which means the parallax offset goes all sorts of wonky.

It also means the only thing you need to do to convert the normal maps from Simplygon to ones that work with the recalculated normals mesh is … invert the green channel.

[hippocoder]

@bgolus

Okay. Wow. So the problem is Simplygon is outputting Direct3D oriented normal maps (-Y), and flipping the bitangent direction to deal with that. That means they don’t actually match the UV orientation, which means the parallax offset goes all sorts of wonky.

It also means the only thing you need to do to convert the normal maps from Simplygon to ones that work with the recalculated normals mesh is … invert the green channel.

Wait, what?

That’s something I tried before a couple of times. I abandoned that path of trial and error because it was screwing up the projected to texture decals. I inverted Y as you have suggested and while it does fix the underlying normals, it has now played havoc with the decals. It seems I can’t quite catch a break.

I don’t use RecalculateNormals - those should be fine, right?

I’m doing RNM to blend the Simplygon generated normal map with the decal normal map. Seems lit wrong so I’m going over all the code to see if it was my fault.

But I guess this is the plan:

1. make simplygon export y inverted tangent normal map (this looks wrong in unity)
2. use unity’s RecalculateTangents (I’m open to a better idea) to fix offset (this now makes it look right in unity)
3. do my decal stuff, but this part’s broken for some reason (the lighting seems off, and it shouldn’t because they’re being reoriented as in the other thread)

At the seams it seems as if the prior work on making decals lit correctly is undone some how:

Ugh!

[hippocoder]

Update: somehow during frenzied night-time coding sessions I managed to turn:

float3x3 tangentMatrix = float3x3(IN.worldT, IN.worldB, IN.worldN);

into

float3x3 tangentMatrix = float3x3(IN.worldT, IN.worldB, IN.worldB);

Why I would consciously do that, I’ll never know. Never a dull moment around here. Thanks for helping, I think things will be OK now!

[bgolus]

How I got it to work was I flipped the normal map in Photoshop, then I reimported the mesh with Unity calculating the tangents, which produces the same result as calling RecalculateTangents().

Alternatively to RecalculateTangents() you could iterate over the mesh’s tangent array and invert all of the w components, which would end up achieving basically identical results give or take some possible minor floating point differences. As best I could tell the mesh’s existing tangents and those after reimport where effectively identical apart from that bitangent direction, which the w component’s sign controls (it’s always only every +1.0 or -1.0). I didn’t compare the values directly, but also couldn’t see a difference between the original mesh & normal map compared to the reimported one & flipped normal map, meaning Simplygon is at least using MikkTSpace.