Incorrect normals on after rotating instances [Graphics.DrawMeshInstancedIndirect]

Unity Engine
1

Hello,

I have some issues rotating instances created with Graphics.DrawMeshInstancedIndirect.

I store per-instance position, scale and rotation in ComputeBuffer and apply them to the instances in setup() shader function as described here - Unity - Scripting API: Graphics.DrawMeshInstancedIndirect
The positioning and scaling of the instances works fine, and the rotation also works, in a way.
However, as you can see in the attached image, it seems that the normals of the instance are not rotated along with the object and as a result rotated objects are lit incorrectly.

I’ve attached the relevant portion of the setup() code below that deals with the positioning and rotation of the instances. Possibly I’m going about this all wrong by multiplying unity_ObjectToWorld with a rotation matrix, but I do see correct rotations of the instances, so it kind of works.

I have a hunch that I also need to transform the vertices or normals somehow, but I can’t figure out what transformation to apply there, does anyone have any ideas what could be going wrong here?

        void setup()
        {
    #ifdef UNITY_PROCEDURAL_INSTANCING_ENABLED
            int idx = CulledBuffer[unity_InstanceID].index;
            int instance = CulledBuffer[unity_InstanceID].instance;
            GPUAtom atom = AtomBuffer[idx];
            float size = lerp(atom.scaleP, atom.scale, atom.lerpProgress);
            float health = lerp(atom.healthP, atom.health, atom.lerpProgress);
            float angle = lerp(atom.angleP, atom.angle, atom.lerpProgress);
            float3 position = lerp(atom.positionP, atom.position, atom.lerpProgress);

            // scale
            unity_ObjectToWorld._11_21_31_41 = float4(size, 0, 0, 0);
            unity_ObjectToWorld._12_22_32_42 = float4(0, size, 0, 0);
            unity_ObjectToWorld._13_23_33_43 = float4(0, 0, size, 0);

            // position
            unity_ObjectToWorld._14_24_34_44 = float4(position, 1);

            // rotation
            float c = cos(angle);
            float s = sin(angle);
            float4x4 rotateYMatrix = float4x4(
                c, 0, s, 0,
                0, 1, 0, 0,
                -s, 0, c, 0,
                0, 0, 0, 1);
            unity_ObjectToWorld = mul(unity_ObjectToWorld, rotateYMatrix);

            unity_WorldToObject = unity_ObjectToWorld;
            unity_WorldToObject._14_24_34 *= -1;
            unity_WorldToObject._11_22_33 = 1.0f / unity_WorldToObject._11_22_33;

    #endif
        }

974×579 562 KB

2

Try calculating the inverse matrix for the unity_WorldToObject

float4x4 inverse(float4x4 input)
{
#define minor(a,b,c) determinant(float3x3(input.a, input.b, input.c))

    float4x4 cofactors = float4x4(
        minor(_22_23_24, _32_33_34, _42_43_44),
        -minor(_21_23_24, _31_33_34, _41_43_44),
        minor(_21_22_24, _31_32_34, _41_42_44),
        -minor(_21_22_23, _31_32_33, _41_42_43),

        -minor(_12_13_14, _32_33_34, _42_43_44),
        minor(_11_13_14, _31_33_34, _41_43_44),
        -minor(_11_12_14, _31_32_34, _41_42_44),
        minor(_11_12_13, _31_32_33, _41_42_43),

        minor(_12_13_14, _22_23_24, _42_43_44),
        -minor(_11_13_14, _21_23_24, _41_43_44),
        minor(_11_12_14, _21_22_24, _41_42_44),
        -minor(_11_12_13, _21_22_23, _41_42_43),

        -minor(_12_13_14, _22_23_24, _32_33_34),
        minor(_11_13_14, _21_23_24, _31_33_34),
        -minor(_11_12_14, _21_22_24, _31_32_34),
        minor(_11_12_13, _21_22_23, _31_32_33)
        );
#undef minor
    return transpose(cofactors) / determinant(input);
}


unity_WorldToObject = inverse(unity_ObjectToWorld);
1 Like
3

My heartfelt thanks to you, Lennart :slight_smile:
That worked, I’m getting correct results now.

And I think I also understand the reason why it didn’t work before - please correct me if I’m wrong.
The Unity’s sample that I used cheaply generates the WorldToObject matrix by simply inverting the position and scale of ObjectToWorld. But if ObjectToWorld is rotated, that trick won’t work and the inverse matrix needs to be built up the hard (and correct) way.

4

Yes. That was probably it. I am guessing your shader used the worldToObject matrix in some normal calc.

If you have static items you could calculate the inverse matrix before and add that to a compute buffer with the object2World matrix.

5

Btw if you are always using a uniform scaling like in your pasted code, you can try:
#pragma instancing_options assumeuniformscaling
It would probably eliminate the use of unity_WorldToObject if you are using Standard lighting model.

6

Hi,

can you by any chance show the structure you use for this compute buffer? (I’m guessing it’s your AtomBuffer?)

I am currently passing positions and rotations in two seperate ComputeBuffers (using a Vector3 array and a Quaternion array respectively) and in the shader using a quaternion to matrix algorithm before setting the unity_ObjectToWorld. Problem is, I can’t get rotations right as the Quartenion.Identity results in a top-down rotated mesh. I hape suspiccions that I hit a gimbal lock somehow, but I am not sure. Below is the relevant bit of my shader code, any help would be appreciated:

#ifdef UNITY_PROCEDURAL_INSTANCING_ENABLED
    StructuredBuffer<float3> positions;
    StructuredBuffer<float4> quaternions;
#endif

void setup()
{
    #ifdef UNITY_PROCEDURAL_INSTANCING_ENABLED
        float3 position     = positions[unity_InstanceID];
        float4 q             = quaternions[unity_InstanceID];
        float qr            = q[0];
        float qi            = q[1];
        float qj            = q[2];
        float qk            = q[3];

        float4x4 rotation;
        float4x4 translation = {
            1,0,0,position.x,
            0,1,0,position.y,
            0,0,1,position.z,
            0,0,0,1
        };

        // quaternion to matrix
        // http://www.euclideanspace.com/maths/geometry/rotations/conversions/quaternionToMatrix/
        // https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotation#Quaternion-derived_rotation_matrix

        rotation[0][0]            = 1.0f - 2.0f*qj*qj - 2.0f*qk*qk;
        rotation[0][1]            = 2.0f*(qi*qj - qk*qr);
        rotation[0][2]            = 2.0f*(qi*qk + qj*qr);
        rotation[0][3]            = 0.0f;

        rotation[1][0]            = 2.0f*(qi*qj+qk*qr);
        rotation[1][1]            = 1.0f - 2.0f*qi*qi - 2.0f*qk*qk;
        rotation[1][2]            = 2.0f*(qj*qk - qi*qr);
        rotation[1][3]            = 0.0f;

        rotation[2][0]            = 2.0f*(qi*qk - qj*qr);
        rotation[2][1]            = 2.0f*(qj*qk + qi*qr);
        rotation[2][2]            = 1.0f - 2.0f*qi*qi - 2.0f*qj*qj;
        rotation[2][3]            = 0.0f;

        rotation[3][0]            = 0.0f;
        rotation[3][1]            = 0.0f;
        rotation[3][2]            = 0.0f;
        rotation[3][3]            = 1.0f;

        unity_ObjectToWorld = mul(translation, rotation);
     
         
        // inverse transform matrix
        // taken from richardkettlewell's post on
        // https://forum.unity3d.com/threads/drawmeshinstancedindirect-example-comments-and-questions.446080/

        float3x3 w2oRotation;
        w2oRotation[0] = unity_ObjectToWorld[1].yzx * unity_ObjectToWorld[2].zxy - unity_ObjectToWorld[1].zxy * unity_ObjectToWorld[2].yzx;
        w2oRotation[1] = unity_ObjectToWorld[0].zxy * unity_ObjectToWorld[2].yzx - unity_ObjectToWorld[0].yzx * unity_ObjectToWorld[2].zxy;
        w2oRotation[2] = unity_ObjectToWorld[0].yzx * unity_ObjectToWorld[1].zxy - unity_ObjectToWorld[0].zxy * unity_ObjectToWorld[1].yzx;

        float det = dot(unity_ObjectToWorld[0], w2oRotation[0]);

        w2oRotation = transpose(w2oRotation);

        w2oRotation *= rcp(det);

        float3 w2oPosition = mul(w2oRotation, -unity_ObjectToWorld._14_24_34);

        unity_WorldToObject._11_21_31_41 = float4(w2oRotation._11_21_31, 0.0f);
        unity_WorldToObject._12_22_32_42 = float4(w2oRotation._12_22_32, 0.0f);
        unity_WorldToObject._13_23_33_43 = float4(w2oRotation._13_23_33, 0.0f);
        unity_WorldToObject._14_24_34_44 = float4(w2oPosition, 1.0f);
 
    #endif
}
7

You are great man, thank you from 2021!

8

I’m confused here. At what line do you paste this code at?