SphereCast & CapsuleCast RaycastHit.normal is not the surface normal as the documentation states...

Unity Engine
1

So the documentation for RaycastHit says that the ‘normal’ property is the normal of the surface hit by the cast:

But this actually isn’t true in the case of SphereCast and CapsuleCast (underneath SphereCast is actually a CapsuleCast, so DotPeek enlightened me to).

Instead the normal returned is the inverse normal of the capsule/sphere that hit the normal.

This is fine when hitting a flat surface head on, as the surface normal, and the inverse normal of the sphere/capsule are identical.

But if you cast at an edge, and the sphere/capsule hit slightly off. You get something like:

From what I can tell this is the expected value to come back as per PhysX documentation. Which basically just says to me Unity just misdocumented the RaycastHit.normal in their documentation, and that it’s not actually a bug. Which would make sense, because RaycastHit in the case of Raycast DOES return the surface normal. It just doesn’t for Capsule/Sphere cast.

Furthermore, this normal can be very useful for various things.

BUT, the actual surface normal would be useful as well.

Currently I have a function to get the surface normal. What I do is check if the hit collider was a MeshCollider and calculate the surface normal based on the mesh data. If it’s not a MeshCollider than I do a short Raycast at the surface to calculate the surface normal. The latter of which means 2 consecutive casts each time AND a slight potential for error.

You can see this here:

        public static Vector3 RepairHitSurfaceNormal(RaycastHit hit, int layerMask)
        {
            if(hit.collider is MeshCollider)
            {
                var collider = hit.collider as MeshCollider;
                var mesh = collider.sharedMesh;
                var tris = mesh.triangles;
                var verts = mesh.vertices;

                var v0 = verts[tris[hit.triangleIndex * 3]];
                var v1 = verts[tris[hit.triangleIndex * 3 + 1]];
                var v2 = verts[tris[hit.triangleIndex * 3 + 2]];

                var n = Vector3.Cross(v1 - v0, v2 - v1).normalized;

                return hit.transform.TransformDirection(n);
            }
            else
            {
                var p = hit.point + hit.normal * 0.01f;
                Physics.Raycast(p, -hit.normal, out hit, 0.011f, layerMask);
                return hit.normal;
            }
        }

I was wondering what the rest of you do? If you use a similar method to me, or if maybe there’s a BETTER option?

9 Likes
2

I just noticed this behavior, and it sort of makes sense. I mean I see the logic in that, but I see normals that would be impossible from a capsulecast. I also see hit points that are impossible (for instance, my hit point doesn’t even rest anywhere on the collider, just close to it)

Either way, I am pretty sure the problem I have is the exact same issue. Been using SphereCasts and CapsuleCasts for a long while now, and never had this problem until today.

I’ll try your code; it looks solid. The only thing I would do different is make it an extension method.

3

You can raycast against the desired collider (hit.collider.Raycast) instead of a generic Physics.Raycast. This should be faster because it only has to test against the one collider.

5 Likes
4

Sorry to necro this, but the issue is still actual, there is a little problem with the code above that on edges of BoxColliders raycasting by using the inverse of the normal of the hit can lead to the wrong normal being detected.
In the example below I SphereCast down on an edge and I get a random normal (red), when applying the function above to the hit I get a wrong normal (blue), since I was SphereCasting down I would expect a normal towards up in this case.
7795002--984633--upload_2022-1-10_10-41-22.png

I’ve solved this with the code below (also implemented @Lord_Emperor 's suggestion and made it an extension method like @rageingnonsense suggested):

/// <summary>
/// Calculates the surface normal for CapsuleCast and SphereCast
/// </summary>
/// <param name="hit">original hit</param>
/// <param name="dir">original direction of the raycast</param>
/// <returns>correct normal</returns>
/// <remarks> https://discussions.unity.com/t/554303 </remarks>
public static Vector3 GetCorrectNormalForSphere(this RaycastHit hit, Vector3 dir) {
    if(hit.collider is MeshCollider) {
        var collider = hit.collider as MeshCollider;
        var mesh = collider.sharedMesh;
        var tris = mesh.triangles;
        var verts = mesh.vertices;

        var v0 = verts[tris[hit.triangleIndex * 3]];
        var v1 = verts[tris[hit.triangleIndex * 3 + 1]];
        var v2 = verts[tris[hit.triangleIndex * 3 + 2]];

        var n = Vector3.Cross(v1 - v0, v2 - v1).normalized;

        return hit.transform.TransformDirection(n);
    }
    else {
        RaycastHit result;
        hit.collider.Raycast(new Ray(hit.point - dir * 0.01f, dir), out result, 0.011f);
        return result.normal;
    }
}
3 Likes
5

I don’t think it’s random. It’s returning the direction to the sphere’s center on hit. The surface normal is already accessible in other ways. It’s very useful for character controllers and anything where you want to know the direction to return force to an “impactor”.

Getting the center of the sphere on impact:

Vector3 center = hit.point + sphereCastRadius * hit.normal
1 Like