RaycastHit.normal - what does it really return?

[PeterB]

I'm testing for proximity of other objects by raycasting a sphere, considerably larger than the object itself (also a sphere). I then examine the array of RaycastHit structs, using the RaycastHit.normal vector to push away from the other object in proportion to the distance. So far, so good.

However, the normal returned isn't perpendicular to the surface of the object. In the case of a wall created from a cube, I would have expected the normal to point 90 from the surface, but it doesn't. In the following screen caps, the blue line is the vector returned by RaycastHit.normal:

The white line is the original direction of the sphere; the red line is the ray from the centre of the sphere to the collision point, for which the normal supposedly is computed. The documentation says the normal is "the normal of the surface of the ray hit". As you can see, the angle is not 90 against the surface.

Here's the same scene from a slightly different perspective:

As you can see, the normal points downwards, which drives the sphere, which will follow the yellow line, down into the ground. Not so good.

The plane beneath the other objects gives different results:

In this case, the normal is always along the hit ray.

When moving the sphere around, I can also sometimes see how the hit point snaps to one of the vertices of the wall and stays there; sometimes the hit point is clearly not the closest point. In the case of the plane, the hit point always seems to be along a triangle vertex.

Am I misunderstanding how normals and/or colliders work here? (I should add that I'm not using physics to move the sphere, but translation. As the next step, I'll implement the same type of steering using rigidbodies and physics; I want to compare the pros and cons of the two different approaches.)

I'm using Unity Pro 3.3. Here's the code attached to the sphere:

var target : Transform;
var speed : float = 1.0;
var comfortDist : float = 10.0;

var move : boolean = false;    
var skip : Transform;

private var r : float;
private var repulsionExtent : float;

function Start() {
    r = transform.lossyScale.x; 
    repulsionExtent = comfortDist - r;
}

function Update() {
    if (!target) return;
    // Turn towards the target
    transform.LookAt(target);

    var pos : Vector3 = transform.position;
    var dist : float = Time.deltaTime * speed;
    var ray : Ray = new Ray(pos, target.position - pos);
    var dir : Vector3 = transform.TransformDirection(Vector3.forward);
    Debug.DrawRay(pos, dir * 10);

    var hits : RaycastHit[] = Physics.SphereCastAll(ray, comfortDist, dist);
    for (var h in hits) {
        if (h.transform != target && 
            h.collider != this.collider &&
            h.transform != skip) { 
            var realDist : float = Vector3.Distance(pos, h.point) - r;

            Debug.DrawLine(pos, h.point, Color.red);
            Debug.DrawRay(h.point, h.normal, Color.blue);

            var repulsion = (repulsionExtent - realDist) / repulsionExtent;

            var currentForward : Vector3 = transform.TransformDirection(Vector3.forward);
            var newdir : Vector3 = Vector3.Slerp(currentForward, h.normal, repulsion);
            transform.rotation = Quaternion.FromToRotation(currentForward, newdir) * transform.rotation;
        };
    };

    var resultdir : Vector3 = transform.TransformDirection(Vector3.forward);
    Debug.DrawRay(pos, resultdir * 10, Color.yellow);

    // Move forward
    if (move) transform.Translate(Vector3.forward * dist);
}

P.S.: If you put in more than one seeker sphere, you'll notice that they behave correctly in respect to each other, which seems to have to do with the fact that the sphere collision normals behave as expected. Or rather, as I would expect them to... ;-)

[Statement]

It seems that the normal returned from the SphereCastAll method I used represents the normal from the SphereCast, in reverse. This seems to happen if the sphere cast does not hit "ray on" the collider, such in the cases of hitting a edge with the "sides" of the casted sphere. Otherwise my tests show it seems to return the normal of the surface. I think this is some sort of "not really the surface normal but the expected surface bounce direction". Or you can view it as an slerped normal for edge cases.

It seems that:

• If hitting straight on a collider, it return collider normal.
• If hitting edge of collider, it return slerped collider normal.
• Bigger radius make errors (I tried with radius 10 and it was completely unexpected).

Another way of looking at it is to imagine your colliders are spherically extruded, and you cast a ray on that:

That can perhaps make some sense in why the normal "bends off" in edge cases.

This is the script I used to get to this conclusion:

using UnityEngine;
using System.Collections;

public class SphereCastTester : MonoBehaviour
{
    public float radius = 1.0f;
    public float distance = 10.0f;

    void Update()
    {
        Ray forwardRay = new Ray(transform.position, transform.forward);
        var hits = Physics.SphereCastAll(forwardRay, radius, distance);

        // White = cast direction
        Debug.DrawRay(forwardRay.origin, forwardRay.direction * distance, Color.white);

        foreach (var hit in hits)
        {
            // Red = from origin to point
            Debug.DrawLine(forwardRay.origin, hit.point, Color.red);

            // Blue = normal on hit point.
            Debug.DrawRay(hit.point, hit.normal, Color.blue);
        }
    }
}

To try this out, just place it on some object, place a few colliders out, hit play and move the transform of the object around a bit.

Edit: I tried setting radius to 10, and some unexpected behavior was witnessed. It seems the normal bends off way too early. I can't explain this.

[Ashkan_gc]

maybe your raycasting code has some problems. posting the code might help. pictures don't display anymore to see. i did not have a problem like this never. using physics for special gameplay types don't work most of the times unless you use some tricks so this way is a better one. however the blurst.com guys might be able to implement anything using physx. :)

[Bampf]

Double-check the collider of the wall, make sure it's what you expect.

[Tabemasu_Games]

I created a small function to fix the normal returned by a sphere or capsule cast:

public static void FixNormal(Vector3 position, ref RaycastHit hit, int layermask) {
	RaycastHit rayHit;
	Physics.Raycast(position, hit.point - position, out rayHit, 2 * hit.distance, layermask);
	hit.normal = rayHit.normal;
}

Basically, it casts a ray, and finds its normal.

Usage:

RaycastHit hit;
if (Physics.SphereCast(startPosition, radius, direction, out hit, distance, layermask)) {
    FixNormal(startPosition, ref hit, layermask);
    // hit.normal is now OK
}