Problem with raycasts

I have a tunnel that the player is supposed to go down and avoid holes. The player object only moves on the Z-Axis, to move left and right, the player rotates the whole tunnel. To check if the player hits a whole, I have simple script that sends a raycast down, and if it does not hit anything, it ends the game. It works, until the player turns the tunnel too fast. I have a debug ray showing that the raycast is hitting the collider. Any idea what would cause this?

Debug.DrawRay(transform.position, Vector3.down, Color.green);
var hit : RaycastHit;
if(!Physics.Raycast(transform.position, Vector3.down, hit)){
	Keeper.end = true;
}

I have random parts of the tunnel turn into wholes. What I was doing was just deleting a part of the wall to make a whole and using raycasts to see if it was above a whole. Instead I just turned the mesh render off and renamed it to “hole”. I then used the raycast to see if it was hitting an object named “hole”.