Raycast passing through object after first couple of times.

Perhaps it’s closer than you think and your raycast is already starting beyond it. Add an extra line that does RaycastAll() and use Debug.Log() to print out what all that ray passes through.

If you only care about one specific Collider, you can use the form of raycast that starts form that Collider.

Raycasting, colliders, planes, Plane, etc: