How is Physics.Raycast implemented in Unity? What is under the hood or how would you do it on your own?
Find all objects where the ray intersects the bounding box of the collider (Bounds.IntersectRay). For each triangle in those colliders (assuming you’re using MeshColliders), find the point where the ray intersects the plane of the triangle (Plane.Raycast). Then test if that point is the triangle. For a triangle ABC and a point P that is the case if and only if Cross(AB, AP), Cross(BC, BP) and Cross(CA, CP) all point in the same direction.
Under the hood, Unity uses Nvidia PhysX for physics but that probably won’t help you much and unfortunately I don’t know how PhysX works in detail.