One way is to stop using 3D colliders and go to 2D colliders, as those all exist at z == 0. This would require some work, especially if you use those colliders for other things.
Another way is to make the 3D colliders large enough in the Z direction that they always get hit, but that might have other issues.
Still another way is to iterate ALL the colliders that you CAN hit (via some central list of them) and cast a fresh ray that matches their Z at them going in the same X/Y direction and see if it hits.
To the point of the last choice, remember this about Raycasting, colliders, planes, Plane, etc: