RayCast appears to bug out for no apparent reason

Hello,
I am having an unusual issue with raycasting an object in my scene. The code works most of the time, but then for seemingly no reason at all the object somehow becomes unidentifiable by the raycast despite its location being sound.
The only way I can find to make the raycast find it again is one of two ways:

  1. toggle on and off in the box collider on the object in the scene view
  2. remove the object as a child in the scene view hierarchy

Attached is a video demonstration which takes a bit to reproduce the bug @40sec and @3:47 :

Also here is the code for the raycast which is being called correctly:

 public ObjectController CheckForObjectAtLoc(Vector3 pos)
    {
        var ray = Camera.main.ScreenPointToRay(pos);
        if (Physics.Raycast(ray, out RaycastHit hit))
        {
            Debug.Log($"The Hit succeeded @loc:{pos} ,   the hit is: {hit.transform.gameObject}");
            return (hit.transform.gameObject.GetComponent<ObjectController>());
        }
        else
        {
            Debug.LogWarning($"The Hit @loc:{pos} somehow missed");
            var bo = GameObject.FindObjectOfType<BuildableObject>();
            if (bo && bo.transform.childCount>0)
            {
                
                var child = bo.transform.GetChild(0);
                var screenloc = Camera.main.WorldToScreenPoint(child.position);
                Debug.LogError($"{child.gameObject.name} is @loc:{screenloc} , and world pos = {child.transform.position}");
            }

        }

        return null;

    }

If anyone has any clue at all why this is occurring I would really appreciate your help.
Thank you

Update:
added:

        var ray = Camera.main.ScreenPointToRay(pos);
        Debug.DrawRay( ray.origin, ray.direction*1350, Color.red, 5);
        if (Physics.Raycast(ray, out RaycastHit hit))

Goes right thru the box

After speaking with a few other unity devs on discord and IRL who have had similar issues, it seems this is likely an issue with unity itself somehow messing up colliders with objects spawned from prefabs.
The tmp hack solution is to toggle on and off the box collider when spawning or resetting the object to (0,0,0) seems to be working

If anyone knows how to escalate or forward this to unity please let me know. (if that would be helpful)