Raycast randomly failing. Do colliders sleep?

Hi all. I have encountered a weird raycast bug.
I have a very simple scene with only a couple of objects. One of these objects has a sphere collider on it, and this is the only collider in the scene.

I’m using a raycast to allow the player to click on the object and drag it around and it’s working correctly 99% of the time. But if I keep clicking on the object, after a random number of clicks the raycast stops working.
Once it stops, I can no longer drag the object around and it stays broken no matter how many times I click on it.

I am doing the raycast in FixedUpdate() (but have also tried normal Update and it makes no difference).
I have stepped through the code and the raycast is running but stops returning true.
I have put a Debug.DrawRay() in, and the Scene view is showing the collider in the correct place and that the ray passes through it. Yet it doesn’t detect the collider.
The number of clicks required to make this start happening is very random, I’ve had it occur after just 5 clicks, and not for hundreds of clicks, but I have reproduced this at least 10 times in my search for answers, I just keep clicking until it stops working.

Now here’s the weird bit that makes me think this is a Unity bug or weird undocumented interaction: Once it’s broken, if I just disable and re-enable the collider in the inspector, everything starts working again.

So my question is: Why would a collider stop being detected by a raycast? It almost feels to me like the object is “sleeping” or similar, but this is not a rigid body.

Here’s the raycast code:

private void FixedUpdate()
{
  if(m_bMouseDown)
  {
    Vector3 v3Mouse = Input.mousePosition;
    v3Mouse.z = Mathf.Abs(Camera.main.transform.position.z);
    Ray ray = Camera.main.ScreenPointToRay(v3Mouse);
    Debug.DrawLine(ray.origin, ray.origin + ray.direction * 10, Color.red, 5.0f);
    RaycastHit hitInfo;
    if(Physics.Raycast(ray, out hitInfo)) //This if statement fails
    {
        m_Selected = hitInfo.collider.gameObject;
    }
    m_bMouseDown = false;
  }    
}

Well I fixed it although I can’t fully explain why it was happening and I suspect there is a subtle bug in the collider somewhere.

Another feature I had in my scene was a text box which you could type a position in to to change the location of the object with the collider.

For dumb reasons I had a convoluted setup where when you dragged the object around, it would do the following:

  1. Script 1 would use a raycast to determine where you dragged the mouse to and pass it to script 2
  2. Script 2 would convert this position into a string and put it in the input text box
  3. script 2 would read the information back our of the input text box and convert it back to a vector and pass it to script 3
  4. script 3 would use this vector to set the objects position.

I don’t know why the intermediate steps of putting the position in a text box and taking it back out again and sending it through 3 different scripts messed things up but as soon as I streamlined this process the bug went away.

What’s particularly weird is that when looking at the scene view, the object was in the correct place, with a visible collider on it, with a debug ray going right through it. So the only thing I can think of is that some timing issue was occurring where the object was not in the correct place when the raycast was processed but was back in the correct place when rendered. But then why did turning the collider component off and on again fix it?

Very odd. Anyway its fixed now by just avoiding this completely.