Raycasts returns unexpected hit after a while

Heyho,
So I have been working on an RTS like game where I select units with left click and order them to move using right click.

However, I recently encountered an error in the raycast that causes the raycast to return the wrong object.
While keeping the mouse in the same position, I clicked 50 times or so and after that, the returned GameObject was the plane behind the object that I clicked on. Before that, the right object was returned flawlessly, however, after a while, the code stops working properly.

Here is what I’ve been using:

public class MouseInteractionScript : MonoBehaviour
{

GameObject selected;
RaycastHit hit;
Ray ray;

// Update is called once per frame
void Update()
{
        ray = Camera.main.ScreenPointToRay(Input.mousePosition);
        // print(ray.direction);
        if (Physics.Raycast(ray, out hit, Mathf.Infinity))
        {
            HandleInput();
        }
}

void HandleInput()
{
    if (Input.GetMouseButtonDown(0))
    {
        if(selected == hit.collider.gameObject)
            Debug.Log("Same object was selected");
        if (hit.collider.gameObject.GetComponent<SquadMemberMovementScript>() != null)
            selected = hit.collider.gameObject;
            else
            {
                Debug.Log(hit.collider.gameObject);
            }
            
    }
    if (Input.GetMouseButtonUp(1) && selected != null)
    {
        if(hit.transform.GetComponent<BaseStatScript>() == null){
            // print("Moving");
            selected.GetComponent<SquadMemberMovementScript>().SetTarget(hit.point);
        }
        else
        {
            selected.GetComponent<SquadMemberMovementScript>().SetTarget(hit.collider.transform);
        }
    }
}

}

The wonderous thing about this for me at least is, that no matter what object I click on next, it will always return the plane or any other object that is not of the same prefab.

Also I’m using Unity 2018.4.2f1

Alright, here seems to be the answer although I’m not 100% sure that it was “it”… So I was clicking on units with a Rigidbody component. The collision detection was set to the default of distinct so I changed it to continuous. This solved my problem at least, so it might help others as well.

The reason is that Input.GetMouseButtonDown and Input.GetMouseButtonUp do not work properly in FixedUpdate. Because this functions return mouse state from last frame. While you in FixedUpdate, unity can draw few frames, so you can “miss” your mouse state. This functions should be called only from Update function.