Problem with Raycast detecting a collider overtime.

I’ve made a basic RTS game. When I start the game and try selecting an Unit it works fine. Overtime though, after like 2 minutes when I try to click on the Unit again in the area where it selects the unit becomes much smaller. If I pause the scene and check the colliders, they are the same size as in the start, but clicking on them doesn’t do anything anymore. See pictures for clarification.

I also find rather odd that if I Debug.Log(agent.velocity) the velocity on the y axis is getting smaller every second. Is gravity doing this?

In this topic, the issue is almost identical but deleting child objects with different layers didn’t help.

Clicking on the object selects the unit, indicated by the blue ring.

186570-pic1.png

Clicking here again after a couple of minutes, doesn’t do anything.
186571-pic2.png

Deleting the rigidbody worked as stated in the other topic and I won’t be needing one for the units anytime soon, this will have to do for an answer for now.