OnMouseDown not called on object for brief period after object moves

I have a menu that is made out of objects with BoxColliders on them. This menu slides down from above and then slides left to show a sub-menu when a button in the main menu is clicked. It sometimes seems unresponsive because none of the buttons respond to clicks for a little bit after the menu finishes moving. The amount of time is variable: sometimes like half a second, sometimes close to 2 seconds, but it’s long enough often enough that I run into it all the time.

I’ve already checked for other collision being in the way, for the colliders turning off somehow, for the colliders changing size. None of that is happening. I’m just animating the position of a parent object over time.

Has anybody run into this and figured out a solution?

If you have an object with a collider attached to it, you should also attach rigidbody component to that and set necessary settings (like unset use gravity and set to kinematic).