GameObject rapidly changing position (flickering)

I have a gameObject that starts with a random rotation and position between a set range.

And when my mouse hovers over (IPointerEnter, IPointerExit) the gameObject it resets its position and rotation the the default values.

The problem is, that when I position my mouse on a specific place over the gameObject (mostly the corners because of the rotation) it starts rapidly changing its position.
(from the default position to the random position that is set at the start)
My question basicly is how to detect that the object is doing that weird position/flickering thing so I can prevent it in code.

well my guess would be that if you have the PointerEnter event your object rotates and the new rotation might be one where your cursor is not over the object anymore and this will automatically call the PointerExit event.

So either remove the object rotation on PointerExit or find a condition which checks if the pointerExit has been called due to the rotation. Something like

“Since we have called OnPointerEnter the object has rotated while the mouse has not moved more than a delta of X and thus the current call of PointerExit should be ignored”