Button color flicker on change ColorBlock

I have a script where I hilight a clicked button by changing the color block Normal color.

But when I reassign the color (not with a click or rollover - mouse isnt interacting with the button ), the color seems to flicker back and forth between the old and new colors. just once then it stays.

just:

button.colors = savedColors;

where savedColors is a ColorBlock whose normalColor and hilightedColor have been changed.

I found a solution to this, though I don’t quite like it.

If I call button.gameObject.SetActive( false );

before setting the color block, and then setting it back to active after I reassign the colors, the flicker goes away.
this does seem a little extreme, however.

I found out the actual answer to this.

In Unity, highlighted color is used whenever the button was the last object clicked. I was expecting it to behave more like a rollOver color.