Make Only Certain UI block raycast/touch

I have a panel which turns on and off at certain points, when I want nothing to be clickable if it is behind the panel. I used this code to achieve this - if (EventSystem.current.IsPointerOverGameObject())
return; However this makes my canvas also block everything making the game unplayable, is there a way to modify this code so only the panel blockes touches/raycast

In your Canvas, make sure whatever you are putting in front of the clickable objects is closer to the bottom than whatever is behind it.

For example
Image (Image blocks the button from being pressed)