How to detect click on naked panel?

In my game I have a panel opening up to show stats and allow configuration. The panel should be closed again when it is clicked at a position no other input element exists.

  • When clicking on pure panel, close
  • When clicking on some text, close
  • When clicking on a button, don’t close
  • When clicking on a toggle, don’t close

Already tried EventSystem.current.IsPointerOverGameObject(), sadly this also closes the panel when clicking on a button or checking the toggle.

Finally found the answer here: Detect UI Panel click - Questions & Answers - Unity Discussions