I tried to add this functionality and it seems this is a physics level thing, specifically that collider casts do not distinguish between face sides.
The character controller also cannot free itself when inside an object, presumably also because of this
