Going through walls when moving irl vr

I am using a character controller and a capsule collider on the VR rig. When I walk using the joystick into the wall, the collisions work fine but if I move in real life, I can just go in the wall. I tried using scripts to push the player back if they try to look in the wall but the corrections are harsh and disruptive even when i try to make it more smooth. Also if i move the headset really fast it still goes in the wall. Is there any fixes for this?

The easy way is to add a collider to the HMD game objects, and black out the screen. Then just revert it once the player is back into place

A while ago I posted my thoughts on this issue here: What is the best solution for Head clipping trought walls and objects?