VR Camera jumps up and does not ignore collider even after turning off interaction between them in Layer Collision Matrix

So this is an issue I have been stuck with for a while and I just cannot even find a decent workaround.

What happens is if you use a sword with a collider to stab yourself then the camera jumps up. Even if this collider does not interact with anything in the Layer Collision Matrix.

If I disable this Collider on the sword then the camera no longer jumps up. However I really need a non trigger collider on the sword for various things.

I am using Unity 2020.3 and OVRCameraRig

Can anyone give some tips?

For anyone still looking for an answer, I was able to fix the issue following the instructions of this video:


Good luck!