Fully working solution with scene on GitHub: UnityMathReference/Assets/Demos/Shaders/DepthBuffToWorldPos at master · zezba9000/UnityMathReference · GitHub
Lots of credits to this: https://gamedev.stackexchange.com/q…in-vr-through-projection-matrix/140924#140924