Need help with HDRP VR peformance

My project seems to not be able to get enough fps, XR.WaitForGPU takes 14ms.
I am using the legacy xr system.
Also RenderPipelineManager.DoRenderLoop_Internal() takes 5ms on an empty scene
After I started my pc today I managed to get 100fps once on an empty scene but that should be more. Yesterday all I could get was 50 fps on a fully emtpy scene and now its back to that.
On my populated scene with no lights and a few walls I still could only get 40fps. Mostly limited by XR.WaitForGPU.
I have a I5-4460 and a GTX 980ti, I could run games like Arizona Sunshine or even got better fps in HL:Alyx.

Can’t really figure out anything