I gave ray tracing with HDRP + XR another try with Unity 6 preview (after finding it way too slow in older Unity versions) and was pleasantly surprised how well it runs! Quite stable, and getting 36FPS with RTAO, RTGI, RTSSR using DLSS on a 3080, 72 FPS if RTGI is turned off.
Two main issues I found:
Performance in builds is a good bit worse compared to running it in Editor (getting 25-34 FPS in build, and constant 36 FPS in Editor). This is really strange and I have no clue how to find out what the reason might be or how to report this as a bug.
SSR and RT SSR is somewhat broken in XR in Unity 6 (I did a bug report).
Ok, I found out what the issue is, and it seems to be a bug:
the resolution of the Desktop screen seems to have a big impact on the performance, even if the screen is not rendering (XRSettings.gameViewRenderMode = GameViewRenderMode.None;), so it seems all rendering is done for both XR screen AND the desktop screen. Edit: Seems also okay if I set the resolution of Desktop screen to match XR resolution.
If I set the Screen resolution to a low value, performance in build jumps up.
I now get 60-70 fps with around 15ms of GPU time with raytraced SSGI, SSAO and SSR! This is HDRP sample scene with one ray traced SMR in XR running on a 3080.
Now this is fantastic, if Unity gets rid of a few bugs this would be the first viable solution to have ray tracing in XR with good performance!