Creating a VR "Portal" shader

I’m using URP in single pass mode.
I’m trying to make a shader that will let me see through an object like a portal to a different part of the scene.
To do this I have a stereo camera at some remote location that is rendering to a render texture:

public Camera portalCamera;
public Material mat;
private RenderTexture rendTex;

IEnumerator Start()
{
yield return new WaitForSeconds(1f); // XRSettings arent updated right away so I wait for a second
rendTex = new RenderTexture(XRSettings.eyeTextureWidth, XRSettings.eyeTextureWidth, 16, RenderTextureFormat.ARGB32);
rendTex.vrUsage = VRTextureUsage.TwoEyes;
portalCamera.targetTexture = rendTex;
mat.SetTexture("_Texture2D", rendTex);
}

In shader graph I have a simple shader that samples the render texture in screen space on some geometry:

8463734--1124009--Screenshot 2022-09-24 231056.png

The end result looks almost correct, but something’s off. It kind of looks like the left eye is rendering to the right and vice versa based on the weird parallax I see. Anybody know how to do this correctly?

Maybe look into the URP stencil features. Used that a lot on my projects for portals and non Euclidean features.
Haven’t tested in VR but don’t see why it wouldn’t work