Displaying a Custom Shader rendered to RenderTexture

@kapolka Thank you for the suggestions. After a bit of massaging, I have something working nicely in the VisionOS Simulator. Here is the approach that is working:

Rather than use Graphics.DrawMeshInstanced to draw our custom content (with custom handwritten shader) into the scene, we:

  • Create a CommandBuffer, set its View and Projection matrices to match the current Camera’s, then use CommandBuffer.DrawMeshInstanced to draw our custom content into color and depth RenderTextures.
  • Next, we must Blit the depth texture into another RenderTexture, due to PolySpatial not being able to use RenderTextures of format RenderTextureFormat.Depth directly. This “output” depth RenderTexture is using GraphicsFormat.R16G16B16A16_SFloat, which PolySpatial supports.
  • Finally, we pass the color and output depth RenderTextures into a Shader Graph that outputs an Unlit Material. The Shader Graph uses the Camera’s inverse view projection matrix, along with sampled depth, to place a dense mesh’s vertices at the correct locations in world space.
    • A note on Camera transforms in the Shader Graph: We must retrieve the Camera’s inverse view projection matrix using the Unity Camera in our C# script and pass that to the Shader Graph as a Uniform. Using the Shader Graph Camera Node does not produce the correct matrix.

I do have one final question:
I have noticed that the camera in the Simulator does not seem to match the camera parameters that we retrieve in Unity. Using the camera transforms provided by Unity does project our content properly into the world, but the field of view of the Unity Camera does not match the field of view of the Simulator. The Simulator seems to have a larger FOV than Unity reports, which means our content gets cutoff before the edge of the Simulator’s view of the scene. Is this an issue with the Simulator? Or will this be true on an actual Vision Pro device as well? If it is the latter, is there a way to retrieve the true camera parameters used by VisionOS in a Unity app?