Displaying a Custom Shader rendered to RenderTexture

We have a project that uses instanced rendering (via Graphics.DrawMeshInstanced) and a custom handwritten shader. Our content renders successfully in the Unity Editor, but does not appear in the Vision Pro simulator. I understand that neither custom shaders nor DrawMeshInstanced are supported in PolySpatial, so we are attempting to find alternate ways of rendering our content.

In some of the documentation (and this video from Apple) there is a vague mention of rendering custom shaders to a RenderTexture and then using that RenderTexture to display content in the Immersive App.

I have modified our project to render to a RenderTexture instead of to the screen, but it is unclear what the next steps are to display the content in the MR world. Any guidance here would be much appreciated.

I am also investigating converting our shader to Shader Graph, although it is unclear if the PolySpatial support for Shader Graph includes GPU Instancing and moving vertices in the vertex shader.

It is worth mentioning that our content renders just fine when building a Fully Immersive VR app for visionOS.

We are using Unity 2022.3.18f1, PolySpatial 1.0.3, and XCode 15.2.

We’re doing a fair amount of this - it sounds like the step you’re missing is to manually call Camera.Render every frame for the camera that is rendering to a texture. To draw in RealityKit just assign that texture to the BaseMap property of an Unlit material instance attached to a mesh

As @joe_nk mentioned, you will either need to call Camera.Render manually (as described in the docs here) or, if not using a Camera, dirty the RenderTexture manually each frame that it changes (as described here). You should be able to use the RenderTexture in a material as you would any other texture.

No support for GPU instancing at the moment, though we plan in the future to investigate at least transform-based instancing through MeshInstanceCollection. For moving vertices, we do support modifying the vertex position in the vertex stage (and it is possible, for instance, to sample textures–including RenderTextures–in the vertex stage and use them to set/modify vertex positions).

Thank you for the suggestions. Using manual dirtying, I am able to visualize the RenderTexture on a plane in the simulator, so I have confirmed that the texture itself is transferring to the host. However that is not exactly what we are attempting to accomplish.

For a bit more context, we are using our custom shader and instanced rendering to create a 3D object made from many small meshes that are generated programmatically at runtime. I am now drawing our custom object to a RenderTexture, from the perspective of the camera. However, I am searching for a way to composite this output into the environment from the perspective of the camera. Rather than rendering this texture on geometry in the scene (because it’s already rendered from the correct perspective), I want to do a depth-based composite directly into the current depth and image buffer.

There’s no direct support for depth compositing in RealityKit (or, really, for writing any depth value other than the one from the geometry). The only way I know of to do something like this is essentially to use a displacement map: a reasonable dense fixed geometry grid that samples the depth texture (which you should be able to render in Unity as an RGBAHalf texture) and uses it to deform the vertices in a shader graph. We’ve had some success with this approach internally.

This approach is showing some promise. I have something rough working that takes the output RenderTexture (color and depth) from my custom shader, and passes that into a new Shader Graph that renders the color onto a mesh and displaces its vertices based on the depthBuffer from my RenderTexture.

However, I’m having trouble accessing my RenderTexture’s depthBuffer in the Shader Graph.

Adding a Custom Function Node that calls SAMPLE_DEPTH_TEXTURE fails to produce a working shader:

[Worker0] Couldn’t parse custom function: UnityEditor.ShaderGraph.MaterialX.CompoundOpParser+ParseException: Unknown operator SAMPLE_DEPTH_TEXTURE at row 1, col 7: Out = SAMPLE_DEPTH_TEXTURE(tex, sampler, coord.xy);

Next, I tried setting up my custom shader’s command buffer target to have separate color and depth RenderTextures, and passing those color and depth textures into my Shader Graph as separate inputs. This works in the Unity Editor, and I’m able to sample the depth from my depth texture via the R channel. But when I attach the visionOS simulator, I get the following error and nothing appears:

Reading pixels from the depth-only render texture

So, as a workaround, I’ve created a third RenderTexture, and after my custom shader draw call, I blit the depth buffer into this third RenderTexture, then pass that into my Shader Graph as the depth input. It’s working in the simulator with this approach, but it seems like there should be a way to access the depth buffer without the extra blit.

Right; this isn’t one of the macros we support for the Custom Function node. More generally, I don’t think that the DrawableQueue API that we use to transfer RenderTextures supports depth texture formats.

Yeah, I’m not sure if there’s a way to do this. Our internal code seems to do a similar blit.

@kapolka Thank you for the suggestions. After a bit of massaging, I have something working nicely in the VisionOS Simulator. Here is the approach that is working:

Rather than use Graphics.DrawMeshInstanced to draw our custom content (with custom handwritten shader) into the scene, we:

  • Create a CommandBuffer, set its View and Projection matrices to match the current Camera’s, then use CommandBuffer.DrawMeshInstanced to draw our custom content into color and depth RenderTextures.
  • Next, we must Blit the depth texture into another RenderTexture, due to PolySpatial not being able to use RenderTextures of format RenderTextureFormat.Depth directly. This “output” depth RenderTexture is using GraphicsFormat.R16G16B16A16_SFloat, which PolySpatial supports.
  • Finally, we pass the color and output depth RenderTextures into a Shader Graph that outputs an Unlit Material. The Shader Graph uses the Camera’s inverse view projection matrix, along with sampled depth, to place a dense mesh’s vertices at the correct locations in world space.
    • A note on Camera transforms in the Shader Graph: We must retrieve the Camera’s inverse view projection matrix using the Unity Camera in our C# script and pass that to the Shader Graph as a Uniform. Using the Shader Graph Camera Node does not produce the correct matrix.

I do have one final question:
I have noticed that the camera in the Simulator does not seem to match the camera parameters that we retrieve in Unity. Using the camera transforms provided by Unity does project our content properly into the world, but the field of view of the Unity Camera does not match the field of view of the Simulator. The Simulator seems to have a larger FOV than Unity reports, which means our content gets cutoff before the edge of the Simulator’s view of the scene. Is this an issue with the Simulator? Or will this be true on an actual Vision Pro device as well? If it is the latter, is there a way to retrieve the true camera parameters used by VisionOS in a Unity app?

Glad to hear you have something working!

We have some internal reports that TextureFormat.RFloat/GraphicsFormat.R32_SFloat works as well; you might try using that to get more precision.

Thanks for the note. The inverse view projection matrix returned by the Transformation Matrix node transforms from RealityKit’s world space, which is different from Unity’s (and, of course, the camera parameters are different).

I’m not aware of a way to get these parameters aside from the head position and rotation, which you can get (only) in unbounded mode. The camera parameters in Unity are not (by default) synchronized in any way with the visionOS camera, whose parameters are not exposed to applications.

GraphicsFormat.R32_SFloat works in the Editor and Play To Device, but crashes when building in Xcode and running in the Simulator. The DrawableQueue throws an error about unsupported format.

We’re currently targeting an unbounded app, so that is fine for now. Are there plans to expose the visionOS camera to Unity in the future? It seems like that would be required for many types of view-dependent rendering effects. If not/in the meantime, do you have any guidance on what parameters to set on the Unity camera so it more closely matches the visionOS camera for unbounded apps?

Good to know; thanks!

This is basically in the hands of Apple, and I don’t know that they have any plans to expose this information to applications. It’s worth submitting feedback (via the Feedback Assistant) requesting this ability.

I mentioned this in another thread, but basically, I think you would have to estimate an IPD value to get matrices offset for each eye, and perhaps reverse engineer the field of view (which I would expect to be fixed, albeit to different values in simulator versus device). Because the projection matrix is accessible in shader graphs, you could do this by extracting the field of view from the projection matrix and rendering it as a color, then sampling that color value. It’s also quite possible that the FOV information is documented somewhere, either officially or unofficially.

I was just playing around with this by changing the Unity camera and rebuilding. For the simulator, at least, a Unity camera FOV of 75 fills the screen. The exact simulator FOV seems to be somewhere between 70 and 75. In our specific scenario, it’s ok if our estimated FOV is slightly greater than the true camera FOV, so I didn’t fine tune it to find the exact FOV.

1 Like

Any ideas to use camera render target in custom shader?

How to get the correct view and projection matrices?Here’s my code:

Matrix4x4 V = m_MainCamera.worldToCameraMatrix;
Matrix4x4 P = m_MainCamera.projectionMatrix;
P = GL.GetGPUProjectionMatrix(P, true);
cmd.SetGlobalMatrix("_Custom_V", V);
cmd.SetGlobalMatrix("_Custom_P", P);

It only works in editor, not in device.

In our case we used something like this:

Matrix4x4 inverseViewProjection = (_camera.projectionMatrix * _camera.worldToCameraMatrix).inverse;

_renderPassMaterial.SetVector("_CamInvViewProjCol0", inverseViewProjection.GetColumn(0));
_renderPassMaterial.SetVector("_CamInvViewProjCol1", inverseViewProjection.GetColumn(1));
_renderPassMaterial.SetVector("_CamInvViewProjCol2", inverseViewProjection.GetColumn(2));
_renderPassMaterial.SetVector("_CamInvViewProjCol3", inverseViewProjection.GetColumn(3));

and then in the Shader Graph, we use a Matrix Construction Node to turn these columns into a 4x4 matrix.

However, it is worth noting again that, because Unity + Polyspatial doesn’t give us access to the actual camera parameters used by the device (except inside the Shader Graph), this matrix does not match the actual matrices used to render the final output. We attempted to estimate the IPD, as @kapolka suggested earlier in this thread, but we were never able to get an accurate enough result. In the end, we started from scratch using a completely different approach (porting our original custom shader to Shader Graph).

1 Like

Thank you for your reply.I’m trying to render real-time shadow map in unity, and use it in shader graph, so it’s fine to get view and projection matrices of light component using this approach.