We have a project that uses instanced rendering (via Graphics.DrawMeshInstanced) and a custom handwritten shader. Our content renders successfully in the Unity Editor, but does not appear in the Vision Pro simulator. I understand that neither custom shaders nor DrawMeshInstanced are supported in PolySpatial, so we are attempting to find alternate ways of rendering our content.
In some of the documentation (and this video from Apple) there is a vague mention of rendering custom shaders to a RenderTexture and then using that RenderTexture to display content in the Immersive App.
I have modified our project to render to a RenderTexture instead of to the screen, but it is unclear what the next steps are to display the content in the MR world. Any guidance here would be much appreciated.
I am also investigating converting our shader to Shader Graph, although it is unclear if the PolySpatial support for Shader Graph includes GPU Instancing and moving vertices in the vertex shader.
It is worth mentioning that our content renders just fine when building a Fully Immersive VR app for visionOS.
We are using Unity 2022.3.18f1, PolySpatial 1.0.3, and XCode 15.2.
We’re doing a fair amount of this - it sounds like the step you’re missing is to manually call Camera.Render every frame for the camera that is rendering to a texture. To draw in RealityKit just assign that texture to the BaseMap property of an Unlit material instance attached to a mesh
As @joe_nk mentioned, you will either need to call Camera.Render manually (as described in the docs here) or, if not using a Camera, dirty the RenderTexture manually each frame that it changes (as described here). You should be able to use the RenderTexture in a material as you would any other texture.
No support for GPU instancing at the moment, though we plan in the future to investigate at least transform-based instancing through MeshInstanceCollection. For moving vertices, we do support modifying the vertex position in the vertex stage (and it is possible, for instance, to sample textures–including RenderTextures–in the vertex stage and use them to set/modify vertex positions).
Thank you for the suggestions. Using manual dirtying, I am able to visualize the RenderTexture on a plane in the simulator, so I have confirmed that the texture itself is transferring to the host. However that is not exactly what we are attempting to accomplish.
For a bit more context, we are using our custom shader and instanced rendering to create a 3D object made from many small meshes that are generated programmatically at runtime. I am now drawing our custom object to a RenderTexture, from the perspective of the camera. However, I am searching for a way to composite this output into the environment from the perspective of the camera. Rather than rendering this texture on geometry in the scene (because it’s already rendered from the correct perspective), I want to do a depth-based composite directly into the current depth and image buffer.
There’s no direct support for depth compositing in RealityKit (or, really, for writing any depth value other than the one from the geometry). The only way I know of to do something like this is essentially to use a displacement map: a reasonable dense fixed geometry grid that samples the depth texture (which you should be able to render in Unity as an RGBAHalf texture) and uses it to deform the vertices in a shader graph. We’ve had some success with this approach internally.
This approach is showing some promise. I have something rough working that takes the output RenderTexture (color and depth) from my custom shader, and passes that into a new Shader Graph that renders the color onto a mesh and displaces its vertices based on the depthBuffer from my RenderTexture.
However, I’m having trouble accessing my RenderTexture’s depthBuffer in the Shader Graph.
Adding a Custom Function Node that calls SAMPLE_DEPTH_TEXTURE fails to produce a working shader:
[Worker0] Couldn’t parse custom function: UnityEditor.ShaderGraph.MaterialX.CompoundOpParser+ParseException: Unknown operator SAMPLE_DEPTH_TEXTURE at row 1, col 7: Out = SAMPLE_DEPTH_TEXTURE(tex, sampler, coord.xy);
Next, I tried setting up my custom shader’s command buffer target to have separate color and depth RenderTextures, and passing those color and depth textures into my Shader Graph as separate inputs. This works in the Unity Editor, and I’m able to sample the depth from my depth texture via the R channel. But when I attach the visionOS simulator, I get the following error and nothing appears:
Reading pixels from the depth-only render texture
So, as a workaround, I’ve created a third RenderTexture, and after my custom shader draw call, I blit the depth buffer into this third RenderTexture, then pass that into my Shader Graph as the depth input. It’s working in the simulator with this approach, but it seems like there should be a way to access the depth buffer without the extra blit.
Right; this isn’t one of the macros we support for the Custom Function node. More generally, I don’t think that the DrawableQueue API that we use to transfer RenderTextures supports depth texture formats.
Yeah, I’m not sure if there’s a way to do this. Our internal code seems to do a similar blit.
@kapolka Thank you for the suggestions. After a bit of massaging, I have something working nicely in the VisionOS Simulator. Here is the approach that is working:
Rather than use Graphics.DrawMeshInstanced to draw our custom content (with custom handwritten shader) into the scene, we:
Create a CommandBuffer, set its View and Projection matrices to match the current Camera’s, then use CommandBuffer.DrawMeshInstanced to draw our custom content into color and depth RenderTextures.
Next, we must Blit the depth texture into another RenderTexture, due to PolySpatial not being able to use RenderTextures of format RenderTextureFormat.Depth directly. This “output” depth RenderTexture is using GraphicsFormat.R16G16B16A16_SFloat, which PolySpatial supports.
Finally, we pass the color and output depth RenderTextures into a Shader Graph that outputs an Unlit Material. The Shader Graph uses the Camera’s inverse view projection matrix, along with sampled depth, to place a dense mesh’s vertices at the correct locations in world space.
A note on Camera transforms in the Shader Graph: We must retrieve the Camera’s inverse view projection matrix using the Unity Camera in our C# script and pass that to the Shader Graph as a Uniform. Using the Shader Graph Camera Node does not produce the correct matrix.
I do have one final question:
I have noticed that the camera in the Simulator does not seem to match the camera parameters that we retrieve in Unity. Using the camera transforms provided by Unity does project our content properly into the world, but the field of view of the Unity Camera does not match the field of view of the Simulator. The Simulator seems to have a larger FOV than Unity reports, which means our content gets cutoff before the edge of the Simulator’s view of the scene. Is this an issue with the Simulator? Or will this be true on an actual Vision Pro device as well? If it is the latter, is there a way to retrieve the true camera parameters used by VisionOS in a Unity app?
Thanks for the note. The inverse view projection matrix returned by the Transformation Matrix node transforms from RealityKit’s world space, which is different from Unity’s (and, of course, the camera parameters are different).
I’m not aware of a way to get these parameters aside from the head position and rotation, which you can get (only) in unbounded mode. The camera parameters in Unity are not (by default) synchronized in any way with the visionOS camera, whose parameters are not exposed to applications.
GraphicsFormat.R32_SFloat works in the Editor and Play To Device, but crashes when building in Xcode and running in the Simulator. The DrawableQueue throws an error about unsupported format.
We’re currently targeting an unbounded app, so that is fine for now. Are there plans to expose the visionOS camera to Unity in the future? It seems like that would be required for many types of view-dependent rendering effects. If not/in the meantime, do you have any guidance on what parameters to set on the Unity camera so it more closely matches the visionOS camera for unbounded apps?
This is basically in the hands of Apple, and I don’t know that they have any plans to expose this information to applications. It’s worth submitting feedback (via the Feedback Assistant) requesting this ability.
I mentioned this in another thread, but basically, I think you would have to estimate an IPD value to get matrices offset for each eye, and perhaps reverse engineer the field of view (which I would expect to be fixed, albeit to different values in simulator versus device). Because the projection matrix is accessible in shader graphs, you could do this by extracting the field of view from the projection matrix and rendering it as a color, then sampling that color value. It’s also quite possible that the FOV information is documented somewhere, either officially or unofficially.
I was just playing around with this by changing the Unity camera and rebuilding. For the simulator, at least, a Unity camera FOV of 75 fills the screen. The exact simulator FOV seems to be somewhere between 70 and 75. In our specific scenario, it’s ok if our estimated FOV is slightly greater than the true camera FOV, so I didn’t fine tune it to find the exact FOV.
and then in the Shader Graph, we use a Matrix Construction Node to turn these columns into a 4x4 matrix.
However, it is worth noting again that, because Unity + Polyspatial doesn’t give us access to the actual camera parameters used by the device (except inside the Shader Graph), this matrix does not match the actual matrices used to render the final output. We attempted to estimate the IPD, as @kapolka suggested earlier in this thread, but we were never able to get an accurate enough result. In the end, we started from scratch using a completely different approach (porting our original custom shader to Shader Graph).
Thank you for your reply.I’m trying to render real-time shadow map in unity, and use it in shader graph, so it’s fine to get view and projection matrices of light component using this approach.
Is this entire discussion focused on the Apple Vision Pro (hereafter referred to as AVP)? I’m encountering a similar issue: I want to sample the scene’s depth buffer in screen space and then perform calculations based on the depth. However, when I use the Scene Depth node in Shader Graph, it works well in the Unity Editor, but it doesn’t seem to work on AVP.
I checked the Unity-compiled usda file, and it doesn’t include the sampling operation. Instead, it directly transforms the World Space Position to View Space using the UNITY_MATRIX_V matrix and then uses positionVS.z as the depth. Should I add a property in the shader graph and use the depth buffer as a material property, then sample it myself using nodes?
So, as a workaround, I’ve created a third RenderTexture, and after my custom shader draw call, I blit the depth buffer into this third RenderTexture, then pass that into my Shader Graph as the depth input. It’s working in the simulator with this approach, but it seems like there should be a way to access the depth buffer without the extra blit.
According to previous discussions, an additional Blit operation is required to copy the depth attachment. However, the URP default pipeline already has a Copy Depth Pass, so I’m a bit confused.
Right; you’ll notice that our notes on supported shader graph nodes indicate that the Scene Depth node just outputs the depth of the fragment being drawn, rather than sampling the depth buffer. That’s because visionOS’s MaterialX support doesn’t provide a way to sample the depth buffer, and there’s no way for us to work around that; visionOS doesn’t provide access to the depth buffer in any way.
This should be possible in some sense, but what you may be missing is that in MR mode, Unity/URP don’t do any rendering by default. Instead, we synchronize the contents of the scene graph to a parallel scene graph in RealityKit, and that performs the rendering. You can use Unity to render to a RenderTexture (and copy the depth to a RenderTexture with a compatible format to use in a shader graph material), but you’ll face the issue that visionOS doesn’t provide access to its camera parameters (such as IPD, or, in bounded mode, even the camera position/orientation). That means that any depth texture you generate will just be an approximation of the one that visionOS uses.
Thank you for your reply. I’d like to know, after I call the Camera.Render() method, will the camera render according to the URP Renderer settings, including the custom Render Features and Render Passes that I’ve added? Or do I need to use the OnRender methods series within MonoBehaviour?
As far as I know, yes, Camera.Render (to a RenderTexture) should use the standard URP setup and support all of its features (as on any other platform).