I have noticed that updating RenderTextures and then calling PolySpatialObjectUtils.MarkDirty causes a Texture2D.ReadPixels followed by Apply/Get/SetPixels32, presumably to serialise the texture data as a byte array for PolySpatial.
Is this technique due to a limitation of RealityKit, or can we expect that in the future RenderTextures can share texture resources with PolySpatial materials and remain on the GPU? My issue is that updating RenderTextures is currently causing significant performance issues.
This is a fallback behavior that we typically perform when the RenderTexture’s format is unsupported, or when the “Disable Native Texture” runtime flag is selected in the PolySpatial settings. The default behavior uses a GPU blit to copy the RenderTexture to a RealityKit TextureResource.DrawableQueue, which is substantially faster.
The first thing would be to ensure that “Disable Native Texture” isn’t enabled. If not, then I would look for a log message in Xcode to the effect of “Texture data conversion required for … Not supported yet, falling back to readback.” If you see that, it’s worth trying a different RenderTexture format. Our RenderTexture test uses R8G8B8A8_UNORM, for example.
I’m very happy to hear how this is intended to work! Unfortunately I’m unable to achieve this so far.
I checked that “Disable Native Texture” isn’t enabled in the runtime flags (it isn’t), and there is no log message like you described. I also tried explicitly setting the RenderTexture format to R8G8B8A8_UNORM (it was previously using the default format), with no change.
I had a quick look at the decompiled (ILSpy) code and noticed that the capability is restricted by PolySpatialRuntime.GetPlatformCapabilities().OutputCapabilities.SupportsNativeTextures, which is returning false for me. This appears to be hardcoded in s_RealityKitOutputCapabilities. I tried overriding this by setting it to true on startup, but hit an EXC_BAD_ACCESS in PolySpatialRealityKit.CreateOrUpdateRenderTextureAsset a few seconds into the app, so I’m guessing this isn’t ready yet.
I’m not sure why you’re getting EXC_BAD_ACCESS, but SupportsNativeTextures should be true. I will check the released packages to see if I can find out why that isn’t the case. In the meantime, it would be very helpful if you could file a report with a reproduction of this.
I believe this is due to the way we compile the package assemblies (the capabilities are relying on proprocessor definitions, like UNITY_VISIONOS, that aren’t set at compile time). We’ll definitely fix this for the next major release, if not a bugfix release. Thanks for pointing it out!
The RenderTexture performance should be substantially improved in the 0.3.3 bugfix release (although we have a report of factors other than texture transfer affecting render-to-texture performance, so your mileage may vary). If you continue to have issues, filing a report would be much appreciated.
The underlying cause of the crash is that the native texture pointer for the RenderTexture is invalid. The first time PolySpatial attempts to transfer a RenderTexture, it caches the native texture pointer associated with the RenderTexture instance (after calling RenderTexture.Create to make sure the native texture exists) and sends that same pointer each subsequent time. We do this because calling GetNativeTexturePtr has a synchronization cost and trying to retrieve it every time the RenderTexture has changed has caused issues for us.
I’m not sure why in your case the native texture pointer is either invalid at the beginning or is becoming invalid, but if it is possible to create a repro case that you can send us, that would help greatly. Alternately, if you can think of anything special or unusual that you’re doing with RenderTextures (like releasing and recreating them), that would help us debug the issue.
That said, I am no longer seeing the issue that caused us to start caching native texture pointers. I think we can change that code to retrieve the texture pointer every time we transfer the RenderTexture, which may address your issue.
Thank you! I finally found the cause – one of my RenderTextures happened to be bound to a camera that was destroyed when the scene unloaded. When the camera is destroyed, unity releases the bound RT (without printing a warning), which destroys the MTLTexture (it will be recreated again as soon as it’s needed). So, relying on the native texture pointer never changing for an RT does not seem entirely correct, but this was easy enough for me to fix on my end once I understood the problem.