Quick contextual summary: My main focus is GPU instancing of trees in a terrain, and possibly some other things but trees are my #1 performance priority. The trees are rendered using Unity’s built-in Terrain system. They aren’t customized or animated in any fashion, and billboarding is also disabled. The game is targeting Oculus VR and uses the Single-Pass Stereo Renderer. The problem also occurs in multi-pass stereo mode.
GPU instancing appears to be working in the Editor, as indicated by the Frame Debugger, assuming I can trust it. The trees are rendered as part of two to four separate instanced draw calls (see image):
Great! I’d be happier if they were a single draw call, and I suspect Unity’s aggressive front-to-back render ordering causes them to get split into a couple buckets. But four draws is still miles better than hundreds, and is plenty good enough.
Step two, I build a Player for my game and attach Nvidia NSight to profile my scene. According to Nsight, each and every tree is being rendered, one at a time, through DX11 drawIndexed calls. In fact, nowhere in the entire scene is a single call to drawIndexedInstanced being made, despite the fact that according to the Editor there are quite a few things that can and should be Instanced. I’ve worked on other DX11 game engines and I find this highly unusual for an engine that supposedly supports instanced draws.
So naturally I tried changing every Player setting I could find, just to see if anything impacted this behavior. Nada. And yes, I disabled static and dynamic batching and other things. Next, I decided to try to attach the Frame Debugger to the Player and see what it says. But it doesn’t work due to what appears to be a bug in Unity 2018.2 and 2018.3 in which I get an error about multithreaded rendering being required.
This had me super bummed until I learned there’s a work around, which is to specify -force-gfx-mt on the command line for the player. Awesome! When I use that switch I can debug my game’s main menu (woo!) but when I try to attach the Frame Debugger to the scene that sports actual terrain, it crashes with a Segmentation Fault. So much for that. I’m back to being super-bummed until further notice.
So without the aid of the Frame Debugger, and given that my trees are supposedly instanced within the context of the Editor, does anyone have any tips or ideas on how I might get instancing working? Some known issues, or conflicts with specific options? Anything?
Or does anyone have tips on using Frame Debugger on a Windows Player? How to avoid it crashing with a Segmentation Fault?
In the meantime, I do plan to try and set up a test-case using drawMeshInstanced() and see if using that directly (via a command buffer or such) actually allows me to instance-draw anything. If that works, maybe I’ll consider disabling the Tree render in the built-in Terrain system and render trees myself that way. Since our trees have no animation or fx, it should (in theory) be relatively easy for me to use drawMeshInstanced() and get them rendering correctly. But trees are just the most obvious thing: There’s plenty of other meshes in the scene that should also be instanced. An engine-level solution that doesn’t require I custom-render everything would be ideal. But does it exist in Unity 2018?