After updating from 0.51 to 1.0exp I’m getting a heavy performance regression in my game but I really don’t know how to interpret the profiler data (see attached image).
Is that wait the CPU waiting for the GPU? gpu profiling is not available in Vulkan…
To give some context, I have around 50k entities, most of them hidden by frustum culling at the moment of taking the profiler ss so probably only a few hundred actually visible. Using only 5 materials with custom shaders. All materials and meshes are stored in the same RenderMeshArray.
The thing is, I get around 80fps on the same machine in the same scene when using 0.51 on 2021.3.10
Nothing changed from my side except for the migration from 0.51 to 1.0 and well, HDRP was updated by default since it’s tied to the editor version.
Any idea on where to start looking?
Thx in advance.
It is a lot easier to diagnose these issues if you switch your profiler view to timeline mode. Then you can see what all your threads are doing at once and it becomes a lot easier to identify differences.
No this is not the CPU waiting for the GPU. It’s an performance regression in the C++ parts of Entities Graphics that we ourselves just found out about yesterday and it only affects very heavy scenes. We will try to fix it as soon as possible and already have a plan. It will require us to land and get released in some future 2022.2 beta though so it can take some time.
Ouch.
Thx for the quick response, now I can stop pulling my hair.
Have a great weekend.
Edit:
btw, just to add more info and make sure it’s the same bug. Entities with transparent materials seems to affect performance orders of magnitude more than opaque ones. Like, on a different machine with a better gpu, we can get 50fps on a scene with a few thousand opaque entities visible (still lower than before), but only 10fps on another with less than a hundred transparent ones.
I think this bug hits transparent entities harder than opaque. Transparent entities needs to get sorted so they will be separate draw commands as they cannot use instancing. The bug causes a slowdown per draw command so it’s not impossible that a fix will help this as well.
Transparent entities will be slower than opaque though, just because it cannot instance and needs to get sorted to get correct results. Right now there is no way getting around that
No, not that I am aware of. We’ll make sure to post here in this thread and say what unity version the fix lands for (we can see this as soon as the fix lands).
Ok. I finally got one of my test projects fully functional in 2022.2.0b10.
This regression is bad!
Simulations roughly an order of magnitude smaller compared to what I used to ship with solid 60 FPS performance on 5 year old gaming machines is now struggling to hit 30. Profiler view of the timeline shows worker threads being hit hard with RenderLoop.Prepare instances. I even modified all my alpha-faded shaders to instead use opaque dithered fade and it is still bad.
In the scenes folder, there is a v2 and a v3, and any of the missions inside of them should be immediately playable. Higher mission numbers feature larger simulations. Missions numbered 3 and 4 are where I see regressions for simulation sizes which aren’t usually problematic.
Feel free to use and abuse this project if it helps you resolve the regression and/or communicate progress.
Yeah it is a bad regression, and we are really sorry about letting that slip through.
The fix is entering testing now and we hope to be able to land it as soon as possible. Hopefully you don’t have to wait too long for it to arrive.
Excellent job…can finally reap the full benefits from BRG!!! In a heavy scene with thousands of tiny meshes framerate jumped from 90FPS to 390FPS (without BRG in Entities 0.51 it was ~290 FPS). You can most certainly be proud of this fundamental overhaul.
I’m not sure what is going on with the ScheduleDraw, but I don’t remember this being a thing in 2020.3.
Also, at this scale, I sometimes get spammed with errors "Assertion failed on expression: IsFinite(distanceAlongView) and similar for IsFinite(distanceForSort). And the game will flash white sometimes when these errors start spamming.
It seems that something still thinks there are transparent draws despite switching to dithered fades, so I will investigate that.
Edit: Fixed the transparency issue. Apparently changing the settings in Shader Graph don’t propagate to the already-created materials. However, opaque has the same performance and I still get the errors from the few objects that are still using transparent shaders.
This suggests that something related to BatchDrawCommand.sortingPosition could be going wrong. For example, invalid positions resulting in NaN distances getting computed, which could explain both the assertions and the performance hit, as invalid sorting distances could lead to very bad sorting orders.
You were right! I had NaN position values that were specific to the very large scenes. So the old beta 10 BRG regression numbers were still valid, but I now have new numbers for this extreme scale.
Those numbers are weird.
In full transparency mode, the fix brought ExecuteRenderGraph times from 5.5 ms to 4.5 ms.
But in Opaque dithered mode, the fix brought ExecuteRenderGraph times from 5.5 ms to 0.75 ms.
I don’t really understand why opaques benefitted from the fix since they have nothing to sort, but performance is excellent now and I am error free. Amazing job!
Glad you figured it out. And thanks for being so responsive!
