Overall Performance degredation in 2017.2

Unity Engine
1

The (first-person) game I’m working on features an automated performance test. It works like this:

  • Camera is placed at a defined position in the scene

  • Camera makes a 360 degree rotation around the Y-axis within 20 seconds (slowly rotating around y)

  • Every second the test captures the average CPU frame time of the last second

The game runs just like it would normally do, but the camera/player does not move and the AI in unable to see the player.

I ran this test with three different builds (Standalone Windows 64bit):

  • 2017.1.1p3

  • 2017.2.0f1 without any modifications

  • 2017.2.0f1 with Physics.autoSyncTransforms=false, Physics2D.autoSyncTransforms=false, Physics2D.autoSimulate=false

2017.2 performs worse in every scene compared to 2017.1. Some scenes are a bit slower only, while others are significantly slower. I picked a few random results and exported the data as images, which you can find below.

The y-axis represents the average CPU time in milliseconds, how long “one frame took”. The x-axis represent the sample points while the camera makes its 360° rotation. Each test runs for 20 seconds, one sample every second, thus 20 samples for each test.

3231659--248035--scene_4_3.png

3231659--248036--scene_5_4.png

3231659--248037--scene_6_8.png

As shown in the last image “SCENE 6_8”, 2017.2 is up to 10ms slower than 2017.1 in my game. Using the new Physics Optimizations makes it run faster, but it’s still 2ms slower than 2017.1.

Is anybody seeing similar performance issues? Is Unity Technologies aware of these performance issues?

7 Likes
2

I can’t find the thread right now but the unity devs were responding to some questions regarding the autosync stuff. From what I was able to gather, by default unity had autosync on so it worked as it always had. (No need to manually sync, raycasts always hitting the most “up to date” versions of objects, etc). It sounded like this should just perform as well as it did in 2017.1. But, now with the property exposed, the user could further optimise their game by disabling autosync and performing it when they need to. This should make their game run better as the sync isn’t performed as often as when it is set to true (the default value).

What’s confusing was that having autosync set to true was for some reason slower than 2017.1, and from your tests it seems even having it turned off isn’t as good as 2017.1.

Seems something is wrong somewhere? One would expect autosync true to run as well as it did in 2017.1. the average user would just upgrade their project from 2017.1 to .2, expect everything to run as it did before (and it will, because autosync is true by default) without any performance drops (which it doesn’t, it is a lot slower). They shouldn’t be expected to make use of a feature targeted at more advance users just to achieve a frame rate that they were hitting in earlier version of the engine (where they weren’t doing anything special - just using the physics as is out of the box)

3

Did you also run gfx only test ?

My test was in-game with full physics ai and post stack running HDR linear with a controller so it is not automated at all.
5.6 was around 140-70 fps
.1 was 120-50
.2 was 70-20fps
Big use of kinematic dynamic physics 2d with 3d rotation which forces a collider rebuild, didn’t auto off

4

Can you pinpoint which profiler tasks seem to take more time than before? Is there a lof of UI or physics in your test scene? Animated skinned meshes? Things that might have used GPU instancing or static/dynamic batching? Occlusion Culling? Are you using .Net 4.5? DX11 or 12? Graphics Jobs? GPU Skinning? These are all things that seem to be at-risk in 2017.2

Comparing all the physics times between 2017.2 autosync off and 2017.1 might already give us interesting info. Maybe they are the same, and the perf difference is due to something else

Another possibility is that things are more and more getting converted to be multithreaded by the job system, but since the job system isn’t fully there yet, it just ends up being slower than before

I think there is some confusion as to what autoSync on is. Basically, Unity needed to make “breaking” changes to the physics engine in order to make it ready for multithreading with the jobs system (and I am really proud of them for doing that). This change means that all transform syncs must now be deferred, so it pretty much changes everything, and there is no real “equivalent” to how things were before (neither with autoSync on or off). But AutoSync off is the closest equivalent to how things were before in terms of performance. AutoSync on is just a “compatibility mode” that should give you the same end results as before in 99% of cases, but is extremely inefficient. AutoSync off is the “real” way of using the physics engine now.

But the problem now is that AutoSync off should at least be as efficient as before

I know for sure that having autoSync on shouldn’t even be considered an option. It should be removed from the engine completely, because the performance hit is just preposterous. I’ve tested some scenes that ran at 200 fps in 2017.1, but ran at 8 fps in 2017.2 with autoSync on. However, with autoSync off, I didn’t really notice a performance difference (but it was a really simple scene, too. Just X amount of rigidbody cubes moving around)

But I’m all in favor of the manual transform sync feature. I’m glad Unity chose to do this. What I’m not at all happy about is that they are going ahead with 2017.2’s release when it clearly isn’t ready yet

2 Likes
5

I see similar problems too, even if autosyncTransform is false.

6

Thanks Peter for this report and all for follow-ups – data you are presenting basically reinforces the behavior I’ve been seeing. I was on 5.6 patch releases for some time and recently tried moving firstly to 2017.1 and now 2017.2. Main reason was the touted improvements for features and performance, but honestly things are going backwards right now.

As per the data and discussions here things have gradually degraded in performance. 5.6 >> 2017.1 >> 2017.2. Things seem way less stable and am losing performance in many ways, as per recent bug reports. Not complaining, however it’s a tad confusing and disheartening at times, particularly when we are looking at imminent release (again).

In hindsight I should have stayed on 5.6 and perhaps will go back to that once more, until I see more evidence that the newer versions of Unity are stable and ready for real usage.

2 Likes
7

I could do that, but I decided to take a break from looking at new Unity releases for now. It’s too frustrating to see all the (optimization) effort I put into the project falling apart with pretty much every new Unity release. I was trying to keep the project up-to-date with Unity, because I was looking forward the Job- and Input-System.

It’s really astonishing how Unity gets slower with (almost) every new release, I can’t get this into my head. It takes so much time and effort to raise these issues. I feel like I spend more time running into Unity issues than actually working on the project, that’s not how it should be when using middleware. I also don’t get anything in return.

I’ll check 2017.2 after a few patch releases if performance improved. However, from past experience, I know these type of performance issues often lay around a really long time before being fixed.

7 Likes
8

Hey but at least we can login to Unity editor with facebook now

9 Likes
9

I’ve decided to give this a try, and I made my own performance test.

Video: https://i.gyazo.com/0a290f007cb591d3604491169a7d30b5.mp4

1119×630 226 KB

I call this test… “The Adams Family”

This first test is mostly just to benchmark rendering performance. The scene has:

  • The entire Adam Exterior Environment
  • Static meshes
  • Skinned meshes (all the Adams are doing a walking animation)
  • Particles
  • Foliage
  • Shadows
  • Many lights
  • Transparency
  • PostFX v2
  • Etc…

The project settings are:

  • DX11
  • .Net 3.5
  • Linear color space
  • Static batching on
  • Dynamic batching off
  • GPU Skinning on
  • Graphics jobs on
  • Deferred rendering
  • Very high res shadows (distance shadowmask)
  • VSync off

I tested the scene both in 2017.1p5 and 2017.2b11, with the exact same settings (except I disabled autoSyncTransforms in .2, even though I don’t think there were any moving rigidbodies in this scene.)

And the results are… 2017.2 performs a tiny little bit better. About 15ms per frame in 2017.2 versus 16ms per frame in 2017.1 (in-editor). In a release build, both hang out at around 44-48 fps. Hard to tell which one is better.

HOWEVER!!!
I noticed something pretty crucial. When I first built the executable after converting to 2017.2, it ran at 32 fps. Then my spider senses told me to try and delete the lighting data (just realtime GI in this one) and recalculate lighting, and see how it goes. After I did that, the performance was back on par with 2017.1 (a bit superior actually). I remembered having this problem in the past. Whenever you convert projects to new versions, be sure to recalculate lighting (maybe even delete the lighting asset completely first. You can never be too sure).

Later this week I’ll do a physics-centered test and report back here

9 Likes
10

Hi guys, just wanted to let you know how great of a job you do with those comparisons, many thanks for those!

11

It would all be so much simpler if Unity released its own internal performance benchmarks with each major engine version

2 Likes
12

Not really because a lot of them are user projects or asset store projects, ie real world scenarios… They can’t do those. I guess they can do whatever other benches they’ve made but I think we can too.

I’m not sure there’s a lot of value here since Unity does run through those benchmarks as far as I understand it, and it still got released. I can only assume they’re wilfully skipping doing benchmarks sometimes or it’d get caught.

Would be nice to get more info.

13

Just did the physics test

The test scene is 3600 interpolated rigidbody spheres freefalling into a box, and 400 interpolated kinematic rigidbody cubes that are moving, rotating, and raycasting on every fixed update.


I call this test… “Eh… that’ll do it”

Here are the results in a release build:

  • 2017.1p5: Goes down to 56 fps when all the spheres are landing in the box, then gradually goes back up to 105 fps
  • 2017.2b11 - AutoSync OFF: Goes down to 65 fps on initial impact, goes back up to 115 fps
  • 2017.2b11 - AutoSync ON: Goes down to 16 fps on initial impact, goes back up to 45 fps

So, have no fear: the physics performance is fine in 2017.2 as long as you remain in AutoSync OFF mode, which is no big deal at all. In several cases you may not even need to ever call Physics.SyncTransforms() at all. The only things that aren’t fine are:

  • AutoSync ON by default, which will no doubt cause massive hysteria and make thousands of devs wonder why their game suddenly runs like a console game
  • this problem , which - thank god - has been resurrected from its “Won’t Fix” status and is now back to “Active”

This doesn’t confirm beyond any doubt that 2017.2 has no performance regression whatsoever, but it’s still somewhat reassuring I think. The only way to find out what did regress now would be to compare profilers for the same project in .1 and .2

Also, I tried the Adams rendering test with DX12 and Vulkan. The performances were abysmal with DX12 (it ran at 25 fps), and it just didn’t work at all with Vulkan (1 fps and everything is purple). Just stay away from those for now, and stick with DX11

4 Likes
14

AutoSync has always been on by default. they are just now giving a way to turn it off for the first time. If you turn it off, you’re going to have to create a replacement system to track when items need synced. So it’s questionable if the performance gain you might get is worth the headache.

15

That’s a common misunderstanding. It really sounded like this was the case in the draft documentation on AutoSync, but it isn’t. See the explanation here : https://discussions.unity.com/t/673769/21

Basically, the closest thing we have to how things were before is AutoSync OFF. But overall it’s a totally different system now, and there is no real equivalent

1 Like
16

Ok they really need to put his stuff from comment #9 into the Google Doc, because the way the Google Doc reads, it’s the complete opposite. Very Confusing.

Now I just wonder what constitutes a “Physics Query”.

17

Raycasts, Overlap tests, etc…

1 Like
18

@PhilSA - The way it is worded here:

Sounds like AutoSync ON is the old behaviour… This crap is so confusing. I wish the devs would chime in again and clear up this confusion.

In the old system, any ‘Physics Query’ was always up to date.

For example…:

Prior to AutoSync…:

transform.position = new Vector3(100,0,0);
Physics.OverlapSphere(new Vector3(100,0,0),0.5f); //true = we find our object at 100,0,0

With AutoSync On

transform.position = new Vector3(100,0,0);
Physics.OverlapSphere(new Vector3(100,0,0),0.5f); //true = we find our object at 100,0,0.
//Position is Auto synced to 100,0,0 prior to Physics Query

With AutoSync Off

transform.position = new Vector3(100,0,0);
Physics.OverlapSphere(new Vector3(100,0,0),0.5f); //false, transform has not yet been synced,
//Physics thinks object is still at 0,0,0

Alternatively…

transform.position = new Vector3(100,0,0);
Physics.SyncTransforms();
Physics.OverlapSphere(new Vector3(100,0,0),0.5f); //true = we find our object at 100,0,0.
//Position has been manually synced
19

I’ll admit my “the closest thing we have to how things were before is AutoSync OFF” statement was poorly worded and needed a pretty big clarification.

What I meant is that AutoSync ON is just a big heavy brute-force ‘hack’ that was added just for the sake of backwards compatibility, even though it pretty much divides your framerate by 4, and it’s not at all representative of how the physics operated before. The engine didn’t call a “sync” before every physics query in 2017.1. It just operated in a way where it didn’t have to do that

But it’s true that it gives more or less the same end results as before. I say “more or less” because there are cases where it doesn’t.

If, with AutoSync ON and within a single frame, you:

  • move a transform to a position

  • do some physics operation that needed to see what would happen if that transform was there

  • then move the transform back where it originally was

  • and THEN move a rigidbody on the same GameObject as the transform by using MovePosition (for interpolation)

… it’ll result in no movement at all. I know it sounds like a really far-fetched use case that no one would really need, but I encountered a situation where I needed that (it has something to do with character controllers and moving platforms). And the solution ended up being to call a Physics.SyncTransforms() between steps 3 and 4 even though AutoSync was on, so that it would un-dirty the transform, and let MovePosition do its thing. ( here’s an unintelligible post that sort-of talks about that )

20

Yep, and this was my concern… I sure hope the performance benefits with AutoSync Off pays off because a majority of developers will never take advantage of a feature like this. For a majority of devs upgrading to 2017.2, I can see 2 outcomes (neither of which are good)

  • TrueSync On by default: Experience a huge performance regression - up to 4x worse performance, just because they upgraded (“If I downgrade to 2017.1 my scene is 4x faster! 2017.2 is broken!!”)
  • TrueSync Off by default: Experience strange new bugs and quirks in their logic that worked fine before that they have no way of explaining (“If I downgrade to 2017.1, it just works! 2017.2 is broken!!”)

And if TrueSync doesn’t end up offering huge performance benifits, I am having trouble understanding what the point of all this is.

Edit:
Just to be clear, I am happy Unity is trying to innovate and experiment with new things!

3 Likes