Experiments on performance difference between different types of numbers

News & General Discussion
1

Hello everyone!
I’ve recently been working on a large open world space project and I (obviously) had to tackle the issue of float precision. This lead me to think of the performance difference between available number types so I made a few tests that may interest some of you.
For each operation I made a function that applies a few operations to a variable of different types. I repeated this functions quite a lot of time and calculated the time to execute the total loop using Time.realtimesincestartup. The results are given in nanoseconds, approximately represent the time taken to make one operation (like one addition or one sqrt) AND SHOULDN’T be used to compare the speed difference between different operations(they’re probably about the right magnitude order, nothing more), only between different var types. For each type I ran the code ~400 million times and I checked the consistency by running all of that a few times. I used a laptop with a 3500u at ~3GHz.

Here are the results I got:

9448676--1326173--upload_2023-11-3_16-45-8.png
9448676--1326173--upload_2023-11-3_16-45-8.png1026×332 18.7 KB
(Thanks Peter77 for the table)

This isn’t an accurate and reliable comparison, but I guess it can offer a good enough estimation.
To resume the results, from best to worse:
Int<Double<Short~Long~Float
This seems quite surprising (especially double beeing faster than float) and if someone with a deeper understanding of computer science than me could explain that, it would be interesting.

Edit: here’s a resume of what’s been said below:
Basically, this test isn’t really useful, wether it be because of everything required to do a meaningful test or the fact that this timings are so small that the difference doesn’t matter except maybe for the most extreme cases (in wich case the actual benefit of using different types will depend on your situatuion). Also sorry for not posting the source code, I’ll definitevly do that next time :wink:

2 Likes
2

That double is massively faster than float is indeed certainly strange.
Apparently in your testcase, not being able to make use of the CPU “pipeline” efficiently does even result in an additional penalty. That’s seen in short vs int as well.
Without even having your code it’s impossible to tell what’s going on though.

If you have just a plain, tight loop, I’d suggest trying loop unrolling to end up with the same amount of bytes each iteration. That could be interesting.

Would unfortunately suspect that your testing method has issues. The JIT just has a lot of unfluence. There is a reason why for regular C# environments, this extremely complex benchmarking suite has been developed: GitHub - dotnet/BenchmarkDotNet: Powerful .NET library for benchmarking
Because it does weird hacks into the interpreter, it is not compatible with Unity’s Mono though.

Oh speaking of Mono - have you tried an Il2cpp build?

2 Likes
3

I havn’t heard of BenchmarkDotNet so thanks, i’ll definitively take a look at it! I havn’t tried with an IL2cpp build, maybe later but now I don’t have time unfortunately.
For my testing method, there’s indeed probably some issues… Maybe one of the problems is I heard unity does some of the calculations marked as float as doubles, then converts the result to a float. There’s probably a lot of smarter and more experienced people that made better tests but I couldn’t find any. Anyway I guess that most of the time, in performance isn’t that big/insignificant for most use cases. However i’ll just ask if you know wether longs are faster than doubles? My instinct would say yes but CPU’s architectures are so complex it’s problably more complicated than that.

4

We’d have to see the code but I wouldn’t be surprised if this were Unity’s weirdness moreso than anything else. I do wish you’d have posted this in Scripting though rather than in this section as those most able to provide insight (eg @lordofduct and @Bunny83 ) don’t visit (or only rarely visit) General Discussion.

1 Like
5

Agreed, I’d like to see the code used. Also I’ve never really used Time.realTimeSinceStartup for my tests as I’m unsure of its implementation under the hood. I usually use something like System.Diagnostics.Stopwatch.

Also your hardware can impact things here. For example while 64-bit is pretty much the standard today, that fact impacts things. If you tested on 64-bit but then built for a 32-bit phone (older cellphones can often be 32-bit) you’ll get very different single/double performance. And when dealing with more complex math operations things like trig methods and sqrts could actually perform fairly differently from intel to ryzen to arm/risc.

Lastly this leaves me wanting to see the code as well.

This says to me that for the ‘float’ you relied on Mathf. But Mathf for most of the methods are just wrappers around System.Math with a casting to and from double:

        //
        // Summary:
        //     Returns the arc-tangent of f - the angle in radians whose tangent is f.
        //
        // Parameters:
        //   f:
        public static float Atan(float f)
        {
            return (float)Math.Atan(f);
        }

        //
        // Summary:
        //     Returns the angle in radians whose Tan is y/x.
        //
        // Parameters:
        //   y:
        //
        //   x:
        public static float Atan2(float y, float x)
        {
            return (float)Math.Atan2(y, x);
        }

        //
        // Summary:
        //     Returns square root of f.
        //
        // Parameters:
        //   f:
        public static float Sqrt(float f)
        {
            return (float)Math.Sqrt(f);
        }

        //
        // Summary:
        //     Returns the absolute value of f.
        //
        // Parameters:
        //   f:
        public static float Abs(float f)
        {
            return Math.Abs(f);
        }

This means your single float test has the overhead over the method call and the casting.

It means your first double test has the overhead of function calls, casting, as well as precision failure.

The last double test is the only one that’d might be accurate-ish.

…

In the end though I generally use the type that fits my needs.

For starters comparing floating point types with integer types is just not even a go to imo as their uses are completely different spaces. Floats for rational values and integers for… integer values. Fixed-point optimizations might come into play if I was specifically targeting a platform that either 1) didn’t have floating point acceleration or 2) the predictability of precision is necessary over the desire for range.

When comparing integer types within themselves I generally stick the the bit-size I need in the moment, landing usually around 32-bit as ‘int’ is generally purpose. Performance concerns really only ever come in when I’m needing to pack large amounts of data… which… doesn’t come up in games very often for me. But it can at times.

As for double vs single. Again its not really performance of arithmetic especially since most modern CPUs do it all as double under the hood (actually it’s usually larger to account for overflow and then truncated back to double or even single depending on your code). Makes sense… why ship with both a single and a double floating point unit in the CPU when the larger can do both?

In unity I’ll stick to single/float because most of unity is single/float in most all cases. If I need double because of a large map like you suggested above (another I use double for is some time stuff, singles lose millisecond precision in only a few hours). My main concern would be limiting my conversions down. Especially because those could introduce the very errors I was attempting to avoid. You know… like using Mathf for doubles like you did in your examples. This can effectively defeat the entire purpose of using a double since every calculation is converting to a single/float.

4 Likes
6

Almost everything was mentioned above what I would have written, so I won’t repeat those things. The only one I haven’t seen is:

What are you trying to measure exactly and in what environment?
Unity is a complex software with many use-cases and many systems.
Some of those systems work purely on the managed side, their performance theoretically could be measured this way and even maybe would be useful.
But Unity also has a C++ side where everything you just measured is out of the window. Not the same libraries, not the same code, not the same quirks.
And then what is the backend if you only measure the managed side? Mono? IL2CPP? Burst + new math library?
Not to mention when it comes to performance of a game engine, the pure double or float calculation performance isn’t even that relevant, physics engine stuck in unmanaged libraries, you need to marshal your data from the managed side anyway. And then, what rendering will you use? HDRP has camera-relative rendering, which means it works just fine through floats…

In this current form your performance measure isn’t really measuring anything useful. You need to answer the question, what is your goal with this first and then build a test-suite to measure the relevant code paths, libraries and methods you will actually want to use.

3 Likes
7

To be honest, measuring individually math functions is irrelevant.
While it is good to know, which method is better than other, most overhead will come from code logic. And for example knowing how to write code much branchless.
Burst and multithreading saves you much more hassle on shifting origin, than worrying, if sin and cos arithmetics are too costful. Most likely you won’t even notice for 10k number of game instances.

And other thing is, which was posted while ago on Unity forum, that arithmetic like a + 5, can yeld different performance than 5 + a. I have never tested that however.

If considering performance of arithmetics, should look into Unity.mathemtaics math, instead Mathf.
Since you can write code optimised for burst.

It is unclear from the test, but seems it is missing on nay matrix tests. Or quaternions.

Time since startup is also not good, as running code may need warmup. It is unclear, how this effect measured time, and if warmup time is spread evenly across different math methods. So as others said, diagnostic stop watch is a better solution, which should be start measured, perhaps few seconds after startup.

So definatelly need script here. Otherwise test is meaningless to be honest.

8

Here it’s as a table, perhaps easier to read for some people :slight_smile:

9448676--1326173--upload_2023-11-3_16-45-8.png

3 Likes
9

This is literally the most pointless test ever, based on how it was performed and coupled with the lack of source code.

Even if there was a major perf different between these types, for it to become a real concern would require you to be using them at a ridiculous volume

Anyone reading this thread should just use the type best for their needs, and leave micro optimization to engine developers and those who it really matters to.

2 Likes
10

That feels like something the compiler should do on its own. It can already reorder statements after all.

True dat. Like let’s take the highest value from the table: 56 naneoseconds.
That means in a game running at 120FPS where you have about 8ms per frame, you could calculate about 150 000 pow operations!
Add and mul would be by factor 150 more.

Furthermore those are the raw calculations with L1-cached data. What often happens in real games or software in general is that the CPU has to wait for memory which is usually way slower.
Memory-aware programming is therefore what you should focus on IF you actually end up in a performance bottleneck (and can confirm that the cause are truly calculations and not the GPU or stuff like memory allocation or the garbage collector).
In Unity you can use Burst Jobs which due to its restrictions, practically enforces memory aware approaches. And then you also have DOTS ontop if you truly dedicate yourself to large scale…

This is an interesting article on that topic: https://gametorrahod.com/memory-awareness-part-1/

1 Like
11

Compute shaders too. A quick glance at the API shows support for 32-bit floats, 32-bit integers, and booleans in addition to support for the usual complex types like vectors, matrices, textures, etc.

https://docs.unity3d.com/ScriptReference/ComputeShader.html

1 Like
12

True!
This makes use of those and would likely be impossible otherwise:
https://assetstore.unity.com/packages/tools/physics/fluxy-2-5d-fluid-simulator-203795

The way I see it is there’s sort of a chain that goes from “easy to use and versatile but also technically slowest” to “complex and narrow application but with the highest potential to unleash the systems power”.
Tends to do somewhat like:
Object-oriented, regular C#-> data oriented regular C# → data oriented, multithreaded C# → Burst jobs (due to Burst compiler, inherently multithreaded but also the mathematics library) → Shaders&Compute Shaders.
Somewhere inbetween should be “external C++ Dll” which also allow to unleash a lot of potential, but only as long as you don’t have to marshall a lot of data back and forth between Unity and the dll (and that becomes a pain for multiplatform support).

1 Like
13

Well thank you all for all the replies! I think it’s pretty clear now that this test isn’t really useful, except maybe to show that the time difference between all types is so ridiculously small that it isn’t even worth considering.
To explain a bit more my concern: I’m working on an n-body gravity simulation. To get both accuracy and a large margin of “time speed up” I’ll need to do a lot of calculations/frames, even using better integration methods such as Runge-Kutta 4. However it’s now clear I should definitevly focus more on code optimization, multithreading support… It’s been a while since I havn’t been on Unity :wink:
I’ve also started learning DOTS which is easier than I thought although it’s obvisously not as straightforward as GameObjects but I’m a bit concerned by the few up to date tutorials compared to “regular” programmation.
For all that asked for the source code: sorry i’m a maniac with useless files and I already deleted it…

Oh and I’m also messing with compute shaders and shaders in general for stuff like terrain generation or atmosphere simulation. They are really interesting and of course amazingly fast for parallel stuff!
Anyway, thanks everyone for your time and have a nice day

14

Well, picking the most ideal variable types is a valid part of this.

Some things to note:

First, realTimeSinceStartup is based on the standard system timer, which isn’t super accurate. Check out System.Diagnostidcs.Stopwatch, which uses a high-resolution timer if available. This even lets you time parts of an algo, rather than the whole thing.

Next, between starting and stopping your timer, you want to avoid all possible overheads other than what you’re measuring. No function calls, no accessing memory (unless it’s a part of the test), just the simplest possible loop with a large number of the actions being measured. You want to maximize your “signal to noise” ratio. If I’m measuring a single add operation then the signal to noise ratio is terrible at best, because the loop is already far more work than the add and will dominate the measurement time.

Last, rather than measuring literally a single operation, I’d measure working implementations of the algorithm I was trying to optimise. This is partly because of the signal-to-noise issue above. It’s also partly because CPUs, compilers, software stacks, etc. are complicated, and it’s entirely possible that the advantage I get in one area with a change can be smaller than inefficiencies it introduces elsewhere. So, if I measure many runs of my algorithm which involves reading numbers, doing some maths, then writing a result, all of which has to happen in a loop anyway, now my signal to noise ratio is dominated by the actual work I am measuring, and I’ll see the sum of benefits and disadvantages. I can make a bunch of variations of that algo with different data types, using different data structures, different implementations of the math, etc. and measure both the overall completion time of each, and relevant sub-parts of it where helpful.

There are whole books written on this, but together, those things should already help you get actionable information.

Unfortunately not. :frowning:

I recently saw an example of a standard math implementation in a major development platform where, regardless of the original type, some operations were implemented by casting to the biggest type, doing the job, and casting back on return. So the cost of any float operation was the cost of the equivalent double operation, plus two casts.

4 Likes
15

That is indeed the case for mathf functions - the source code confirms that:

But that cannot be the explanation for it happening to “add” and “mul”, right? Or you mean the internal implementation of operators?
Unity has much control over the Mono interpreter, so it would certainly be strange if they’d not optimize that.

16

Why not? Those things are still defined somewhere. But I’ve not looked at those implementations, so I’ve no idea one way or the other.

17

I mean we are talking about “+” and “*” those are part of the language… Why should any interpreter/compiler do type conversion there, let alone one that has been modified for a game engine?

However, we are currently speculating based on data that is well possibly highly flawed!

At very least a loop unrolled version would have been very interesting since I rather suspect instruction prefetching to be the culprit.

If I’m bored next week, I’ll cobble together some benchmarks…

18

Those are part of the language for built in types, sure. C# does not have built-in add and mul for Vector3 and Quaternion, for starters. As for “the language”, it has at least 2 implementations, and allows overloading, and is only one part of a significant stack between us writing some code and electrons flowing along some metal.

Anyway, I wasn’t saying that it should do type conversions, and I specifically did not speculate that it is doing any, I was simply saying that it’s not impossible.

2 Likes
19

Just compile to il2cpp first because it can make a huge improvement to this kind of thing.
Otherwise there are some surprising performance failings in C# with float, but you will find double is fast, the problem is you have to convert it back to float at some point for Unity and the graphics card in general.
Also Mathf will give penalties for float indeed - at least if this is still the implementation: UnityCsReference/Runtime/Export/Math/Mathf.cs at master · Unity-Technologies/UnityCsReference (github.com)