Parallel theaded work that is fast in BenchmarkDotNet is super slow in unity.

[Eldalote]

Hi,
I have some weird results that I do not understand, and would love some help with;

I have a (at this point still very basic) move search function, that does a recursive mini-max search.
To speed things up, I made it so that the first loop splits the initial search into parallel threads, and then added an option to split the second layer also up into further parallel threads (called double threading in the function).
I’ve run benchmarks on the functions with BenchmarkDotNet, and as expected, the original function doesn’t stress the CPU much, the threaded function stresses the CPU a lot more, and the double threaded version works the CPU the most. This is reflected in the Benchmark results. The threaded version is significantly faster than the original, and the double threaded is a bit faster still.
Benchmark results: FastIsSlow/BenchmarkDotNetResult.png at main · Eldalote/FastIsSlow · GitHub

I’ve also checked the results from the functions, they all search the exact same number of nodes, come to the exact same evaluation conclusion, and report the same answer back. All should be good.

When I put these exact same functions in unity, the results are almost the exact opposite.
The original still doesn’t work the CPU that hard, the threaded works the CPU harder, and the double threaded works the CPU to 100% all cores, yet the speed is completely opposite. The original is by a wide margin the fastest, the threaded is slower, and the double threaded by a wide margin a LOT slower. This is the case when the search is called outside the main game thread in my game, or in a “start” of the single object in an otherwise empty unity project.
Unity test results: FastIsSlow/UnityBenchmarkResult.png at main · Eldalote/FastIsSlow · GitHub
As you can see in the results, the number of nodes searched and the evaluation of them all is the exact same, but still the results are counter to what I expect.

One “solution” that I came up with is to have the searcher run as a separate commandline program, and have unity ask it via the console for searches, but that is obviously not the solution I’m searching for. I’d like to understand why the function that by all rights should be faster (as I understand things) is so much slower.

Demo unity project that only has one object that does the searcher test, used to get the UnityBenchmarkResults picture: GitHub - Eldalote/FastIsSlow: Weird multithreading behaviour in unity

I got these results (both in unity and the benchmarks) on a Ryzen 9 3900X 12 core, 24 thread CPU.

If anyone understands what is happening here, and can explain it to me, Thanks a lot in advance!

Kind regards,

[CodeSmile]

You are benchmarking this in the editor. Try benchmarking it in an actual build, ideally in release configuration.

Also you may want to consider using jobs and burst for a real difference. I can also imagine that using custom threading thrashes the job system‘s threading if it also does parallel work at the same time, so possibly the results you see may be due to thread congestion.

[Eldalote]

@CodeSmile

You are benchmarking this in the editor. Try benchmarking it in an actual build, ideally in release configuration.

Also you may want to consider using jobs and burst for a real difference. I can also imagine that using custom threading thrashes the job system‘s threading if it also does parallel work at the same time, so possibly the results you see may be due to thread congestion.

Thank you for the suggestion, I will look at doing it in an actual build. I will have to look into the jobs and burst, since I am not familiar with it. Thank you for giving me a direction to look to.

[Ryiah]

@Eldalote

Benchmark results: https://github.com/Eldalote/FastIsSlow/blob/main/BenchmarkDotNetResult.png

I’m assuming you ran this benchmark outside of a Unity project? Because Unity’s framework and compiler are out of date too. For example we only have part of C# 9.0 and even that part comes with caveats.

Unity - Manual: C# compiler

[Eldalote]

@Ryiah

I’m assuming you ran this benchmark outside of a Unity project? Because Unity’s framework and compiler are out of date too. For example we only have part of C# 9.0 and even that part comes with caveats.

https://docs.unity3d.com/Manual/CSharpCompiler.html

The BenchmarkDotNet benchmark was compiled via command line dotnet build -c Release.
The unity “benchmark” was done by just pressing play in the unity editor.
Of course I’m not surprised that it is much slower in unity because of the compilation, but I was very surprised that the order of speeds was reversed.

[Eldalote]

@CodeSmile

You are benchmarking this in the editor. Try benchmarking it in an actual build, ideally in release configuration.

Also you may want to consider using jobs and burst for a real difference. I can also imagine that using custom threading thrashes the job system‘s threading if it also does parallel work at the same time, so possibly the results you see may be due to thread congestion.

I did a release build of the example unity project, and it still had the same behavour of the function that should be slower being faster than the one that should be faster. In fact, for some reason, the results were actually slower than just pressing play in the unity editor. Screenshot of the results:
https://github.com/Eldalote/FastIsSlow/blob/main/ReleaseBuildUnityResult.png

I did a little research on Jobs, and what I gathered is that it is a unity specific thing, not a C# wide thing. I’d rather use C# generic code, because honestly I’m using unity more as a pleasing way to visualise the results of the programming than the end goal of the programming.

[Sluggy1]

@Eldalote

I did a release build of the example unity project, and it still had the same behavour of the function that should be slower being faster than the one that should be faster. In fact, for some reason, the results were actually slower than just pressing play in the unity editor. Screenshot of the results:
https://github.com/Eldalote/FastIsSlow/blob/main/ReleaseBuildUnityResult.png

I did a little research on Jobs, and what I gathered is that it is a unity specific thing, not a C# wide thing. I’d rather use C# generic code, because honestly I’m using unity more as a pleasing way to visualise the results of the programming than the end goal of the programming.

If it makes you feel better my experience has been that many relatively simple cases will perform better using C# Tasks rather than the job system. Jobs are really more if you are already leaning into the DOTS workflow and working within it’s limits or the work being done is fairly heavy and outweighs the overhead of spinning up jobs.