By this I mean benchmarkers often run common App’s to benchmark the PC rig they are testing, usually they use a Video production, 3D rendering and maths applications to get an idea how well the PC will do common tasks.
But have you seen a benchmark where a 3D game engine like Unity are used to build/light a scene and the test rigs performance is timed and compared.
Just as this could be a great way to work out what hardware is better for game development.
For example, You would like to know which CPU upgrade would be ideal for lighting your scenes in Unity?
No. It wouldn’t make much sense. There standard tools already, using something written in game engine adds overhead the renders the results meaning less. It would tell you about the efficiency of the tool, not the pc.
But if your main use for a PC is developing with Unity having a benchmark that compares different PC’s performance within Unity would be ideal for helping you decide on your next buy/build/upgrade options.
Someone would have to build the software and use it to evaluate the most common configurations. To do so would either require considerable time to catch on in popularity with review sites, or you would have to set up a company dedicated to providing those results.
Not really most benchmarks of 3D renderers just use a complex scene/model as the test. In Unity, you could use a similar approach, load up a complex scene and have Unity bake it as a level, navigation, lighting etc the things that are time-consuming in our day to day usage of Unity.
You’re only focused on the effort of building the software. You’re neglecting the main advantage to modern benchmarking suites like FutureMark and PassMark. They aren’t simply offering benchmarking tools. They’re also offering entire databases with statistics.
There are too many variables in configs and usage. Performance could be affected/different based on the hardware, drivers (config and version), os (config and version), what else is running, the build and the competence of the developer of the app, not to mention that a general app is going accurately reflect what the individual developer may be doing. Too many variables Also any modern PC isn’t going generate any significant differences that would matter. Moreover if you are developer, the tool doesn’t give any valuable information, because a dev rig is not always reflective of a player rig, maybe in hardware, but not at all in config. Any dev knows what they need.
As far as development benchmarking, it is very sloppy practice to rely on third party tools for benchmarking your app. You build a tech prototype to simulate your actual (and smoke tests), needs. That way you can see the direct impact of your choices and adjust. You don’t benchmark the device, you benchmark your code/app.
All those things very radically from app to app. That is why developers already do this using their own code. There is no value in testing someone else’s game. It would be a waste time.
Are we even on the same page here, the idea is that if PC/Computer benchmarkers added a Unity benchmark then a Unity developer would be able to make a more informed decision on what PC/Computer to buy as a development rig.
Yes, the best game development PC would probably be a higher end machine than the game’s target hardware, the same way a game made for a mobile device that is not expected to have the performance of the developers computer.
No developer would ever do that. It wouldn’t give you any information other than how one specific app runs. Reading the specs would give you tons more information. That isn’t what benchmarks are. Faster machine is better. OS/software is bigger impact. No, it provide no information that is useful or applicable.
General benchmarks are generally useless. Useful benchmarks are narrow and have something controlled and measurable with a control. Putting a random piece of software on a random piece of software isn’t benchmarking, it is just sort of non specific test.
More informed than what? There are already plenty of information sources for general computing and 3D performance.
Broadly speaking that will translate directly to comparative performance in Unity.
If you need more detail than that then you really have to do tests of your own specific to your use case but, again broadly speaking, there’s not typically a lot of value to that. Consider how much your time is worth, and then consider how much practical improvement you get from obsessing over a few percent difference in performance at a given price point.
I’ve thought about this some more and changed my mind. Do it and run a blog with the results. Put up a few ads on the page. Make a subscription model where people can pay a small fee to get the full reports.
Forget about the core game dev market. There are plenty of new users that seem overly concerned about the specs required to make games in Unity. There is a whole market out there of wannabe devs that might jump on something like this. Especially if you ran tutorials on how to build an MMO. How to launch a game on steam. Direct comparisons between various ad platforms. How to monetise a game idea. All sorts of things.
After all, marketing is often about giving people what they think they want. Not what they actually need or is good for them.
I once also thought about using unity to create a benchmarking tool, but not intended to be used this way. I thought of using it as “Can I run this game?” tool. You could create a very small game (even browser based), in which you could check how much can the user’s computer handle at above 30 fps and give it a score, you can compare that score with a optimal score, like your computer’s and inform the user about results.
