Unity3d grew a lot and we can see in blogs that have many improvements in the test and quality. But is a fact that each release comes with some major defects.
I will not extend here how this affect unity confiability and how sad is for us developers.
These are some proposes:
a) Do not call a release a version that is not confirmed without regressions bugs. You can roll 10 RC for a year, but a “RC” complete change our decision how to upgrade and we receive a bug.
b) Have a open test group before release. Many of these bugs was easy to detect by simple upgrade, build and run. Until the automatic tests coverage everything you need mass test. I have sure that many of us will be happy to expend a hour to test before each release.
There’s already a beta group with several hundred participants.
Why those several hundred participants didn’t catch the issues people have seen in 4.5’s launch is a fair question, but it does on the face of it suggest that simply throwing more people at the problem isn’t necessarily going to help.
To make it more interesting for unity, give beta option for those with pro licenses or subscription… (kind of extra bonus)
they said something that the problem is getting too much noise (in bug reports etc),
then have some extra field in the reports “this was submitted from extendted BETA group”, so they can be handled in different queue…(lower priority if needed).
It should be limited some way for efficiency (like the current invite only way) or maybe extend it to pro users too. Opening the floodgates is not a good idea. Also what mgear suggested about tagging the reports by extra group might work if it easy enough to integrate to current workflow.
We certainly had bugs at release that we would have considered shipstoppers, so we have just today made a build we will QA and release when we are confident in it, to fix some of those issues (and one more patch cooking already).
I’m not sure that group would be happy about being downprioritized either. If we go to general public with our early beta builds, you would know that the pain beta testers go through is real and that the vast majority of people would not want that. We actually find and fix hundreds and hundreds of bugs pre-ship.
It’s not easy to do, since we don’t have the information to tag a report from user to a serial number. There could be other ways, but none are without effort.
Once you release the floodgates will become open anyway. You don’t need to remove the current process and closed group, simple open for public beta a week (or a day) before release to have feel.
I understand that current closed beta not only test but give opinion and guidance in features. It is not the case of open beta, it is simple a massive random test to safe guard against critical bugs.
We will not say that gizmos are ugly, but that rocks don’t roll anymore