I have read from Unity Blog long time ago that talks about project stripping, explaining reproduce steps, making it easier for the QA team to make it into an actual bug.
I understand, there are way too many thousands of fake bugs either from inexperienced developer getting into their own problem that you can’t simply follow just mere words or clues for everyone. A reproducible project is the king that helps QA team get to work fast, getting real bug fixed.
But one sentence that I agree with very much : “If it is a CRASH bug, please always report it no matter what if you could attach reproduce steps or a project or not, or even if it occurs randomly. Because crash contains stack trace that is always automatically helpful.”
I think it was a blog from some 4.x release that the feature was mainly fixes bugs that I can’t find. But I love it. It reassure you that if you happen to run into some mysterious crash, at least Unity team could also be informed about them.
But nowadays that promise became invalid. The QA team became too strict about having reproduce project. And I mean “project”, because simply explaining steps will most likely get a reply to ask for a project anyways. You just can’t get your words pass that gate if you don’t have a reproduce project. (I had reported a bug reproducible with vanilla Timeline and one audio clip, possible to do in steps in a few minutes, but I still was asked for a project with a TimelineAsset and required tracks made for them!)
You need to change how QA team works if you want to get rid of instabilities one at a time.
Crash bug should always be an exception because stack trace screams at specific location in a code. With reproduce steps? Cool. With a project? Even better. But what’s different about crash bug is that the QA team should let it pass to an actual programmer responsible for that code around the stack trace. If I am the programmer, I would very much like to know all the stack traces that could occur at my asserts. Even if the source of that is unknown, often just that leads to huge discovery.
While requesting for repro project make sense mostly (resource wise, spam filtering wise), it doesn’t make sense for one case : if it is a crash bug. You should always treat crash bug with high priority and doesn’t even need a Quality Assurance. You can assure that the stack trace is very high quality as average coders can’t intentionally fake one easily. Maybe you should just QA that the stack trace looks like a stack trace with some Regex.
For example, as I worked on Unity it randomly closes down with this on editor log :
From the stack, it seems like something with animation. And the game of course contains tons of animation. Sometimes it crash even if I am doing nothing as the animations are running. Sometimes it runs as I take action such as resizing the Game view.
But you get the point, it happens sometimes but not always. Even if I attach the project, I would have to tell “just work on the game normally” on the reproducing steps. That is not helping anything.
But because this is a crash bug, luckily I got stack trace that let me know it is something about animation in the first place. So I used the popup that appears after, with “Crash Report” selection locked, greyed out, to submit this valuable clue without the project. I just want to let the programmer around that line of code know.
This is the reply I got :
In the message I even said it occurs randomly while I work on the game, so the report is not even meant to be reproduced. Yet I still get this kind of message, it left very bad taste as if not a human processed my report, but simply see that it has no project and auto replied “I can’t reproduce it”.
You tell me to strip down. But this bug occurs randomly while I work on the game. After stripping down what am I supposed to work on if the game is not the same game anymore? Randomly click things? And what if the bug disappear after I stripped things down? Should I click randomly for N days before I give up? Do I have to submit a new report with full project anyways, that describe that after stripping down X then the bug is fixed? In the end I must submit a full project and I can’t just hand the developer the stack trace.
Weighting with massive amount of bugs you guys receive that you need to always request a project, which I also understand. Then the best solution is :
QA team should make an exception for CRASH bug to get through. It is 100% an actual bug with no need to prove it. You can 99.99% trust that no one wrote the stack trace up by hand just to waste your time, unlike “mere words” bug reports with no steps or project. The cause is unknown but clues are always valuable to the programmer of your engine.
Please make submitting crash bug hopeful again like that blog post from long time ago said. Submitting stack trace is the most I could quickly help you and no way that a stack trace is going to be false negative, please make it count again and not disappeared into QA void. These days it is just too hopeless. Thank you.