In the past few months like many of you I’ve had the opportunity to try out Unity 4.6 Beta and as a pre-order customer Unity 5.0 Beta. Love the new game GUI API in 4.6 btw. During this time I’ve reported many bugs. I’m sure they are of various levels of quality w.r.t. data captured, repro’ steps etc. BUT in that time I’ve heard nothing from the Unity Team beyond an automated response from the FogBugz system.
A couple of proposals. My 2c.
Some kind of non-automated, bug triage, response from Unity QA.
It would be simply awesome to see some kind of ‘triage’ message from QA for these issues. Something as simple as, ‘we’re looking at this’, ‘forget it!’, ‘not enough information’. Granted in nearly 3 months it looks like the cases have increased by approx. 18,000 (assuming the case numbers are sequential) so perhaps that’s not practical. Are customers using FogBugz as a form of technical support? 18,000 new issues in the past 3 months is A LOT. Certainly I’m guessing a substantial number are duplicates, cruft, already fixed, not reproducible.
Ability to edit/annotate issues post submission.
Any way a customer who reported a bug could be able to edit and annotate the issue? Some of the bugs below got ‘corrupted’ titles and I’d love to improve the message.
Second that. I reported various bugs (at lest 10) in the last two years, and got replies for only TWO of them. All the others are still marked as “open” on fogbugz, and that’s it. At least marking them as fixed/won’t fix/working on… would be nice.
Totally agree with you guys. Not been pleased with the Unity QA for some time. I check the board each day and if you really want a bug fixed you just got keep pushing and pushing and hope that one the devs sees it from your view as a important issue and they then look into it personally.
Thank you developers for the support on here, I think its great especially when you get involved in specific cases although your not QA sometimes its about being proud of your work and really helping others.
Cant say much about the QA as all the run ins I had with them in forum have been negative.
I’ve reported bugs too, but I’ve received manual responses from the QA Team, so they pretty much are responsive for me. I’ve had several responses to the bugs which seem critical.
For sure with this one’s mileage might vary. It’s ok. I’m entering issues and the latest reported case was repro’d and fixed for b15. So awesome as I helped improve the product which is the point with this Beta. I think though, for what it’s worth, that the feedback loop and process could be more consistent and tidied up.
The only unanswered report there is the earliest one, when, I suspect, I forgot to save the demo scene before sending the report. I have always created a small repro project featuring the bug, and the latest reports also have a link to a small screen-cast video.
Yep, I agree that every issue reported should carry succinct repro’ steps and a project that demonstrates the issue if feasible and I’ve strived to do that where appropriate. But my initial point is that in my experience, and yours too apparently, the response tends to be either ‘we’ve reproduced this issue’ or no response at all.
It is kind of a lonely feeling to send a bug and have it listed as “Open” for months. Half of my submitted bugs are still open even though they all contain a succinct repro that is 100% reproducible on my machine. If QA has failed to reproduce it on theirs, I’d like to know so I can help. I suspect in some cases, the bug has simply been set aside. I understand that they have to focus on critical bugs over noncritical bugs, but after a revision is released the bug is then even lower priority for belonging to an earlier version of Unity. I’d like to know once a bug report has been completely abandoned so that I can report it again if necessary.
Tons of those are not bug reports; you get case #s when uploading assets to the store, for example. I do get emails from Unity about bug reports, such as “this was reproducible, handed over to be fixed”, though not all the time.
I guess you are actually in luck. Many also say we reply “We couldn’t reproduce, please send a stripped repro project.”.
Here’s the conundrum. We can decide to do totally closed alphas and betas, which we did for most of 4.x, in whic case we closed out about 98% of all reported incidents (incident = customer reported issue. Bug = Reproduced internally). Or we can decide to go open beta and be absolutely horribly swamped in incidents that are mostly duplicates. You simply can’t have both an open beta and us being able to respond to every incident.
And to give you numbers: In 4.6 we have received 4252 incidents. Of those we have processed 40.5%, while we have also had to process feedback on 5.0 and 4.5. It is really simple. We are not going to process everything and we have a rating script to help us chose the highest chance incidents to look at. (see http://blogs.unity3d.com/2013/10/28/bug-reports-incidents-and-some-bashing/).
It is certainly understandable that this whole process is very time consuming and my personal impression is that it is constantly being improved.
Being someone who always tries to create a minimal reproduction with a short but precise description, it is quite frustrating not to get any feedback at all. Of course, I am not asking for a personalized response, but getting an automated reply with the rating of the incident would already feel a lot better. If the bug could not be reproduced or if it was assigned to a developer (though it is not clear whether or when it might be resolved), it would help the person who submitted the bug to feel more comfortable.
At the moment the process of submitting a bug feels like a huge black box. Some of the bugs took me half a day, just to create a minimal reproduction. After several months without a reply it feels pretty weird, because I have no feedback at all. I don’t even know whether it could reproduced or if anyone had a look at it. Getting an automated response whether you were able to reproduce it, whether it was assigned to a developer or even a response that it won’t be fixed would help the submitter a lot.
When I submit a bug and take the time to make it right, I would like to get some kind of automated feedback, such that I know my time wasn’t wasted. Whenever I submitted one or two bugs and got no feedback at all, I tend to not submit further bugs, because it feels like wasted time.
I’ve been always wondering, since I read that article a while ago, what’s the difference between “very little description”, “small amount of description” and “good amount of description”?
Some bugs don’t really need many words to describe, like
What’s wrong: when I do this, that thing breaks. To reproduce: open the demo scene, press Run, notice that this thing bounces (for examaple) instead of standing still. http://screencast.com/abcdefg
Is this a very little amount, small amount or good amount?
We are really grateful to have users like you, kind sir <3
I’d like to assure you that if your case gets looked by a QA, no matter the situation - even if doesn’t have sufficient info, it’s not reproducible, duplicate or a confirmed bug - you’ll get a response.
While in closed alpha/beta stage we tend to process around 90% of the cases, but with open betas and public releases the amount of reported cases increases tenfold and we don’t manage to handle all of them.
We do prioritise cases like yours (from newest versions and with highest ratings), but right now we have several hundreds of those top-notch cases alone.
We are trying our best to improve our processes and tools (public issue tracker, working on new bug reporter), to increase visibility, minimise your effort with duplicate bugs and handle cases more efficiently. We are also continuously actively hiring to have more manpower reacting to rapidly increasing user base.
Feel free to drop me a pm with your most important and critical bugs, so that I can assure they get looked at.
Thanks for the reply. For me it is very obvious that there were lots of improvements in the previous years on your side. In my opinion the QA Team does a fantastic job!
Even if I am aware that someone checked all the issues that I submitted, it would motivate me a lot more to submit further bugs if I just got a small update like:
We were able to reproduce the issue
The priority of the issue was changed to …
The bug was assigned to a developer
It is unclear whether the issue will be resolved
The bug should be resolved in version … that was just released
Getting just those little information snippets would make a huge difference for me and would motivate me a lot more, just because I could see that something is happening.
Probably I wasn’t clear enough, if some of your cases didn’t get a response - they weren’t looked at, otherwise they would definitely get some kind of update.
As I already mentioned we simply have too many cases to go through all of them, but ones with higher rating have way higher chance to get looked at.
Please drop me a PM if something critical was missed.
Good to hear. I just checked my bug reports. There is around 8 for which I never got a reply, still I took the time to make the description short and to create a small reproduction. My motivation to take any time to submit bug reports just… Well…
I am kind of speechless. Should I spend my time to complain here in the forum instead of creating a meaningful reproduction? Sounds like a dumb strategy, but it seems to work better then mine.
This is actually WORSE because it means at least 1/3 of the bugs I submitted during 2014 have never been looked at. Which also means they never make it into IssueTracker, so nobody can tell if they’re submitting a duplicate bug or vote on fixes which are important for their projects.
Good point also they forgot to mention - any report without a repo does not get looked at , these words of the Qa manager himself in a thread i read month or two ago.
so like someone mentioned, something that does not need a repo is not looked at…simple things and reports without i think they put them buttom…Im just shocked at this way of working and of course i would make noise and make concern about it
