Your thoughts about reporting bugs?

Unity Engine
1

I was adding a FAQ-like question about bug reporting to our answers site when one of our trusted community members, Ashkan, replied to it as I was a part of the community and not a hired QA Specialist here at Unity Technologies.

His answer looked very much like some of the answers that I reply with to some of our users that are a bit unclear about their problems, so it made me think :idea:

  • From the community’s POV, what is the best way of reporting bugs?

  • Can you help us at UT concentrating on fixable bugs by asking each other if you doubt whether your problem is a bug? (Of course you can - and you are good at it, but are you OK with that?)

  • Do you have any wishes concerning the way we process bug reports?

  • What kind, level, and exposure of information would help you better explain your problems with Unity?

2

I’m just going to write what I’ve written before in order to make sure it gets implemented. If you’ve read this before and taken it to heart, thank you!

The bug reporter has been my nemesis since I started using it. It gets
the job done, but it’s missing all the little things that make
reporting a bug pleasurable. Instead, I always feel like I’m fighting
it or just stepping on it because it’s asking the wrong questions.
I’ve been using Unity for three years, and I know what you guys want
me to report. Even though I know what I’m doing, the current bug
reporter makes me feel bad giving you the right information.

I filed a report on it over a year ago. I’ve updated it a bit to
reflect changes in the reporter since then, but most of it is relevant
unchanged.

After reading what people have said already in this thread, I would
like to add that I agree with the idea of novice and expert modes.
However, I think the names should be “simple” and “detailed”. It’s a
tiny bit of psychological warfare, but it’s inoffensive while
encouraging people to use the latter mode. People who want to submit
simple reports will do so no matter what. People who want to submit
good reports might still be scared off by the name “advanced”, whereas
“detailed” is just an invitation to provide more information.

Also, if there are two modes, please make sure that the reporter
remembers which one the user last picked. If I have to switch to
“detailed” manually every time I report a bug, I’ll just stop
reporting them. Well, not really, but it’ll be annoying.

The original (slightly edited) case 34855 :

Unity Bug Reporter has a mismatched set of fields.

Reporting bugs is made slightly confusing by the fact that some
instructions and fields are irrelevant for some types of problems. I
think that some language could be changed and some fields could be
made dependent on the type of issue.

Problems and solutions I see with the current setup:

“Problem” vs. “Issue” vs. “Bug”
The menu item in Unity is called “Report a Problem…”, the
program it launches is called “Report Bug”, the window is titled
“Report an Issue” and the text in the window refers to the “problem”
again.
I think that “issue” is the most appropriate word, and at the very
least, the menu item and the window title and contents should use this
word to refer to what could be a bug, feature request, or
documentation-related request.

Types of issues are not comprehensive, and may overlap
Most bugs I report have to do with API misbehaving. I tend to
report these as editor bugs, but that isn’t really accurate. There
should be an “engine problem” or “script problem”.
I think you’re supposed to file any crashing bug (player or
editor) as a “Crash Bug”. If this is the case, it should be made clear
by calling it “Crash (Player or Editor)” or something like that.

Reproducability is not always needed
This pop-up should only appear when it is relevant, i.e. not for
feature and documentation requests. At the very least, it should not
be required for these issue types.

The default text is not always relevant
I think the best way to inform users of the information needed
would be to have separate fields based on the type of issue. Each
field should have a title and a very short explanation of what should
go in it.

  • for bugs (like Apple’s format):
    — Summary
    — Steps to Reproduce
    — Expected Results
    — Actual Results
    — Regression
    — Notes
  • for feature requests:
    — Description of feature
    — Scenario where the feature would be useful to you
    — Notes
  • for documentation requests:
    — Description of requested documentation change or addition
    — Notes

Also, what happens when you click
“submit” is terrible:

  1. If there is an upload and you know how it’s doing, please provide a
    progress bar. Multi-gigabyte uploads have to happen every now and
    then, and it would be nice to know everything’s still Ok. Since the
    reporter takes a while to submit reports even without attachments,
    perhaps an explanation of what it’s doing, along with a progress bar,
    would be good.
  2. If there is a problem with the network, and the reporter can
    identify it, the current response is to tell you the submission failed
    and only provide a quit button! You can’t even select the text you
    entered before pushing it. Incredibly frustrating.
  3. The reporter doesn’t need to bounce more than once in the Dock when
    it’s finished. The fact that my bug has been submitted is great, but
    it barely warrants my attention.
3

First thing is to get rid of FogBugz, that thing is horrible, you can only reply in a back and forth fassion for a limited number of times before it stops showing any replies past a certain one, Unity is well aware of this fact and it has been a bug reported about the bug system now for over a year.

Aside from aformentioned format changes to the system, some real effort in communication as to the status of the report back to the users who reported the issue, some collaboritive system that takes that report, scans it for key words or common issue’s and drops it into a system that is browseable by all users via a web interface. This becomes more of a system that other users can browse to see if the same issue was seen or experienced by others and counts up how many times it has been reported. Anyone assigned to the bug could add comments to it for others to read.

When we submit bugs, after the submission, have the system open a web page to the bug report repository with a lookup for any error code reported by the user, think Microsoft and how they allow you to do a “report problem” which in turn directs you to known solutions if any or documentation about the problem directly.

The problem is, no one actually knows if Unity is serious about the problem, sure the problem is reported, it goes into a black hole of hundreds of bugs, and thats about it. As mentioned in other threads I could link to, Unity might or might not ever get around to looking at or resolving the bug, depending on internal resources and in the majority of cases, Unity will not communicate back to the users about the bug unless further information is needed, so to the users, this simply looks like Unity has a huge file 13 cabinet where these disappear into.

Unity is growing exponentially, that said, so are the bug reports and frankly put, Fogbugz is not the place or the technology to use. It was fine when you had less than 50 subscribers to your product, it is not useful on an enterprise level with hundreds - nay - thousands of users, possibly tens of thousands of users.

So taking that into consideration, if a system is not put into place that catches each of these reports and combines the like problems into one, you will end up in utter chaos real quick. The internal support staff is great, but I would summize that 75% of the issues can be taken care of with a working bug report system that is fluid and interactive, something more than what Unity currently has, something that has a better hands off for people to look up not only their issues but common issues or the odd issues reported by other people.

The system could be 80% automated also, which means that when the bug is reported, the system takes over, catagorizes the bug, puts it in a bucket, and gives the user a link to watch any progress, if any, on the bug. People are happier when they know an open communication channel is available and can ‘watch’ progress on issues. Right now, it is a simple view, here is how it looks to the community:

“Unity has crashed send in bug report?”
…User says yes…
“Thank you user for submitting the bug”
…User can see the submitted bug on FogBugz…
…The report goes from open status to closed…
…User thinks… was it fixed???..Sends in email or responds to bug asking what was fix…
“Sorry, we close things automatically after a while if you respond it gets re-opened, no ETA on resolution”
…User is more frustrated…

…Another 100 users come along and experience like problem, like issue, different situation, repeates steps above individually…very counter productive since all 100 users are unable to figure out if Unity is working on it, or if it is even a bug, but now Fogbugz has 101 users with a report that is similar in nature, and could have easily looked it up on a bug report site of Unity that shows a one line comment by Unity that says “No ETA on resolution”, which then avoids the additional 100 comments on the same problem…

Right now, the system is very convoluted and very non-user friendly. It is the system that is the problem, not the support staff. Unity has an entire staff of the best programmers on the planet, but the worst bug system on the planet, so a replacement of the bug system is really required at this point in the game.

4

What I hate, is that after the bug is reported, there is no way of knowing if you actually sent a file with it. I’d like to know that I didn’t screw that up, because I know I have forgotten in the past (usually when overwhelmed and sending multiple reports in a row).

It’s definitely a waste of everyone’s time to not know if your bug has been reported before. Any sort of search function would be nice.

5

I have to agree. Not being able to search for bugs and related incidents is the most frustrating thing about Unity’s bug reporting system. Not only would it ease our pain as developers, but it would have the side benefit of keeping Unity honest.

6

Another thing is - if you close it, say why.
Maybe it’s just a duplicate of some other bug, but you’re instead giving the message “we don’t care”.

7

What gets me is if when sending a detailed report thats taken time to write, if it doesnt get sent for some reason (eg if using dialup the line may have dropped) the “message cannot be sent” dialouge appears, and the written message vanishes into thin air. I’ve already donated time to write the bug report, no way in hell am I rewriting it again. This has happened a dozen or so times to me.

AC

8

Guys,

I like what I see… You are awesome!

Keep the feedback coming.

Cheers
Nico

9

It’s a small problem, but the fact that the bug reporter won’t resize baffles me. The “Problem details” box is pretty tiny for what we usually need to enter in it, so I usually have to actually type my report in some other text editor, then copy-paste it in there, which takes longer and makes me not want to report bugs.

Just make that dialog resizable and I’ll report bugs more often through it :wink:

10

Ya, this has to be the most annoying thing about the bug reporter. I often just end up sending an email instead of using the bug reporter because of this.

11

I say start a bug webpage the same as the feedback/requests site.
That way we can add bugs, people can vote on them and comment on them.
Unity people can tick if they’re in progress or completed.
Unity will know how much certain bugs mean to people and can prioritise.
Duplicate bug reports will be greatly reduced due to the predictive part that comes up as you are typing.
The feedback site allows catagories also.
The crash reporter can send you to the site.

Sounds ideal to me.

12

One thing that always trips me up is how the bug reporter automatically assigns the first words of your bug report as the title of the bug in FogBugz. It’s pretty rare that this actually makes a descriptive title, heh.

13

The biggest problem I have is the intransparency of bugs that got fixed and not. I often report bugs twice or more during different software test cycles only to find out that those are still open. Makes bug reporting not easier as you never know what got fixed and what not (except you get a direct feedback from a dev asking for more details and telling when it gets fixed).

About the bug reporter app: it is simply not at the standard of the rest of Unity software. It does for example not show any progress reports when uploading (only if you know to log in the system logs on Mac OS, but that is very user-unfriendly). The reporter is not able to recover a suspended session (like from a network shortage or whatever reason).

It also does not save the text you enter for a later use, like the AS commit dialog inside Unity. It shall also have the ability to just drag and drop files on it like screenshots, movies etc, that would make life easier as I usually don’t have the screenshots of the bug or the movie stored inside the project folder which I then upload too.

Further it would be nice to have an option to exclude folders and files from being uploaded to the server. I often save a lot of extra stuff in the project folder like screenshots, concept art and whatsnot and I usually need first to clone the project folder, remove that art stuff, zip it, upload it. Lots of extra work that could be avoided.

14

I think my suggestion a few posts up would help most of your problems (“I say start a bug webpage”…)

15

I would second and third all of the above suggestions and offer another. I have used Unity\Unity iPhone Pro (for exployer) and basic editions at home that I own personally for over 2 years. I continually have to fight with and guess at what portions of Mono .NET that you guys support. You up until recently claimed Mono .NET 1.1 support and more recently 2.1 support while fully supporting neither at any point in Unity’s history.

I would suggest that the best way to report bugs is in NOT having to. As QA for Unity I feel it is your job to fully realize what portions of the Mono .NET framework you support on which Unity setup and not mine. It would take a couple of days on your end to fully compile a list of what you support and what you don’t on each specific Unity configuration and then make this list available to us as developers.

Edit: I would imagine the number of reported bugs would drop if this information were available. Feature support or lack of support in .Net framework isn’t a “bug” it is a “feature” or lack there of so they shouldn’t be in the same pool on your end. It should just be documented clearly up front.

16

Just don’t collect bugs for documentation.

Allow to edit the documentation in a collaborative way. Perhaps a wiki? You will spend your time rather moderating than fixing the docs yourselves.

17

I agree completely with Martin Schultz’s post.
Is neccessary an option to exclude folders and files from being uploaded to the server because the process now is long and slow.

18

After considerable experience reporting bugs, let me add one thing:

Give the bug reporter tool the “package select” ability. My projects are often too large to upload (2 GB+) in whole, but the problem can be shown in one scene which is small. If I export a package, support tells me to upload the project. Which I can’t (would take several hours on my ADSL).

If I could tell the bug reporter which SCENE is buggy, and it could upload that scene and included assets only, it would help a lot.

19

How is that any different than exporting a package? You just select the scene and hit Export Package. All the required assets are included by default, for just that scene. I do this all the time and support seems fine with it.

20

Being able to add comments/files to a bug would be helpful. There are times when I don’t have any clue how to repro a bug( like a crash bug ), but after investigation, I narrow down the issue. Right now, I’m pretty much forced to file a duplicate bug report once I have more information, which wastes Unity QA’s people time( since they will most likely get the bug with less information first ), and wastes my time.

It would also be helpful if Unity fixed bugs before adding new features. There are unfixed bugs, like hideflags.hideinhierarchy, which have been sitting around for at least six months now. Although I realize that not all bugs can be fixed in time for each major release, that’s not an excuse to not fix them ever.