Why we need to Radically Change Bugzilla, Part 2

First, let me clarify a few things. By “End-User Bugzilla” and “Developer Bugzilla” I didn’t mean two totally separate Bugzilla installs. While that could be made to work, there are much easier ways to go about it that are much better. I will continue to call it “End-User Bugzilla” for lack of a better term however. The End-user Bugzilla could take the form of a product on BMO, a separate database on another website, a whole range of ideas, I’ll lay those out in a future post.

Current Bugzilla

While slightly out of date, this chart does give the basic idea of how every bug in Bugzilla goes through life. Developer bugs miss the UNCO state usually, but all end-user bugs follow this pretty much from beginning to end. In theory. In reality, end-user bugs sit in one of two states for years. UNCO and NEW. Very few become ASSIGNED or FIXED. The majority end up becoming INVALID, DUPLICATE or INCOMPLETE. Reducing that number, and raising the number of FIXED bugs is the challenge here.

Proposed Bug Process

For Developers, the bug process remains largely the same, a bug is reported, assigned (or resolved WONTFIX, etc.), then FIXED. For End-Users, the bug is put into a staging area. This staging area is where triage, QA and Support can all work together to improve bug reporting for out end-users. A user reports an issue (ticket, bug, whatever we want to call it). Triage jumps on it, ensures that the issue has all the information it needs, filters out spam, and if it is ready, sends it off to the proper “department”. Support questions go to SUMO, Extension bugs are tied into AMO somehow (feedback on an extension perhaps?), and legitimate bugs are sent to BMO, where they are then put into the proper product and component (this is all a one-step process, much like changing products on BMO today). Issues where the reporter never gets back are marked INCO within a certain period of time.

Why this is Better

For Developers, the excess wading through bugs submitted by end-users that may or may not be real bugs is avoided. BMO is devoted to only real bugs, and triage doesn’t have to worry about tripping over a developer and visa versa.

For End-users, this ensure more personal attention. Instead of having their bug ignored, marked INVALID (how uncaring does INVALID sound? It is a valid issue to them, or they wouldn’t have raised it) or some other confusing resolution. Instead of simply being brushed off (which, no matter how we mean it, it appears that way to the general public), users will now get their bug moved straight into the proper place. If they need support, they are happy because boom, there it is. If they did find a bug, it can be an entry into more bug reporting in the future because we move it straight into BMO. If it is another issue like an extension, then the information can be moved where it needs to be. Perhaps even feedback can be taken, then submitted to Hendrix as higher quality information.

A Clearing House

So in essence I envision the “End-User Bugzilla” as a central place for input and feedback from users. Taking the loose ends and pieces of Mozilla’s carious portals, and tying them all into an easy place for end-users to get help, report issues, and make a difference.

Advertisements
    • Boris
    • May 17th, 2011

    There are some failure possible modes here (e.g. bugs moved to evang or marked invalid without the relevant module owner/peer being involved). Or is the plan to not do that in the staging area?

    • The staging are should not be for INVALID or WONTFIX, etc. INVALID would be reserved for spam bugs, obvious junk, etc. The main reason we use the INVALID state right now as traigers is either an extension bug, or a support request. Support requests would be moved right over to SUMO, and hopefully some way could be through up of helping with the extension bugs too, perhaps tying them into AMO somehow.

  1. How clear-cut is the boundary between a “developer” (all whose bugs would start in the “developer” bugzilla) and a “user” (all whose bugs would have to go through a preliminary “user” bugzilla)? Not very, IMHO, and for this reason this proposal seems to me to be a step backward. I’ve had EDITBUGS privileges for some months, and I’ve had some success at triageing bugs, but does that mean that none of my recent and future bugs will ever be resolved INVALID or INCOMPLETE? Not by far, and with good reason. So, should my privileges be revoked? Maybe, but then the SeaMonkey community will have to find itself someone else to do the QA work I’ve been doing.

    • Tony, you are right, at this time, the difference is not clear. I don’t plan on changing much on how BMO itself is running for those who have CANCONFIRM and higher. The staging area will be for those who do not have any privs, first time bug reporters. They will be directed to the staging area instead of BMO. Then triagers will move their bugs over to BMO. People who have submitted legitimate bugs could then apply for privs on BMO, just like now.

      • What about people who have privs but *want* a particular bug to be run through the staging area? For instance, I pretty regularly have bugs that I don’t know which component to file them in.

      • If you want to use the staging area you can. It is like the guided and advanced bug forms on BMO today. Beginners are given the guided form by default, but you can always switch back and forth.

      • I mean, I can goof just as much as the rawest beginner. Time and again (and even now that I have some privs) I’ve filed some bugs honestly thinking that they were real legit bugs, then later after some closer investigation they were resolved INVALID, sometimes even by me, once it appeared that some bug was “not a Mozilla bug” or that the incriminated “actual behaviour” was actually the “intended behaviour”. So I don’t think it’s a good idea to say that bugs can move from the staging area to BMO but not vice-versa, and that INVALID and INCOMPLETE resolutions won’t exist at all on BMO anymore.

      • Tony, I’m saying most of these things from a triager standpoint. The INVALID resolution would not go away, but for triagers it would not be needed. Developers would be using it just like they currently do. I want to have very little change in BMO for Developers in terms of workflow. Perhaps a little simplified, but not much. And yes, the Staging area would be a two way street.

    • LpSolit
    • May 18th, 2011

    IMO, triagers can already do the same job with bugs reported to bmo and marked as UNCONFIRMED. And developers probably already ignore such bugs. When a bug is confirmed and is not a duplicate, triagers just mark it as NEW. I don’t really understand where the benefit is.

    • No, I’m sorry, but UNCO is not working. If UNCO worked, why did we have more than 12,000 bugs last years ranging from ten years ago? We simply do not have the man power to deal with them in BMO. Developers and triagers are always tripping over each others feet. Certain module owners and peers have different ways they want their bugs triaged, and if you don’t abide by the rules you get scolded. It is not working, we need something better.

        • Michael
        • May 18th, 2011

        I think the point is that this might just be moving things around. How will module owners and peers know that “their” bugs in the “user” system are being handled according to their rules? I guess the answer is that they won’t, which makes it seem that your solution to developers and triagers tripping over each other is to shut developers out of some of the triage process.

        I haven’t actively done QA in bugzilla for some time, but I don’t think bugs that are obviously support requests or obviously junk/spam are the majority of the ones that stay as UNCO for ages.

        The ones that stay as UNCO for ages are enhancement requests or reports of problems in obscure cases where there is little chance of reproducing the problem. As I understand it, some developers want those reports left around until they can be properly assessed. If your system is going to pass those through, then there will still be lots of UNCOs. If it isn’t, then it’s still going against the wishes of those developers that are scolding about that happening in bugzilla…

    • Dan
    • May 18th, 2011

    I like that this is being thought about, but this widens the “us and them” mentality I think, which is bad for the community.

    • I think this reduces the us vs them mentality. We are taking user reports more seriously now. We are simply separating them out making it easier to work with. The idea is to give higher quality of service to end users and make it easier for triagers and developers.

    • I agree with Tyler. To a user who isn’t yet part of the development community, the existing very developer-centric Bugzilla sends the message that we don’t want to hear from them.

      • Most of the time I haven’t had this impression, even when I was still a beginning user of BMO; yet there’s one case of “we don’t want to hear from you” that I’ll remember forever: one bug that I had reported about Firefox 0.9 or 1.0 was duped to a Suite bug; later, when I commented about Firefox on that Suite bug, I got a harsh reply amounting more or less to: “Go away, this is a Suite bug; comments about Firefox are not welcome” (that was at a time when the Suite was still the star Mozilla product, and Firefox was only its little brother). If that happened again nowadays (probably in the opposite direction), I would immediately REOPEN the original bug, but at that time I was too conscious of my “low rank”, and I didn’t dare. This case, however, was more the exception than the rule.

      • I think that there are two responses and one non-response that en-users typically hear.
        1. Submit a bug with useful information, a developer happens to stumble upon it, goes “Darn, that is a real bug” and fixes it. User is happy, developer is happy, software is happy.
        2. Submit a bug with no useful information, triager or developer says “that isn’t a real bug” closes RESOLVED>INVALID(INCO, WFM, etc). User is confused, the issue didn’t go away for them so it is definitely not RESOLVED, and obviously it is a valid issue for them or they wouldn’t have filed a bug. User is unhappy and confused, triager continues on their way, software stays the same.

        The non-response is just that, user submits a bug, nobody ever looks at it. Then three years later, I comment asking if the issue can be reproduced, user has since changed email addresses, bug gets resolved INCOMPLETE.

        The goal of the staging area is to totally eliminate the non-response as much as possible (of course, there are people who simply won’t reply, no matter what we do), and to make the #2 more friendly. Instead of INVALID, moving the bug to support. End-result, happy user, they got the help they wanted without excess effort on their part. Also, this plan should make the occurrence of #1 more common.

    • Slavic
    • May 18th, 2011

    I still haven’t understood one important thing. Let’s look at this diagram from the point of common user (such user may be very advanced, but not one of Mozilla developers anyway). User has filed a bugreport and eventually managed to convince QA and Support that it’s a real bug, reproducible in some way in some configurations and circumstances. OK, bug is accepted as legitimate and sent to developers’ Bugzilla. Question: has the user an access to a bug after this point or he completely lose an option to trace it, look at discussion and comment if need? It’s not unusual when developers want to get some additional info, ask for more test results, like in different environment, for reproducibility etc. Also the developers sometimes count a bug fixed without carrying out the sufficient amount of tests, while the stubborn user can check the apparently fixed bug, which he is interesting in, more carefully, right?

    • No, Users will not be locked out of bugzilla, and any bug that is moved to BMO would still be open and accessible to the user. Bugzilla is not going to be changed in anyway, the staging area is envisioned as a database to work alongside BMO, not replacing it.

    • Will
    • May 18th, 2011

    I am a seasoned corporate admin who has done the occasional bug report on Bugzilla, because I do 2nd level support for your product and other browsers. Please do not confine guys like us to some sort of toy Bugzilla, like some of your competitors do. Following along bugs on Bugzilla, I have gained great confidence and even trust in the professional way you deal with problems. Please don’t take that away from the “public”. It would be a shame.

  2. Hey Tyler. So, you’re not misguided here. You’re heading sort of in the right direction, but ultimately you’re ending up putting lipstick on a pig. In order to accomplish your very valid goal (effective bug-tracking and rapid response times), ultimately the fact will be “In order to accomplish this goal, we need X people to do triage.” Mozilla does not and has never had X people. It has never even had 1/2 of X people. I’d say most of the time, there have been perhaps 1/20th of the required resources allotted to triage.

    So, when you ask “Why are there so many UNCO bugs?” the answer is, “Because nobody took the time to triage them.” You could put up another tool in front of Bugzilla, but you’d still have no triggers. And to make it worse, developers wouldn’t even look at the staging system, so you’d lose what triage effort developers contribute now.

    There is often a modest at Mozilla and other places that somehow tools will magically solve every problem. It’s not entirely untrue–improvements in tools can go a long way to making a situation better. For example, I think auto dup detection is going to lighten the triage load once the new guided form is active.

    But ultimately, it’s called a “tool” because it’s something a human uses to accomplish a task. If there’s nobody wielding the hammer, the nail isn’t going to pound itself in. If there aren’t enough actual human beings doing triage, you will never ever solve this problem with any amount of tooling. I have seen numerous systems in numerous places, and some do really help out the triagers, but the effectiveness of such systems *always* boils down to how much manpower is actually behind them.

    -Max

    • Autocorrect errors:
      “triggers” should be “triagers”
      “modest” should be “mindset

    • I’ve been banging on this drum a bit myself. In the context of making the tools better, it occurs to me that maybe we should start with more manpower. Hiring people would be ideal, but for an initial experiment we might be able to get away with “bugday” type community recruiting. Turn these people loose on the UNCO bugs, the :General bugs, and the really old NEW bugs (in that order, I think) and then debrief them extensively. That’d give us a better picture of what the tools are actually missing.

      I have the impression that Tyler started this conversation based on personal experience trawling through UNCO, so he’s got some idea of what would make the job easier, but more people’s experiences would only help.

      • Zack, i agree than more manpower is what we need. However, Bugdays have historically been flops. I’ve organized several of them in the past, and typically attendance is rather low. I find more triage gets done on non-bugday days than the days with bug days. While this all depends on the motivation of those working in the bugday. Maybe some way of making them more organized and tied into real work instead of just “close bugs older than x days”.

        I think what is desperately needed are 3-4 triage people, hired by Mozilla, who’s job is to do triage. That is it. They would be able to organize and help the community triagers, and make more focused efforts on triaging more aggressively.

        • Will
        • May 22nd, 2011

        I would be willing to help out with bug confirmation / triage. Is my Bugzilla account sufficient, or do I have to have some official Mozilla function? Is a stock Nightly Build sufficient, or do I have to roll my own?

  3. Max Kanat-Alexander :
    So, when you ask “Why are there so many UNCO bugs?” the answer is, “Because nobody took the time to triage them.” You could put up another tool in front of Bugzilla, but you’d still have no [triagers]. […]
    -Max

    You can say that again, and the same applies to the “General” component of most Products. Even if the current one (which isn’t yet finished) is leaving me practically exhausted, I feel that there should be more “events” where people would be encouraged to come together, triage bugs from “General” to wherever they belong, attempt to verify UNCO bugs, ask for more information if the existing bug report is unclear, etc. etc. etc.

  4. Will :
    I would be willing to help out with bug confirmation / triage. Is my Bugzilla account sufficient, or do I have to have some official Mozilla function? Is a stock Nightly Build sufficient, or do I have to roll my own?

    A stock nightly, if built by Mozilla, is better than an own-compile or a build compiled by some Linux distribution, because there are no uncertainties in how it was made.

    A Bugzilla account is necessary; apart from that, is is better (though not absolutely indispensable) if you have a few permissions set on your Bugzilla account: CANCONFIRM to confirm bugs, or EDITBUGS to triage them. There used to be an online document about how to get those permissions but I cannot find it back. IIRC the general idea is: (1) collect proof that you deserve them, in the form of URLs of legitimate bugs which you’ve reported in the past, and/or of useful comments that you’ve written; and (2) send these to Gervase Markham in an email with a permissions request. Maybe some other commenter can give you better details.

  1. August 27th, 2011
  2. September 7th, 2011

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: