What to do, with Old NEW Bugs?

Well, I knew that sometime this would have to be tackled. After closing another batch of ~1100 old UNCO bugs the other night, I decided to turn to our other forgotten bug problem, the NEW bugs. First, a bit of backstory.

NEW has been almost like the second trial of a bug. If a bug makes it through the graveyard that is Firefox General UNCO, it either is Assigned to a developer, and is fixed relatively quickly, or, it gets thrown into the NEW pile. These are bugs that are a legitimate problem (or at least, were bugs when marked NEW). However, the developers either need more information about the bug, or are waiting for time to fix them. Unfortunately, because we have so much to do and so little time, many of these bugs end up being forgotten. Almost 2000 in fact. That is almost 2000 NEW bugs in Firefox (not including enhancement requests) that have not been changed in over 500 days. That’s well over a year.

Many of these bugs may still be valid issues. Some could be meta bugs tracking other issues that are being fixed. Some might be still issues. The question is, how many of those bugs are actually fixed in recent releases of Firefox, and could legitimately be closed?

Would it be worth the time putting a comment on all very old Firefox General bugs, asking for the reporter or reproducer to retest, and then update the bug if it is still an issue? Then, those bugs which do not receive a reply, in 3 months or so, I can go back, and mark them as CLOSEME for a date a month down the road. Then if they have still not been commented on, they can be closed as INCO.

This will give three reminders to the reporter to comment on the bug. The initial request for retesting, the second marking as CLOSEME, then the actual closing. We would filter out as many known legitimate bugs as possible (bugs blocking other bugs, bugs being blocked, meta bugs, bugs that are in components that peers ask not to be touched, etc).

It is just my feeling that this backlog of NEW bugs is hurting triage as much as the UNCO bugs were. It also hurts QA by giving them too much bug noise, and developers have a hard time determining if a new bug really is legitimate anymore.

Comments?

Advertisements
  1. Yeah, I agree with you: this backlog of NEW bugs is hurting triage as much as the UNCO bugs were. 😦

    However, I think we should wait Firefox 4 final before polling all these reporters.

    • Yes, this probably won’t happen until 4 is released. After 4 is released I’m also planning something similar on the 3.x UNCO bugs as well.

    • Archaeopteryx
    • February 4th, 2011

    If a user files a bug and it gets confirmed (-> new) and contains steps to reproduce, requesting to check if it’s still a problem without any indication that the problem has gone is mostly pissing people off. Especially if they again don’t get fixed after the retest.

    • I agree that it may offend some users. However, the benefits of clearing out the NEW backlog is way better than having the bugs sit there and never be touched. If we don’t ask for retests now, they are never going to be done. That is why I suggest the longer process for NEW bugs than UNCO, just to give reporters more time.

      • I feel _very strongly_ that Mozilla personnel should attempt to reproduce these bugs _before_ asking the reporter anything. It may well be that there is already enough information to determine that the bug still exists, or has already been fixed. If there genuinely is not enough information in the bug to move forward, by all means ping, but bothering people when we already know everything we need to know is just rude.

        http://www.jwz.org/doc/cadt.html

      • That would be the ideal case. I would love to be able to personally test and try to reproduce every bug on Bugzilla, and I know that our other volunteers would want the same. However, with 6400 NEW bugs and 6700 UNCO bugs that all need to be reproduced, and perhaps two dozen (on a good day) community members who do triage on ALL of Bugzilla, that is simply impossible. There is a fine line between updating a bug and bothering people. That is why I suggest bugs older than 500 days. That’s 500 days without anything on the bug changing. Those bugs are the most likely candidates for being closed as WFM, simply because other bugs have been filed since then
        While I agree we don’t want to bug people, simply letting Bugzilla rot for the sake of not bothering a bug reporter is frankly what has let is get to the terrible state it is today.

        • Ian Thomas (thelem)
        • February 5th, 2011

        Zack, I don’t think JWZ is complaining about the bug being closed, he’s complaining that nobody could be bothered / find the time to fix it.

        If we’re worried about pissing people off with auto close, then we could just request that the reporter re-tests the bug in the latest firefox.

        “Firefox 4 has recently been released and may include a fix for this bug. In order to help us prioritise the remaining bugs please re-test this in Firefox 4 and let us know whether or not it is still an issue.”

        Lets say half the reporter respond to the message, and half of them report that it is still an issue. You’ve now got 1,000 bugs that are candidates for a harsher auto-close message, 500 bugs that have been closed and 500 bugs that require further attention (much better than the list of 2000 that you had before)

    • Anonymous
    • February 4th, 2011

    Poked at the list for a while; it has a few dozen less now.

    • guanxi
    • February 4th, 2011

    Would it be worth the time putting a comment on all very old Firefox General bugs, asking for the reporter or reproducer to retest, and then update the bug if it is still an issue?

    While I understand the instinct to clean this up and appreciate the effort — it takes guts to stare down Bugzilla! — I think you are underestimating how much unhappiness this could cause. and I want to give you another perspective:

    From the point of view of someone who has volunteered for Mozilla projects, it sends the message that Mozilla does not value or have worthwhile use for people’s time; re-triaging bugs that will never be fixed is not the best use of my day. Also it rubs reporters’ faces in the fact that Mozilla didn’t act on their efforts in the first place (likely with good reason, but nevertheless …) — now they’re asking you to repeat it so they can ignore it again! It wouldn’t give people warm, fuzzy feelings.

    From a productivity standpoint, consider that this solution assigns limited resources to the organization’s least productive assets (i.e., old, untouched bugs). If people spend an average of 15 minutes on each bug, that’s 500 person-hours or a quarter of a year. I would call the bugs “Dormant” or something similar, and if a developer wants to work on the bug, they can request reconfirmation at that point.

    Good luck cleaning up Bugzilla. It’s a herculean task.

    guanxi

    • You highlight one of the biggest issues with Triage, productivity.
      To a Triager, spending an hour to triage 100 bugs, and 98 of those bugs getting closed as INVALID, WFM, INCO, or DUPE, and the other two getting FIXED, that is a productive time. To a developer, 98% of the time spent was unproductive.
      This back log of old bugs is slowing Bugzilla down and hindering it’s usefulness. Having to wade through thousands of bugs that may or may not be valid is not good for anyone. Triage, QA, development, etc. Yes it will hurt. I never said it wouldn’t. And no, Mozilla failed and still fails with Bugzilla. However, we need to accept that some people won’t be happy with it, but the greater good to BMO is worth the pain and time.

        • Boris
        • February 4th, 2011

        Any developer who thinks 98% of the time was spent unproductively in that example is dumb. 😉

        That said, if a bug has a testcase attached, it’s worth taking a few seconds to see if the problem appears. If that can’t be determined from the testcase in a few seconds, then asking the reporter is totally reasonable.

        Perhaps do a first pass of bugs with attachments treating them that way, and then a separate pass as you suggest for bugs without attachments?

      • Yes Boris I agree, however, that seems to be the typical outlook of many developers in Mozilla. Because Triage does not produce results. It is not easily quantified. It can’t be done in “Q1 2011”. It’s a long arduous process, so unfortunately the feeling among some Triagers is Mozilla is taking to ignoring it.

        I agree, bugs with testcases should not be treated in the same way as other bugs. I’ll craft any query that I do to exclude bugs that have testcases, WIP patches, etc.

    • James Napolitano
    • February 5th, 2011

    I remember there being talk of adding a READY status for bugs that had not only been confirmed (like NEW) but also had clear steps to reproduce, had been checked for dupes, had testcases, etc., so they were ready for developers to start working on them. Is that still something being considered? Are there any new keywords like “needs_testcase” or “not_a_dupe” to mark what needs to be done or has already been done with a bug report, so triagers don’t need to keep spending time reexamining the same bug? Also, can anything be done about the huge inflow of (often invalid) bug reports from users? I often thought it counterproductive that new users are often invited to submit bug reports and so spend their time doing something that actually ends up costing the project resources (in terms of time spent triaging). Maybe some better warnings or info on the problem could be adding to b.m.o.?

    • Yes, there has been talk of multi-state triage flags that would significantly help, however those died off. There have been suggestions about changing the bugstatus types, however I’m not sure what ever came of that. I’ll be posting about that soon.

        • njn
        • February 5th, 2011

        Please, don’t add more statuses, Bugzilla doesn’t need that and it won’t help anything.

        From a reporter’s point of view, the status means almost nothing. It’s the responses you get from devs that matter. A bug marked UNCONFIRMED with lots of talking is much better than a bug marked ASSIGNED that sits there untouched.

        From a dev’s point of view, the status doesn’t mean much either. (The only thing that *really* matters is blocker flags.)

        Really, status is only useful to people who are looking at heaps of bugs at once, ie. triagers and managers. So the statuses should reflect the searches they need.

      • And Triagers and Managers don’t deserve to have their work made easier? I don’t think that Status changes are the best idea, but I think that taking away NEW, and splitting it into two statues is a good idea. What is better is the multistate Triage flag, which has been suggested before and deserves to be brought up again.

        • Boris
        • February 5th, 2011

        njn, blocking flags mean something to full-time developers right before a release.

        A “this is ready to work on” status is really useful to everyone else, and in particular to people looking to get involved with the project…

      • The thing that will make or break this status, is will it be used. If we create two statuses, there has to be consensus among developers and triagers that they will be used. Or else we end up in the same pickle we are in now.

    • We already tried splitting up the NEW state. It became UNCO and NEW. And now here we are.

      I don’t see any reason to believe that splitting NEW again would have different results this time.

      Furthermore, the line between NEW and READY is even more blurry than the line between UNCO and NEW. So it will be more cognitive load for less gain.

    • B.J. Herbison
    • February 5th, 2011

    “I’ll craft any query that I do to exclude bugs that have testcases, WIP patches, etc.”

    Bugs with WIP patches should probably be assigned to the person who created the patch. Could you scan for that case (and maybe other cases) and kick the bugs out of the “NEW” state?

    • That could definitely be done. However, not all people who have created WIP patches want to be assigned the bug, some may have purposely unassigned themselves because they do not have time, are no longer active, etc. So just reassigning those bugs back to them is not a good idea.

  2. Please mention the reporter’s name whenever you ask for a bug to be retested. I get several bugmails a day where a comment starts with “Reporter,” and I don’t have time to figure out which ones are bugs I reported. (I reported 1265 open bugs and I’m cced on 2358 open bugs.)

    • Jesse,I would love to put a name on each one, but, when running through a list of 1,000 bugs that’s a little impractical.

  3. Ultimately, until Mozilla organizes triage such that bugs can be triaged at a rate faster than they are filed, there is no system that will really allow effective response and triage on every bug that gets filed.

    The Automatic Duplicate Detection system in Bugzilla 4.0 should help cut down on dups, at least, which could possibly make this easier.

    Also, the experience of triagers should be gathered and made into a Bugzilla Extension that helps automatically detect bugs that are likely INVALID (for example, ones with “Google Toolbar” in the summary) and directs people to the correct channels for getting help with their problem, instead of having them continue filing a bug.

    -Max

  1. February 6th, 2011
  2. March 5th, 2011
  3. September 7th, 2011

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: