Why We Close Bugs

It seems that my recent post on requesting retest on NEW bugs showed a divide in the community. There are certain community members who agree that old bugs should be pinged, while others think that it is an annoyance rather than a help. And I agree it is a fine line that needs to be walked. There are pros and cons, and I think we as a community need to determine where to draw the line. Closing bugs as INCOMPLETE hurts, but it helps too.

Cons of closing bugs:

  • It can be annoying. Getting thousands of bugspam emails at once is pretty annoying (I know, I get them too). And if your bug has slipped into a query it can be annoying to be asked if you’ve reproduced a bug in a newer version of Firefox when you already have. It happens.
  • It can appear that Mozilla does not care about users and their Bugs. When a user files a bug in 2006, and it never gets touched until a comment is placed on it in 2011 asking the user to test again in the most recent Firefox version, reporters can understandably feel unappreciated.
  • Real bugs can be closed in the process. In closing bugs, we run a slight chance of accidentally closing a bug that is an edge case and has not had any dupes filed.

Pros of closing bugs:

  • It assists with finding bugs that will actually be fixed. If a developer has a list of 1000 bugs covering multiple versions of Firefox, it is a bit hard to find a bug with enough information to fix. However, if Triage has gone through that list, weeded out the ones that do not have active reporters anymore and closed them, that list could potentially go down to 300 bugs. All 300 with recent comments from their reporters or other users who have reproduced the bug, giving recent relevant information. A list of 300 up-to-date bugs is much better than 1000 of varying quality.
  • It enlists the help of reporters to reproduce bugs. There are simply too many bugs for the Triage team and QA team to reproduce all of them at once. By asking reporters to reproduce, we can help enlist our community more.
  • We have nothing to lose. These bugs are sitting there stagnate anyway. Nothing will change if we don’t go after them. If we do comment on the bugs, the chance that we will get valuable information is significantly higher than if they just sat. We have nothing to lose, and plenty to gain.

Now, there are some ways we can improve:

  • Write better follow up comments. Possibly, something like David Eaves suggested in his blog post. Prettified emails. Perhaps Bugzilla could be modified to send an mmail saying “Since you reported this bug, two new versions of Firefox have been released, with many changes in x y z. You can read the release notes of these updates at mozilla.com. The Mozilla community would appreciate it if you would take the time to retest your bug XXXXXX with the latest version using a fresh profile. And update your bug accordingly.” Or something like. I’m not sure how feasible this is, but we definitely need to make our bug replies more friendly.
  • Be better at following up ourselves. I mean come on, we ask the reporter to follow up on their bug, then we as Mozilla let the bug sit, again. It is disheartening to a reporter to be like “They asked me to follow up, maybe they are finally going to fix the darn thing.” Then it sits, again, for years because as a community we dropped the ball. I think that is what is more of a discouragement to our community of bug reporters. That they report a bug that never gets fixed. They spent time that they didn’t have too creating a bugzilla account, reporting a bug, retesting a bug (perhaps multiple times) then it sits. We really really need to pick up the game here. And start to focus on our community a little more.

A Picture is Worth a Thousand Words

Thanks to a fellow community member for pointing it out. Graph is here. (Click Image for full size)

What to do, with Old NEW Bugs?

Well, I knew that sometime this would have to be tackled. After closing another batch of ~1100 old UNCO bugs the other night, I decided to turn to our other forgotten bug problem, the NEW bugs. First, a bit of backstory.

NEW has been almost like the second trial of a bug. If a bug makes it through the graveyard that is Firefox General UNCO, it either is Assigned to a developer, and is fixed relatively quickly, or, it gets thrown into the NEW pile. These are bugs that are a legitimate problem (or at least, were bugs when marked NEW). However, the developers either need more information about the bug, or are waiting for time to fix them. Unfortunately, because we have so much to do and so little time, many of these bugs end up being forgotten. Almost 2000 in fact. That is almost 2000 NEW bugs in Firefox (not including enhancement requests) that have not been changed in over 500 days. That’s well over a year.

Many of these bugs may still be valid issues. Some could be meta bugs tracking other issues that are being fixed. Some might be still issues. The question is, how many of those bugs are actually fixed in recent releases of Firefox, and could legitimately be closed?

Would it be worth the time putting a comment on all very old Firefox General bugs, asking for the reporter or reproducer to retest, and then update the bug if it is still an issue? Then, those bugs which do not receive a reply, in 3 months or so, I can go back, and mark them as CLOSEME for a date a month down the road. Then if they have still not been commented on, they can be closed as INCO.

This will give three reminders to the reporter to comment on the bug. The initial request for retesting, the second marking as CLOSEME, then the actual closing. We would filter out as many known legitimate bugs as possible (bugs blocking other bugs, bugs being blocked, meta bugs, bugs that are in components that peers ask not to be touched, etc).

It is just my feeling that this backlog of NEW bugs is hurting triage as much as the UNCO bugs were. It also hurts QA by giving them too much bug noise, and developers have a hard time determining if a new bug really is legitimate anymore.

Comments?

My Triage Process

Well, I’m back. The puncture wounds have healed quite a bit (though I shall have more scars to add to my collection of accident related injuries). The Radial nerve got injured so I still have no feeling on my left thumb, and occasional shooting pain, but that should go away within the next few weeks. So, I’m back to blogging, and bug triage.

When I first started triage almost 3 years ago, I didn’t really have a process for triage. It was usually to +start with a list of all the Bugs in a certain component, and just start at the top and work my way down. This is unfortunately one of the only ways we can triage with so many UNCO bugs. trying to run queries either pulls up very small and limited results, or pages that are hundreds or thousands of bugs long. The situation is improving though.

When I triage, I have several methods that I use. The most usual is running through a query that returns all the bugs filed in the past 24 hours. It returns all bugs in Firefox, Toolkit and Core from the past 24 hours that are UNCO, NEW, ASSIGNED and RESOLVED. (link to come soon, BMO is taking forever to run queries ATM for some reason). I will then browse the whole list (usually around 100-150 bugs a day).

I first start with what you could call “low hanging fruit”. I go through and find all bugs that look like they will be support issues, extension issues (ones that have words like “Google Toolbar” in the summary are usually a good start).  I will go through, and try to resolve those bugs first to cut down on bug noise. Extension issues usually end up getting reported to the developer of the problem extension, and support bugs get directed to SUMO.

Then I begin to go through the other bugs. It is not possible to look at or touch every bug. I am not an expert in every component of the products I triage (not an expert in any I have to admit), and some bugs are simply above my head. Other bugs have already been touched by some of our other excellent triagers. However I try to at least look at every UNCO bug submitted in any given day. I usually visit this query several times a day.

I then start to plunge into the depths of BMO. I will either go through and mark untouched bugs as CLOSEME (I usually focus on that more at the beginning of the month), or I will start to run specific queries I want to focus on. For example, some months I will run queries in Firefox looking for bugs in printing (which is often improperly filed in Firefox General instead of the Core Components). Other days I will look for specific extensions or common issues I have seen trending recently and will go and search for more dupes.

What is your triage process? Any ideas for ways I can change or improve? Most of triage is a personal style. There is no cut-and-dried “perfect” technique to Triage. Different people work better different ways. It is all about how you work that makes our Triage community unique and vibrant. Our differences make Mozilla what it is today.

I’m trying to toss around some ideas on improving specific Triage, I’ll be posting soon on some thoughts for improving communication between developers and Triagers, something that is sorely lacking at the moment.

Short Hiatus from Blogging

Well Saturday night I had a most unfortunate accident. A friend’s dog that I was watching decided to bite me on both wrists. This has curtailed my ability to type for extended periods of time and made me lose feeling in my left thumb. So I’m going to have to delay my planned posts on Triage, continuing from the previous few. However, I would greatly appreciate any reader feedback and suggestions. Maybe you’ll have a few ideas that I can throw into a blog post 😉 I hope to be back in a week or two, depending on how my appointment with the hand specialist goes.

Extension Bugs

One common source of bugs that are resolved as “INVALID” on Bugzilla are bugs caused by third-party extensions (Google Toolbar for example). Currently in Bugzilla, most Triagers will move a bug that has been determined to be caused by an extension  to the “Extension Compatibility” component of Firefox, close the bug as RESOLVED>INVALID, and also mark in the whiteboard the offending extension (If known). Sometimes these bugs will morph into blacklist bugs, and there are cases where addons caused legitimate bugs in Firefox that needed to be addressed by Mozilla Developers (see Bug 327859 for a bug in Firefox that Adblock Plus exposed).

However, most extension bugs are bugs in the extensions themselves, not Firefox. There is little our developers can do to fix the issues. It is up to the extension developers themselves to fix the issues. Normally after closing a bug, the triager will ask the reporter to report the issue directly to the developer of the extension. This does rather unfairly place a burden on the original reporter. The majority gladly goes and reports the issue. However, extension reporters can be hard to find and get a hold of. AMO doesn’t necessarily do a great job of this. (Comments aren’t the place to report bugs, and not all developers have the resources or ability to have a bug tracker).

In trying to reach a solution for this, we run into scalability issues. There are over 6,000 addons for Firefox, and each of these has the potential to have bugs. I’ve considered having a specific Extension Product created (similar to the Pugins component on BMO). This approach works for Plugins because there are only 20-30 on BMO. There is no way we could list all 6,000 addons and keep them updated. So I’ve thought of two ways to approach the issue, one much more simple than the other, but not as efficient.

  1. Create an Extension Product in BMO with one component. Move all extension bugs into that product, and mark in the whiteboard and/or Summary what the offending extension is. Then, hope extension developers check for their extension in the list of bugs. This has the benefit of being very simple to implement, but the Extension product then runs the risk of becoming a bug graveyard, similar to the current Firefox:General Component.
  2. This one is more complex and would require more work on the part of Bugzilla and especially AMO developers. However, we already offer plenty of resources to Addon developers on AMO. Why can’t we offer a very simple bug tracker for them as well? Logged on users would be able to go to a bug’s page on AMO, and report a bug on the AMO bug tracker. Then addon developers would be able to see all reported bugs, and mark them as FIXED when they are fixed in the addon. In a way I envision this being similar to Get Satisfaction. Not just a comment section on a blog, but not a full bug tracker either. In addition, web developers that already have a bug tracker and don’t want to use the simple AMO one, can disable it and redirect users to their own bug tracker.
    The nice thing about this is we can direct users to the bug tracker, it would be the same for all extension on AMO, and users would not feel that the burden of reporting a bug is entirely left up to them. We could still resolve bugs on BMO as INVALID, but we can then put a “See Also” link to the appropriate AMO bug when it gets filed. This would make duplicate tracking much easier and users wouldn’t feel like their problem is being ignored.

I am not sure how feasible the second option is, but I don’t see it being too difficult. I plan to try to discuss this with a few AMO guys when I get the opportunity to. Comments are always welcome, maybe there is another option I haven’t thought of or ways to improve the above two.

My Goals for 2011

Well, I’ve decided that I should try to put my goals into words, so then I can go back to them at the end of the year, and see how I’ve done. I always will try to set goals, then modify them halfway through. Maybe if I post them for the Internets to see, I shall be held accountable 😉

  1. Get the number of UNCO Bugs in Firefox down to the 2,000 range and steady. This doesn’t mean closing them all as INCO. I had to do that to get the numbers down to a humanly reachable level. But now that they are almost there (I consider ~4,000 as humanly manageable) I instead to go through and change quantity for quality. Hopefully this will not backfire on me, but I do intend to clean BMO up as much as I can, thus improving it for triagers and developers alike.
  2. Improve the Triage process. I am going to try to brainstorm with other triagers as much as possible, and then maybe this year get some of the ideas that we discussed at the Summit into reality.
  3. Improve Triage relationship with Developers. I would like to see someday a developer or module peer to come to the triage community and be able to say “I have this query that I would like help in going through”. And the triage community would do it. I like the idea of bug days that we used to have, but those ended up turning into just “old bug cleanup days”, rather boring and uninspiring while important non the less. If we can get developers and triagers working together, then I think we can make Bugzilla work for us, not against us.
  4. And other bits and pieces. But these 3 main goals will be enough to keep me busy for a while at least 😉