Recently, I have been focusing on cleaning up old, forgotten and outdated bugs on Bugzilla. Currently we have almost 8000 bugs that are UNCO in the Firefox Product (an extremely handy table) (there are roughly 1100+ bugs waiting until Jan 30th to be closed as INCO). This number is down from a total of 12,000+ in July at the Mozilla Summit (Not a very good graph, but gets the point across). As far as I can tell this was an all-time high in Firefox, which began around the middle of 2009, when UNCO bugs really took off. Over the next few months I’m going to try to analyze why this massive jump in UNCO bugs happened, what Mozilla can do to help prevent it, problems with the current method of CLOSEME etc. Tonight I’m going to give some first impressions of this 6+ month personal project, as well as ways to help improve the triage of the 6-7 thousand bugs left.

My Process

My usual process in cleaning up bugs in the past few months has been three main steps:

1. Run a query of bugs I want to target. This started with bugs that had been filed against Firefox 1, 1.5, and 2.0. These versions of Firefox have been End of Life for quite a while now. I encouraged all reporters of these bugs to retest their bug against the most current version of Firefox at the time I ran the query, update their plugins, create a fresh profile, then report the results. The vast majority of bug reporters never got back, which is sad, but something for another blog post. Of those that did (I’d say roughly <10%), the majority didn’t see the issue anymore with the current Firefox (Kudos to our developers, some of the greatest out there). Those few (~1%) that still saw the issue I then went back (yes I was getting hundreds of emails a day for weeks after running a query) and updated their bug with correct version info and asked for anymore information that was needed. I also tried to place it in a better component than Firefox:General (the location most my queries ran through).
2. When I went through and marked the bugs as CLOSEME in the whiteboard, I typically picked a date 3-4 weeks into the future. I like to give reporters plenty of time to come back. In fact today I received a bug mail from a bug I had marked as CLOSEME in Nov 2010, and closed as INCO in December. The reporter came back with more information. When the date rolled around, I’d typically wait even a few days after the date, then I’d run through and close all bugs that hadn’t received a reply as RESOLVED INCOMPLETE.
3. After the mass INCO, I’d usually receive another round of emails. Reporters who missed the first email, people whose bugs had slipped though (unfortunately it happens, my bugzilla query skills are by no means ninja), and people who instead of commenting mark their bug as RESOLVED FIXED, without a patch being checked in.

So that pretty much sums up my methods for closing those forgotten bugs in BMO. Now, some issues I’ve run into…

Whiteboard UNCO Bugs

In the process of marking bugs as CLOSEME and INCOMPLETE, I ran into an issue with the current process of running mass queries and marking bugs as CLOSEME. This issue is mainly the fault of a bug in Bugzilla itself, which has sat inactive since 2006. It is that when you go to change multiple bugs in a query at once and make a whiteboard change, it erases the whiteboard, instead of appending the new info. I’m hoping this bug gets fixed sometime soon (birthday present Bugzilla developers? ;). In the meantime, I think this has exposed a bit of a flaw in our process for dealing with bugs.

Currently, we have the keyword list for bugs, which has a large list that we can choose from to mark bugs (checkin-needed being one I use from time to time myself). However, there are many de-facto “keywords” that are used in the whiteboard that aren’t in the keyword list. DUPEME, CLOSEME, etc. Various groups and development leads within Mozilla also use the whiteboard for their own uses ([needs review] [sg:dos] [see comment X], etc.).

Unfortunately, while running through the bug queries of bugs that qualified for closing (I usually make it so the minimum days before I close a bug is roughly 300), I ran into several bugs, in the Firefox:General component that had not been touched in well over 300 days that had this comments in the whiteboard. Due to this Bugzilla bug of erasing whiteboard entries, and me not thinking about it before running queries, these bugs lost the previous whiteboard entries. So, to you guys that lost bugs due to my queries, I do sincerely apologize. Several people have kindly brought this to my attention, and I’m going to be excluding bugs that have content in the whiteboard from future queries.

Closing Real Bugs

While running these queries I’ve often tried to exercise caution that I don’t accidentally close a real legitimate bug, hence my long waiting period before actually closing. I fear that doing mass closings could leave rarely run into bugs in Firefox in the wild without being fixed. However, my reasoning behind this is this: These bugs are not getting any attention right now as they are. The majority of them (and this has been backed up for the past 6 months in bug replies) have been fixed already in new versions of Firefox. Closing as INCO is completely reversible so the bugs are not lost. You can still run queries against them. Closing the bug actually increases the chances that the reporter will come back and look at it, giving us more much needed information. And having a backlog of 12,000 bugs is hurting Firefox development more than helping. I feel that the pro vastly outweigh the cons.

Now, if you have a bug that was closed but you are still seeing it using the current Firefox release or a nightly, please, reopen it, remove the CLOSEME, and mark it against the version you tested. This doc might help you. Just because your bug was closed is nothing against you, nor does it mean we don’t think it is valuable. It just means that more information is needed, and it wasn’t provided.

Ways to Improve the Process

It’s too soon for me to really have too many ideas on how to “fix” bug closure. It has always been painful, and I think it will always be. There may be some ways that we can improve it though.

• Fix the bugs: On my wish list for fixing, Bug 330509 andBug 577561. These would help future bug closures much easier, making whiteboard dataloss harder, and marking newly filed Firefox bugs with the correct version number, helping prevent 6,000+ Unconfirmed Unspecified version bugs.
• Take the whiteboard out of the equation in closing bugs: Using the whiteboard for CLOSEME is really a hack that has turned into accepted practice. I don’t know how we could fix this, but possibly having a checkbox that we can use to mark bugs as Inactive after a certain number of days. These bugs wouldn’t be INCOMPLETE per say, but they would be excluded from queries unless you select to search Inactive bugs, and as soon as a comment is made, the Inactive field would be cleared automatically.
• Make it so end user’s cannot close as FIXED: The majority of our UNCO bugs are reported by end-users. And let’s face it, Bugzilla is not friendly towards end-users. Even long-time developers still fight with Bugzilla from time to time. These reporters see a “RESOLVED FIXED” status, think “My bug disappeared when this update came through”, and naturally think “I should mark this FIXED”. They are only trying to be helpful, we don’t give them much else of a chance. We don’t let just anyone mark a bug as NEW, why should we let them mark it as FIXED? Maybe we can change Bugzilla permissions so anyone who does not have CANCONFIRM can’t close a bug a FIXED. I’ve filed Bug 624135 to address this.

I’m sure I’m going to have more ideas and more issues as the months go on, and I plan on trying to expand my ideas further. This is kind of just a first take, I’m just throwing ideas out there. Please, let me know if you have any ideas on ways to improve triage. Let’s brainstorm.