UNCO Bugs and Useful Information: First Take

Recently, I have been focusing on cleaning up old, forgotten and outdated bugs on Bugzilla. Currently we have almost 8000 bugs that are UNCO in the Firefox Product (an extremely handy table) (there are roughly 1100+ bugs waiting until Jan 30th to be closed as INCO). This number is down from a total of 12,000+ in July at the Mozilla Summit (Not a very good graph, but gets the point across). As far as I can tell this was an all-time high in Firefox, which began around the middle of 2009, when UNCO bugs really took off. Over the next few months I’m going to try to analyze why this massive jump in UNCO bugs happened, what Mozilla can do to help prevent it, problems with the current method of CLOSEME etc. Tonight I’m going to give some first impressions of this 6+ month personal project, as well as ways to help improve the triage of the 6-7 thousand bugs left.

My Process

My usual process in cleaning up bugs in the past few months has been three main steps:

  1. Run a query of bugs I want to target. This started with bugs that had been filed against Firefox 1, 1.5, and 2.0. These versions of Firefox have been End of Life for quite a while now. I encouraged all reporters of these bugs to retest their bug against the most current version of Firefox at the time I ran the query, update their plugins, create a fresh profile, then report the results. The vast majority of bug reporters never got back, which is sad, but something for another blog post. Of those that did (I’d say roughly <10%), the majority didn’t see the issue anymore with the current Firefox (Kudos to our developers, some of the greatest out there). Those few (~1%) that still saw the issue I then went back (yes I was getting hundreds of emails a day for weeks after running a query) and updated their bug with correct version info and asked for anymore information that was needed. I also tried to place it in a better component than Firefox:General (the location most my queries ran through).
  2. When I went through and marked the bugs as CLOSEME in the whiteboard, I typically picked a date 3-4 weeks into the future. I like to give reporters plenty of time to come back. In fact today I received a bug mail from a bug I had marked as CLOSEME in Nov 2010, and closed as INCO in December. The reporter came back with more information. When the date rolled around, I’d typically wait even a few days after the date, then I’d run through and close all bugs that hadn’t received a reply as RESOLVED INCOMPLETE.
  3. After the mass INCO, I’d usually receive another round of emails. Reporters who missed the first email, people whose bugs had slipped though (unfortunately it happens, my bugzilla query skills are by no means ninja), and people who instead of commenting mark their bug as RESOLVED FIXED, without a patch being checked in.

So that pretty much sums up my methods for closing those forgotten bugs in BMO. Now, some issues I’ve run into…

Whiteboard UNCO Bugs

In the process of marking bugs as CLOSEME and INCOMPLETE, I ran into an issue with the current process of running mass queries and marking bugs as CLOSEME. This issue is mainly the fault of a bug in Bugzilla itself, which has sat inactive since 2006. It is that when you go to change multiple bugs in a query at once and make a whiteboard change, it erases the whiteboard, instead of appending the new info. I’m hoping this bug gets fixed sometime soon (birthday present Bugzilla developers? ;). In the meantime, I think this has exposed a bit of a flaw in our process for dealing with bugs.

Currently, we have the keyword list for bugs, which has a large list that we can choose from to mark bugs (checkin-needed being one I use from time to time myself). However, there are many de-facto “keywords” that are used in the whiteboard that aren’t in the keyword list. DUPEME, CLOSEME, etc. Various groups and development leads within Mozilla also use the whiteboard for their own uses ([needs review] [sg:dos] [see comment X], etc.).

Unfortunately, while running through the bug queries of bugs that qualified for closing (I usually make it so the minimum days before I close a bug is roughly 300), I ran into several bugs, in the Firefox:General component that had not been touched in well over 300 days that had this comments in the whiteboard. Due to this Bugzilla bug of erasing whiteboard entries, and me not thinking about it before running queries, these bugs lost the previous whiteboard entries. So, to you guys that lost bugs due to my queries, I do sincerely apologize. Several people have kindly brought this to my attention, and I’m going to be excluding bugs that have content in the whiteboard from future queries.

Closing Real Bugs

While running these queries I’ve often tried to exercise caution that I don’t accidentally close a real legitimate bug, hence my long waiting period before actually closing. I fear that doing mass closings could leave rarely run into bugs in Firefox in the wild without being fixed. However, my reasoning behind this is this: These bugs are not getting any attention right now as they are. The majority of them (and this has been backed up for the past 6 months in bug replies) have been fixed already in new versions of Firefox. Closing as INCO is completely reversible so the bugs are not lost. You can still run queries against them. Closing the bug actually increases the chances that the reporter will come back and look at it, giving us more much needed information. And having a backlog of 12,000 bugs is hurting Firefox development more than helping. I feel that the pro vastly outweigh the cons.

Now, if you have a bug that was closed but you are still seeing it using the current Firefox release or a nightly, please, reopen it, remove the CLOSEME,Β and mark it against the version you tested. This doc might help you. Just because your bug was closed is nothing against you, nor does it mean we don’t think it is valuable. It just means that more information is needed, and it wasn’t provided.

Ways to Improve the Process

It’s too soon for me to really have too many ideas on how to “fix” bug closure. It has always been painful, and I think it will always be. There may be some ways that we can improve it though.

  • Fix the bugs: On my wish list for fixing, Bug 330509 andBug 577561. These would help future bug closures much easier, making whiteboard dataloss harder, and marking newly filed Firefox bugs with the correct version number, helping prevent 6,000+ Unconfirmed Unspecified version bugs.
  • Take the whiteboard out of the equation in closing bugs: Using the whiteboard for CLOSEME is really a hack that has turned into accepted practice. I don’t know how we could fix this, but possibly having a checkbox that we can use to mark bugs as Inactive after a certain number of days. These bugs wouldn’t be INCOMPLETE per say, but they would be excluded from queries unless you select to search Inactive bugs, and as soon as a comment is made, the Inactive field would be cleared automatically.
  • Make it so end user’s cannot close as FIXED: The majority of our UNCO bugs are reported by end-users. And let’s face it, Bugzilla is not friendly towards end-users. Even long-time developers still fight with Bugzilla from time to time. These reporters see a “RESOLVED FIXED” status, think “My bug disappeared when this update came through”, and naturally think “I should mark this FIXED”. They are only trying to be helpful, we don’t give them much else of a chance. We don’t let just anyone mark a bug as NEW, why should we let them mark it as FIXED? Maybe we can change Bugzilla permissions so anyone who does not have CANCONFIRM can’t close a bug a FIXED. I’ve filed Bug 624135 to address this.

I’m sure I’m going to have more ideas and more issues as the months go on, and I plan on trying to expand my ideas further. This is kind of just a first take, I’m just throwing ideas out there. Please, let me know if you have any ideas on ways to improve triage. Let’s brainstorm.


Firefox 3.5 EOL?

Has Firefox 3.5 been End Of Lifed? I think it’s about time, seeing as it’s been almost a year since we released Firefox 3.6, we are getting ready to release Firefox 4, and the download page for 3.5 says it isn’t receiving security updates, even though it is. It’s alot of work to maintain 3 versions of Firefox.

Busy busy

Alright, well, this summit has been much busier than anticipated. I haven’t been able to keep up with blogging as much as I desired. But, it is quite rewarding. Last night I had a 4 hour discussion with tmyoung, ctalbert, and several other triage people about how we can improve our triage process. This afternoon we are going to have a breakout session about this same topic, and it promises to be very rewarding.

Off we go, into the wild blue yonder!

I am currently sitting in the airport, waiting for my 8:18 flight from DIA to Vancouver. Late night and early morning for me, but I think it will be worth it. I am looking forward to the Summit, especially meeting new people. But I also plan to have a full schedule of keynotes, breakout sessions, and listening to lightening talks, and impromptu meetings on how we can improve our triage methods and get more bugs confirmed, and fixes sent to our users.

I had a run in with some coffee this morning in an attempt to wake me up, and of course I was wearing a white shirt. So, mozillians, that is why I have coffee stains, it is not my normal appearance. If you are looking for me, look for a slightly disheveled guy with glasses and a white shirt with coffee stains πŸ˜‰ He will probably look somewhat tired as well….

Gearing Up for Summit

With only 3 more weeks until Mozilla Summit 2010, things are getting busy. I received my tickets the other day, and it looks like I’m going to be riding a CRJ to Vancouver,Β  and an A320 back. As my last ride in a CRJ was very smooth, but rather uncomfortable seating, let’s see how it goes this time πŸ™‚ It also looks like I have to wake up dark and early to leave, but my flight back will be later in the day, which is always nice.

For my time at summit, it has been kinda tough to schedule, seeing st the schedule is not released yet. However, there looks like there will be some breakout sessions on triage which I plan to attend, and some other interesting topics to be discussed. The face-to-face will be great as well. I will be posting pics and updates daily at least (as I am able to) from this blog, so stay tuned.

As for my little status update, until I get back from Summit I will be quite busy. With my day job, volunteering at the County stampede, plus prepping for the trip, not many bugs will be getting squashed. I hope to come after them with a vengeance!

New Name for Blog

On an unrelated topic, how about a new name for this blog? Something a bit more interesting and, shall I say, witty ;).

Ideas are welcome! πŸ˜€

Auto Filling the Version Field

Right now, in the Firefox product, there are over 7300 bugs that have the version set to “Unspecified”. This makes it difficult to accurately track which bugs are filed against each version of the product, as you have to go in, read the build identifier (which is a comment), and manually change the version field. We automatically UA sniff and fill in the platform and OS fields, could we not automatically fill in the product version field for the Firefox product? This would not work for most products (eg. Thunderbird), but for bugs filed in the Seamonkey, Camino and Firefox products is it possible to auto fill this field in, then ask the user to confirm that it is correct, as we already do with the platform and OS? And if the user is using another browser (eg, using IE to file a bug against Firefox, or using Firefox to file a bug against Seamonkey), just detect that the UA is not for the product it is being filed against, and set to Unspecified? This would not help those bugs already on Bugzilla, but would make triaging these bugs in the future easier.

However, knowing little about BMO development and maintenance, I’m not sure how feasible or practical this would be πŸ™‚