Posts Tagged ‘ bugzilla ’

Why We Close Bugs

It seems that my recent post on requesting retest on NEW bugs showed a divide in the community. There are certain community members who agree that old bugs should be pinged, while others think that it is an annoyance rather than a help. And I agree it is a fine line that needs to be walked. There are pros and cons, and I think we as a community need to determine where to draw the line. Closing bugs as INCOMPLETE hurts, but it helps too.

Cons of closing bugs:

  • It can be annoying. Getting thousands of bugspam emails at once is pretty annoying (I know, I get them too). And if your bug has slipped into a query it can be annoying to be asked if you’ve reproduced a bug in a newer version of Firefox when you already have. It happens.
  • It can appear that Mozilla does not care about users and their Bugs. When a user files a bug in 2006, and it never gets touched until a comment is placed on it in 2011 asking the user to test again in the most recent Firefox version, reporters can understandably feel unappreciated.
  • Real bugs can be closed in the process. In closing bugs, we run a slight chance of accidentally closing a bug that is an edge case and has not had any dupes filed.

Pros of closing bugs:

  • It assists with finding bugs that will actually be fixed. If a developer has a list of 1000 bugs covering multiple versions of Firefox, it is a bit hard to find a bug with enough information to fix. However, if Triage has gone through that list, weeded out the ones that do not have active reporters anymore and closed them, that list could potentially go down to 300 bugs. All 300 with recent comments from their reporters or other users who have reproduced the bug, giving recent relevant information. A list of 300 up-to-date bugs is much better than 1000 of varying quality.
  • It enlists the help of reporters to reproduce bugs. There are simply too many bugs for the Triage team and QA team to reproduce all of them at once. By asking reporters to reproduce, we can help enlist our community more.
  • We have nothing to lose. These bugs are sitting there stagnate anyway. Nothing will change if we don’t go after them. If we do comment on the bugs, the chance that we will get valuable information is significantly higher than if they just sat. We have nothing to lose, and plenty to gain.

Now, there are some ways we can improve:

  • Write better follow up comments. Possibly, something like David Eaves suggested in his blog post. Prettified emails. Perhaps Bugzilla could be modified to send an mmail saying “Since you reported this bug, two new versions of Firefox have been released, with many changes in x y z. You can read the release notes of these updates at mozilla.com. The Mozilla community would appreciate it if you would take the time to retest your bug XXXXXX with the latest version using a fresh profile. And update your bug accordingly.” Or something like. I’m not sure how feasible this is, but we definitely need to make our bug replies more friendly.
  • Be better at following up ourselves. I mean come on, we ask the reporter to follow up on their bug, then we as Mozilla let the bug sit, again. It is disheartening to a reporter to be like “They asked me to follow up, maybe they are finally going to fix the darn thing.” Then it sits, again, for years because as a community we dropped the ball. I think that is what is more of a discouragement to our community of bug reporters. That they report a bug that never gets fixed. They spent time that they didn’t have too creating a bugzilla account, reporting a bug, retesting a bug (perhaps multiple times) then it sits. We really really need to pick up the game here. And start to focus on our community a little more.

What to do, with Old NEW Bugs?

Well, I knew that sometime this would have to be tackled. After closing another batch of ~1100 old UNCO bugs the other night, I decided to turn to our other forgotten bug problem, the NEW bugs. First, a bit of backstory.

NEW has been almost like the second trial of a bug. If a bug makes it through the graveyard that is Firefox General UNCO, it either is Assigned to a developer, and is fixed relatively quickly, or, it gets thrown into the NEW pile. These are bugs that are a legitimate problem (or at least, were bugs when marked NEW). However, the developers either need more information about the bug, or are waiting for time to fix them. Unfortunately, because we have so much to do and so little time, many of these bugs end up being forgotten. Almost 2000 in fact. That is almost 2000 NEW bugs in Firefox (not including enhancement requests) that have not been changed in over 500 days. That’s well over a year.

Many of these bugs may still be valid issues. Some could be meta bugs tracking other issues that are being fixed. Some might be still issues. The question is, how many of those bugs are actually fixed in recent releases of Firefox, and could legitimately be closed?

Would it be worth the time putting a comment on all very old Firefox General bugs, asking for the reporter or reproducer to retest, and then update the bug if it is still an issue? Then, those bugs which do not receive a reply, in 3 months or so, I can go back, and mark them as CLOSEME for a date a month down the road. Then if they have still not been commented on, they can be closed as INCO.

This will give three reminders to the reporter to comment on the bug. The initial request for retesting, the second marking as CLOSEME, then the actual closing. We would filter out as many known legitimate bugs as possible (bugs blocking other bugs, bugs being blocked, meta bugs, bugs that are in components that peers ask not to be touched, etc).

It is just my feeling that this backlog of NEW bugs is hurting triage as much as the UNCO bugs were. It also hurts QA by giving them too much bug noise, and developers have a hard time determining if a new bug really is legitimate anymore.

Comments?

My Triage Process

Well, I’m back. The puncture wounds have healed quite a bit (though I shall have more scars to add to my collection of accident related injuries). The Radial nerve got injured so I still have no feeling on my left thumb, and occasional shooting pain, but that should go away within the next few weeks. So, I’m back to blogging, and bug triage.

When I first started triage almost 3 years ago, I didn’t really have a process for triage. It was usually to +start with a list of all the Bugs in a certain component, and just start at the top and work my way down. This is unfortunately one of the only ways we can triage with so many UNCO bugs. trying to run queries either pulls up very small and limited results, or pages that are hundreds or thousands of bugs long. The situation is improving though.

When I triage, I have several methods that I use. The most usual is running through a query that returns all the bugs filed in the past 24 hours. It returns all bugs in Firefox, Toolkit and Core from the past 24 hours that are UNCO, NEW, ASSIGNED and RESOLVED. (link to come soon, BMO is taking forever to run queries ATM for some reason). I will then browse the whole list (usually around 100-150 bugs a day).

I first start with what you could call “low hanging fruit”. I go through and find all bugs that look like they will be support issues, extension issues (ones that have words like “Google Toolbar” in the summary are usually a good start).  I will go through, and try to resolve those bugs first to cut down on bug noise. Extension issues usually end up getting reported to the developer of the problem extension, and support bugs get directed to SUMO.

Then I begin to go through the other bugs. It is not possible to look at or touch every bug. I am not an expert in every component of the products I triage (not an expert in any I have to admit), and some bugs are simply above my head. Other bugs have already been touched by some of our other excellent triagers. However I try to at least look at every UNCO bug submitted in any given day. I usually visit this query several times a day.

I then start to plunge into the depths of BMO. I will either go through and mark untouched bugs as CLOSEME (I usually focus on that more at the beginning of the month), or I will start to run specific queries I want to focus on. For example, some months I will run queries in Firefox looking for bugs in printing (which is often improperly filed in Firefox General instead of the Core Components). Other days I will look for specific extensions or common issues I have seen trending recently and will go and search for more dupes.

What is your triage process? Any ideas for ways I can change or improve? Most of triage is a personal style. There is no cut-and-dried “perfect” technique to Triage. Different people work better different ways. It is all about how you work that makes our Triage community unique and vibrant. Our differences make Mozilla what it is today.

I’m trying to toss around some ideas on improving specific Triage, I’ll be posting soon on some thoughts for improving communication between developers and Triagers, something that is sorely lacking at the moment.

My Goals for 2011

Well, I’ve decided that I should try to put my goals into words, so then I can go back to them at the end of the year, and see how I’ve done. I always will try to set goals, then modify them halfway through. Maybe if I post them for the Internets to see, I shall be held accountable 😉

  1. Get the number of UNCO Bugs in Firefox down to the 2,000 range and steady. This doesn’t mean closing them all as INCO. I had to do that to get the numbers down to a humanly reachable level. But now that they are almost there (I consider ~4,000 as humanly manageable) I instead to go through and change quantity for quality. Hopefully this will not backfire on me, but I do intend to clean BMO up as much as I can, thus improving it for triagers and developers alike.
  2. Improve the Triage process. I am going to try to brainstorm with other triagers as much as possible, and then maybe this year get some of the ideas that we discussed at the Summit into reality.
  3. Improve Triage relationship with Developers. I would like to see someday a developer or module peer to come to the triage community and be able to say “I have this query that I would like help in going through”. And the triage community would do it. I like the idea of bug days that we used to have, but those ended up turning into just “old bug cleanup days”, rather boring and uninspiring while important non the less. If we can get developers and triagers working together, then I think we can make Bugzilla work for us, not against us.
  4. And other bits and pieces. But these 3 main goals will be enough to keep me busy for a while at least 😉

UNCO Bugs and Useful Information: First Take

Recently, I have been focusing on cleaning up old, forgotten and outdated bugs on Bugzilla. Currently we have almost 8000 bugs that are UNCO in the Firefox Product (an extremely handy table) (there are roughly 1100+ bugs waiting until Jan 30th to be closed as INCO). This number is down from a total of 12,000+ in July at the Mozilla Summit (Not a very good graph, but gets the point across). As far as I can tell this was an all-time high in Firefox, which began around the middle of 2009, when UNCO bugs really took off. Over the next few months I’m going to try to analyze why this massive jump in UNCO bugs happened, what Mozilla can do to help prevent it, problems with the current method of CLOSEME etc. Tonight I’m going to give some first impressions of this 6+ month personal project, as well as ways to help improve the triage of the 6-7 thousand bugs left.

My Process

My usual process in cleaning up bugs in the past few months has been three main steps:

  1. Run a query of bugs I want to target. This started with bugs that had been filed against Firefox 1, 1.5, and 2.0. These versions of Firefox have been End of Life for quite a while now. I encouraged all reporters of these bugs to retest their bug against the most current version of Firefox at the time I ran the query, update their plugins, create a fresh profile, then report the results. The vast majority of bug reporters never got back, which is sad, but something for another blog post. Of those that did (I’d say roughly <10%), the majority didn’t see the issue anymore with the current Firefox (Kudos to our developers, some of the greatest out there). Those few (~1%) that still saw the issue I then went back (yes I was getting hundreds of emails a day for weeks after running a query) and updated their bug with correct version info and asked for anymore information that was needed. I also tried to place it in a better component than Firefox:General (the location most my queries ran through).
  2. When I went through and marked the bugs as CLOSEME in the whiteboard, I typically picked a date 3-4 weeks into the future. I like to give reporters plenty of time to come back. In fact today I received a bug mail from a bug I had marked as CLOSEME in Nov 2010, and closed as INCO in December. The reporter came back with more information. When the date rolled around, I’d typically wait even a few days after the date, then I’d run through and close all bugs that hadn’t received a reply as RESOLVED INCOMPLETE.
  3. After the mass INCO, I’d usually receive another round of emails. Reporters who missed the first email, people whose bugs had slipped though (unfortunately it happens, my bugzilla query skills are by no means ninja), and people who instead of commenting mark their bug as RESOLVED FIXED, without a patch being checked in.

So that pretty much sums up my methods for closing those forgotten bugs in BMO. Now, some issues I’ve run into…

Whiteboard UNCO Bugs

In the process of marking bugs as CLOSEME and INCOMPLETE, I ran into an issue with the current process of running mass queries and marking bugs as CLOSEME. This issue is mainly the fault of a bug in Bugzilla itself, which has sat inactive since 2006. It is that when you go to change multiple bugs in a query at once and make a whiteboard change, it erases the whiteboard, instead of appending the new info. I’m hoping this bug gets fixed sometime soon (birthday present Bugzilla developers? ;). In the meantime, I think this has exposed a bit of a flaw in our process for dealing with bugs.

Currently, we have the keyword list for bugs, which has a large list that we can choose from to mark bugs (checkin-needed being one I use from time to time myself). However, there are many de-facto “keywords” that are used in the whiteboard that aren’t in the keyword list. DUPEME, CLOSEME, etc. Various groups and development leads within Mozilla also use the whiteboard for their own uses ([needs review] [sg:dos] [see comment X], etc.).

Unfortunately, while running through the bug queries of bugs that qualified for closing (I usually make it so the minimum days before I close a bug is roughly 300), I ran into several bugs, in the Firefox:General component that had not been touched in well over 300 days that had this comments in the whiteboard. Due to this Bugzilla bug of erasing whiteboard entries, and me not thinking about it before running queries, these bugs lost the previous whiteboard entries. So, to you guys that lost bugs due to my queries, I do sincerely apologize. Several people have kindly brought this to my attention, and I’m going to be excluding bugs that have content in the whiteboard from future queries.

Closing Real Bugs

While running these queries I’ve often tried to exercise caution that I don’t accidentally close a real legitimate bug, hence my long waiting period before actually closing. I fear that doing mass closings could leave rarely run into bugs in Firefox in the wild without being fixed. However, my reasoning behind this is this: These bugs are not getting any attention right now as they are. The majority of them (and this has been backed up for the past 6 months in bug replies) have been fixed already in new versions of Firefox. Closing as INCO is completely reversible so the bugs are not lost. You can still run queries against them. Closing the bug actually increases the chances that the reporter will come back and look at it, giving us more much needed information. And having a backlog of 12,000 bugs is hurting Firefox development more than helping. I feel that the pro vastly outweigh the cons.

Now, if you have a bug that was closed but you are still seeing it using the current Firefox release or a nightly, please, reopen it, remove the CLOSEME, and mark it against the version you tested. This doc might help you. Just because your bug was closed is nothing against you, nor does it mean we don’t think it is valuable. It just means that more information is needed, and it wasn’t provided.

Ways to Improve the Process

It’s too soon for me to really have too many ideas on how to “fix” bug closure. It has always been painful, and I think it will always be. There may be some ways that we can improve it though.

  • Fix the bugs: On my wish list for fixing, Bug 330509 andBug 577561. These would help future bug closures much easier, making whiteboard dataloss harder, and marking newly filed Firefox bugs with the correct version number, helping prevent 6,000+ Unconfirmed Unspecified version bugs.
  • Take the whiteboard out of the equation in closing bugs: Using the whiteboard for CLOSEME is really a hack that has turned into accepted practice. I don’t know how we could fix this, but possibly having a checkbox that we can use to mark bugs as Inactive after a certain number of days. These bugs wouldn’t be INCOMPLETE per say, but they would be excluded from queries unless you select to search Inactive bugs, and as soon as a comment is made, the Inactive field would be cleared automatically.
  • Make it so end user’s cannot close as FIXED: The majority of our UNCO bugs are reported by end-users. And let’s face it, Bugzilla is not friendly towards end-users. Even long-time developers still fight with Bugzilla from time to time. These reporters see a “RESOLVED FIXED” status, think “My bug disappeared when this update came through”, and naturally think “I should mark this FIXED”. They are only trying to be helpful, we don’t give them much else of a chance. We don’t let just anyone mark a bug as NEW, why should we let them mark it as FIXED? Maybe we can change Bugzilla permissions so anyone who does not have CANCONFIRM can’t close a bug a FIXED. I’ve filed Bug 624135 to address this.

I’m sure I’m going to have more ideas and more issues as the months go on, and I plan on trying to expand my ideas further. This is kind of just a first take, I’m just throwing ideas out there. Please, let me know if you have any ideas on ways to improve triage. Let’s brainstorm.

Auto Filling the Version Field

Right now, in the Firefox product, there are over 7300 bugs that have the version set to “Unspecified”. This makes it difficult to accurately track which bugs are filed against each version of the product, as you have to go in, read the build identifier (which is a comment), and manually change the version field. We automatically UA sniff and fill in the platform and OS fields, could we not automatically fill in the product version field for the Firefox product? This would not work for most products (eg. Thunderbird), but for bugs filed in the Seamonkey, Camino and Firefox products is it possible to auto fill this field in, then ask the user to confirm that it is correct, as we already do with the platform and OS? And if the user is using another browser (eg, using IE to file a bug against Firefox, or using Firefox to file a bug against Seamonkey), just detect that the UA is not for the product it is being filed against, and set to Unspecified? This would not help those bugs already on Bugzilla, but would make triaging these bugs in the future easier.

However, knowing little about BMO development and maintenance, I’m not sure how feasible or practical this would be 🙂

Pac-Man Turns 30. And the number of bugs jumps considerably

So, Pac-Man turned 30 today. While I’m not quite that old, I do remember playing Pac-Man on library computers roughly 12 years ago. As Google so often does, it changed it’s logo to a Pac-Man stylized Google. As Google has never done before, they turned the “doodle” into a playable one. So, when you go to the Google homepage, you are greeted by Pac-Man sounds, sirens, and flashing colors, with a little mac-man (or Mrs. Pac-Man if you find the secret button) controllable with your keyboard. Quite fun and nostalgic. And soon enough, quite irritating.

It has also apparently attracted the attention of quite a few bug reporters on bugzilla. Kudos to the support guys for putting up an article so quickly to address this.

What surprised me is the sheer number of people who have no idea where the sounds are coming from, yet instantly identify them as “Pac-Man”. Congrats Namco for creating possibly the most universal sound in the computing world. Not even the AIM closing door sound is as recognized.