Archive for February, 2011

Why We Close Bugs

It seems that my recent post on requesting retest on NEW bugs showed a divide in the community. There are certain community members who agree that old bugs should be pinged, while others think that it is an annoyance rather than a help. And I agree it is a fine line that needs to be walked. There are pros and cons, and I think we as a community need to determine where to draw the line. Closing bugs as INCOMPLETE hurts, but it helps too.

Cons of closing bugs:

  • It can be annoying. Getting thousands of bugspam emails at once is pretty annoying (I know, I get them too). And if your bug has slipped into a query it can be annoying to be asked if you’ve reproduced a bug in a newer version of Firefox when you already have. It happens.
  • It can appear that Mozilla does not care about users and their Bugs. When a user files a bug in 2006, and it never gets touched until a comment is placed on it in 2011 asking the user to test again in the most recent Firefox version, reporters can understandably feel unappreciated.
  • Real bugs can be closed in the process. In closing bugs, we run a slight chance of accidentally closing a bug that is an edge case and has not had any dupes filed.

Pros of closing bugs:

  • It assists with finding bugs that will actually be fixed. If a developer has a list of 1000 bugs covering multiple versions of Firefox, it is a bit hard to find a bug with enough information to fix. However, if Triage has gone through that list, weeded out the ones that do not have active reporters anymore and closed them, that list could potentially go down to 300 bugs. All 300 with recent comments from their reporters or other users who have reproduced the bug, giving recent relevant information. A list of 300 up-to-date bugs is much better than 1000 of varying quality.
  • It enlists the help of reporters to reproduce bugs. There are simply too many bugs for the Triage team and QA team to reproduce all of them at once. By asking reporters to reproduce, we can help enlist our community more.
  • We have nothing to lose. These bugs are sitting there stagnate anyway. Nothing will change if we don’t go after them. If we do comment on the bugs, the chance that we will get valuable information is significantly higher than if they just sat. We have nothing to lose, and plenty to gain.

Now, there are some ways we can improve:

  • Write better follow up comments. Possibly, something like David Eaves suggested in his blog post. Prettified emails. Perhaps Bugzilla could be modified to send an mmail saying “Since you reported this bug, two new versions of Firefox have been released, with many changes in x y z. You can read the release notes of these updates at mozilla.com. The Mozilla community would appreciate it if you would take the time to retest your bug XXXXXX with the latest version using a fresh profile. And update your bug accordingly.” Or something like. I’m not sure how feasible this is, but we definitely need to make our bug replies more friendly.
  • Be better at following up ourselves. I mean come on, we ask the reporter to follow up on their bug, then we as Mozilla let the bug sit, again. It is disheartening to a reporter to be like “They asked me to follow up, maybe they are finally going to fix the darn thing.” Then it sits, again, for years because as a community we dropped the ball. I think that is what is more of a discouragement to our community of bug reporters. That they report a bug that never gets fixed. They spent time that they didn’t have too creating a bugzilla account, reporting a bug, retesting a bug (perhaps multiple times) then it sits. We really really need to pick up the game here. And start to focus on our community a little more.

A Picture is Worth a Thousand Words

Thanks to a fellow community member for pointing it out. Graph is here. (Click Image for full size)

What to do, with Old NEW Bugs?

Well, I knew that sometime this would have to be tackled. After closing another batch of ~1100 old UNCO bugs the other night, I decided to turn to our other forgotten bug problem, the NEW bugs. First, a bit of backstory.

NEW has been almost like the second trial of a bug. If a bug makes it through the graveyard that is Firefox General UNCO, it either is Assigned to a developer, and is fixed relatively quickly, or, it gets thrown into the NEW pile. These are bugs that are a legitimate problem (or at least, were bugs when marked NEW). However, the developers either need more information about the bug, or are waiting for time to fix them. Unfortunately, because we have so much to do and so little time, many of these bugs end up being forgotten. Almost 2000 in fact. That is almost 2000 NEW bugs in Firefox (not including enhancement requests) that have not been changed in over 500 days. That’s well over a year.

Many of these bugs may still be valid issues. Some could be meta bugs tracking other issues that are being fixed. Some might be still issues. The question is, how many of those bugs are actually fixed in recent releases of Firefox, and could legitimately be closed?

Would it be worth the time putting a comment on all very old Firefox General bugs, asking for the reporter or reproducer to retest, and then update the bug if it is still an issue? Then, those bugs which do not receive a reply, in 3 months or so, I can go back, and mark them as CLOSEME for a date a month down the road. Then if they have still not been commented on, they can be closed as INCO.

This will give three reminders to the reporter to comment on the bug. The initial request for retesting, the second marking as CLOSEME, then the actual closing. We would filter out as many known legitimate bugs as possible (bugs blocking other bugs, bugs being blocked, meta bugs, bugs that are in components that peers ask not to be touched, etc).

It is just my feeling that this backlog of NEW bugs is hurting triage as much as the UNCO bugs were. It also hurts QA by giving them too much bug noise, and developers have a hard time determining if a new bug really is legitimate anymore.

Comments?

My Triage Process

Well, I’m back. The puncture wounds have healed quite a bit (though I shall have more scars to add to my collection of accident related injuries). The Radial nerve got injured so I still have no feeling on my left thumb, and occasional shooting pain, but that should go away within the next few weeks. So, I’m back to blogging, and bug triage.

When I first started triage almost 3 years ago, I didn’t really have a process for triage. It was usually to +start with a list of all the Bugs in a certain component, and just start at the top and work my way down. This is unfortunately one of the only ways we can triage with so many UNCO bugs. trying to run queries either pulls up very small and limited results, or pages that are hundreds or thousands of bugs long. The situation is improving though.

When I triage, I have several methods that I use. The most usual is running through a query that returns all the bugs filed in the past 24 hours. It returns all bugs in Firefox, Toolkit and Core from the past 24 hours that are UNCO, NEW, ASSIGNED and RESOLVED. (link to come soon, BMO is taking forever to run queries ATM for some reason). I will then browse the whole list (usually around 100-150 bugs a day).

I first start with what you could call “low hanging fruit”. I go through and find all bugs that look like they will be support issues, extension issues (ones that have words like “Google Toolbar” in the summary are usually a good start).  I will go through, and try to resolve those bugs first to cut down on bug noise. Extension issues usually end up getting reported to the developer of the problem extension, and support bugs get directed to SUMO.

Then I begin to go through the other bugs. It is not possible to look at or touch every bug. I am not an expert in every component of the products I triage (not an expert in any I have to admit), and some bugs are simply above my head. Other bugs have already been touched by some of our other excellent triagers. However I try to at least look at every UNCO bug submitted in any given day. I usually visit this query several times a day.

I then start to plunge into the depths of BMO. I will either go through and mark untouched bugs as CLOSEME (I usually focus on that more at the beginning of the month), or I will start to run specific queries I want to focus on. For example, some months I will run queries in Firefox looking for bugs in printing (which is often improperly filed in Firefox General instead of the Core Components). Other days I will look for specific extensions or common issues I have seen trending recently and will go and search for more dupes.

What is your triage process? Any ideas for ways I can change or improve? Most of triage is a personal style. There is no cut-and-dried “perfect” technique to Triage. Different people work better different ways. It is all about how you work that makes our Triage community unique and vibrant. Our differences make Mozilla what it is today.

I’m trying to toss around some ideas on improving specific Triage, I’ll be posting soon on some thoughts for improving communication between developers and Triagers, something that is sorely lacking at the moment.

Follow

Get every new post delivered to your Inbox.