Posts Tagged ‘ bugs ’

End-User Bugs on Bugzilla

Mozilla offers an opportunity normally not available to users of most software. The ability to submit bugs directly to the developers of the software and watch the progression of the bug. With this comes a responsibility normally not faced by software companies. Because users can report these bugs, it means that unless they are looked at, the reporter and other user’s experiencing the issue quickly become frustrated that they are being ignored. The whole benefit of having end-users be able to watch the bug process can bite Mozilla in the tail. It frequently seems that the trend in Bugzilla is to divide bugs into quickly fixed and never fixed.

Bug Species

Bugs reported by regular end-users fall into one of two species.

  1. Not a real Firefox bug. Extension issues, spam bugs, support bugs, etc. These bugs are usually directed to either SUMO or AMO to talk to those who can properly assist.
  2. A real Firefox Bug. These are then split down further.
  • A bug that is already reported, is on a developer’s radar, etc. These bugs are typically resolved relatively quickly. Either as DUPE, FIXED, etc.
  • A bug that has not been reported before, or is not affecting a developer’s “pet project”. These bugs are typically ignored and shoved aside. They may be fixed in later years when the component they are filed against is rewritten, or some developer stumbles against the bug later on and fixes it. These bugs may be edge-case bugs, or really minor issues, but they are still bugs.

Now I realize developers have to have priorities. P1, P2, etc. I’m not expecting every bug to be treated the same. not all bugs were created equals, and there is no way to fix every bug within the first week it is filed. And fixing bugs isn’t very glamorous work. It looks a lot better with a new release to say “Redesigned component X with New Features A, B and C” compared to “Fixed 20 bugs in print preview”.

Scenario

But which makes a greater impact on our users? Ensuring that our users have a solid product with minimal bugs, or giving them all the bling that “the Other Browser” has. I love new Features. I think all the hard work in Firefox 4 like the redesigned Addons Manager, Theme, etc are great and help make a competitive product. But, running through triage I see so many bugs marked as NEW, and then never looked at again (As I commented on in another post). This is disheartening to users. Imagine the scenario:

John is IT tech at a local community bank. He finally convinced the IT Dept. Head to let him run a test run of Firefox on all the bank’s computers. Everything seems to be running great. Until, a few days later, John starts to notice complaints that their proprietary web-based plugin site does not allow the use of navigation keys when the plugin form has focus. This is a major issue for the bank because the tellers are trained to only use the keyboard for sake of speed. Some terminals don’t even have mice, meaning they essentially become trapped once focus is captured by the plugin. With no easy answer to the issue and no patch forthcoming in over 10 years, John is forced to switch to a browser that doesn’t have the same issue. All the great features, speed, and functionality came to nothing when a relatively minor bug caused a major breakdown in the bank’s operations.

All of this could have been avoided if a developer had taken notice of the bug and it had been addressed years before.

Solutions?

Now as someone I respect very much in Mozilla has said “If all bugs were security bugs, they would be fixed very quickly”. And major regressions, and common bugs are frequently fixed very quickly (ie. within a month or so). However, the sad reality is that many bugs reported by end-users go ignored for a very long time. If it is not acceptable for a company making a physical product to have a 2% failure rate, how can we ship a product with over 7000 bugs in NEW, ASSIGNED and REOPENED? I know it is impossible to reduce that number to even 1,000. But, I think we should be able to cut those NEW bugs at least. Maybe what is needed is a developer (or three) who is assigned for a while to go through, reproduce old NEW bugs and fix them. I think that the Papercuts push was a great idea, the unfortunately kinda flopped. Maybe a push like this on every major release of Firefox would help, so long as the follow through was there. Ideas?

Advertisements

A Picture is Worth a Thousand Words

Thanks to a fellow community member for pointing it out. Graph is here. (Click Image for full size)

My Goals for 2011

Well, I’ve decided that I should try to put my goals into words, so then I can go back to them at the end of the year, and see how I’ve done. I always will try to set goals, then modify them halfway through. Maybe if I post them for the Internets to see, I shall be held accountable 😉

  1. Get the number of UNCO Bugs in Firefox down to the 2,000 range and steady. This doesn’t mean closing them all as INCO. I had to do that to get the numbers down to a humanly reachable level. But now that they are almost there (I consider ~4,000 as humanly manageable) I instead to go through and change quantity for quality. Hopefully this will not backfire on me, but I do intend to clean BMO up as much as I can, thus improving it for triagers and developers alike.
  2. Improve the Triage process. I am going to try to brainstorm with other triagers as much as possible, and then maybe this year get some of the ideas that we discussed at the Summit into reality.
  3. Improve Triage relationship with Developers. I would like to see someday a developer or module peer to come to the triage community and be able to say “I have this query that I would like help in going through”. And the triage community would do it. I like the idea of bug days that we used to have, but those ended up turning into just “old bug cleanup days”, rather boring and uninspiring while important non the less. If we can get developers and triagers working together, then I think we can make Bugzilla work for us, not against us.
  4. And other bits and pieces. But these 3 main goals will be enough to keep me busy for a while at least 😉

UNCO Bugs and Useful Information: First Take

Recently, I have been focusing on cleaning up old, forgotten and outdated bugs on Bugzilla. Currently we have almost 8000 bugs that are UNCO in the Firefox Product (an extremely handy table) (there are roughly 1100+ bugs waiting until Jan 30th to be closed as INCO). This number is down from a total of 12,000+ in July at the Mozilla Summit (Not a very good graph, but gets the point across). As far as I can tell this was an all-time high in Firefox, which began around the middle of 2009, when UNCO bugs really took off. Over the next few months I’m going to try to analyze why this massive jump in UNCO bugs happened, what Mozilla can do to help prevent it, problems with the current method of CLOSEME etc. Tonight I’m going to give some first impressions of this 6+ month personal project, as well as ways to help improve the triage of the 6-7 thousand bugs left.

My Process

My usual process in cleaning up bugs in the past few months has been three main steps:

  1. Run a query of bugs I want to target. This started with bugs that had been filed against Firefox 1, 1.5, and 2.0. These versions of Firefox have been End of Life for quite a while now. I encouraged all reporters of these bugs to retest their bug against the most current version of Firefox at the time I ran the query, update their plugins, create a fresh profile, then report the results. The vast majority of bug reporters never got back, which is sad, but something for another blog post. Of those that did (I’d say roughly <10%), the majority didn’t see the issue anymore with the current Firefox (Kudos to our developers, some of the greatest out there). Those few (~1%) that still saw the issue I then went back (yes I was getting hundreds of emails a day for weeks after running a query) and updated their bug with correct version info and asked for anymore information that was needed. I also tried to place it in a better component than Firefox:General (the location most my queries ran through).
  2. When I went through and marked the bugs as CLOSEME in the whiteboard, I typically picked a date 3-4 weeks into the future. I like to give reporters plenty of time to come back. In fact today I received a bug mail from a bug I had marked as CLOSEME in Nov 2010, and closed as INCO in December. The reporter came back with more information. When the date rolled around, I’d typically wait even a few days after the date, then I’d run through and close all bugs that hadn’t received a reply as RESOLVED INCOMPLETE.
  3. After the mass INCO, I’d usually receive another round of emails. Reporters who missed the first email, people whose bugs had slipped though (unfortunately it happens, my bugzilla query skills are by no means ninja), and people who instead of commenting mark their bug as RESOLVED FIXED, without a patch being checked in.

So that pretty much sums up my methods for closing those forgotten bugs in BMO. Now, some issues I’ve run into…

Whiteboard UNCO Bugs

In the process of marking bugs as CLOSEME and INCOMPLETE, I ran into an issue with the current process of running mass queries and marking bugs as CLOSEME. This issue is mainly the fault of a bug in Bugzilla itself, which has sat inactive since 2006. It is that when you go to change multiple bugs in a query at once and make a whiteboard change, it erases the whiteboard, instead of appending the new info. I’m hoping this bug gets fixed sometime soon (birthday present Bugzilla developers? ;). In the meantime, I think this has exposed a bit of a flaw in our process for dealing with bugs.

Currently, we have the keyword list for bugs, which has a large list that we can choose from to mark bugs (checkin-needed being one I use from time to time myself). However, there are many de-facto “keywords” that are used in the whiteboard that aren’t in the keyword list. DUPEME, CLOSEME, etc. Various groups and development leads within Mozilla also use the whiteboard for their own uses ([needs review] [sg:dos] [see comment X], etc.).

Unfortunately, while running through the bug queries of bugs that qualified for closing (I usually make it so the minimum days before I close a bug is roughly 300), I ran into several bugs, in the Firefox:General component that had not been touched in well over 300 days that had this comments in the whiteboard. Due to this Bugzilla bug of erasing whiteboard entries, and me not thinking about it before running queries, these bugs lost the previous whiteboard entries. So, to you guys that lost bugs due to my queries, I do sincerely apologize. Several people have kindly brought this to my attention, and I’m going to be excluding bugs that have content in the whiteboard from future queries.

Closing Real Bugs

While running these queries I’ve often tried to exercise caution that I don’t accidentally close a real legitimate bug, hence my long waiting period before actually closing. I fear that doing mass closings could leave rarely run into bugs in Firefox in the wild without being fixed. However, my reasoning behind this is this: These bugs are not getting any attention right now as they are. The majority of them (and this has been backed up for the past 6 months in bug replies) have been fixed already in new versions of Firefox. Closing as INCO is completely reversible so the bugs are not lost. You can still run queries against them. Closing the bug actually increases the chances that the reporter will come back and look at it, giving us more much needed information. And having a backlog of 12,000 bugs is hurting Firefox development more than helping. I feel that the pro vastly outweigh the cons.

Now, if you have a bug that was closed but you are still seeing it using the current Firefox release or a nightly, please, reopen it, remove the CLOSEME, and mark it against the version you tested. This doc might help you. Just because your bug was closed is nothing against you, nor does it mean we don’t think it is valuable. It just means that more information is needed, and it wasn’t provided.

Ways to Improve the Process

It’s too soon for me to really have too many ideas on how to “fix” bug closure. It has always been painful, and I think it will always be. There may be some ways that we can improve it though.

  • Fix the bugs: On my wish list for fixing, Bug 330509 andBug 577561. These would help future bug closures much easier, making whiteboard dataloss harder, and marking newly filed Firefox bugs with the correct version number, helping prevent 6,000+ Unconfirmed Unspecified version bugs.
  • Take the whiteboard out of the equation in closing bugs: Using the whiteboard for CLOSEME is really a hack that has turned into accepted practice. I don’t know how we could fix this, but possibly having a checkbox that we can use to mark bugs as Inactive after a certain number of days. These bugs wouldn’t be INCOMPLETE per say, but they would be excluded from queries unless you select to search Inactive bugs, and as soon as a comment is made, the Inactive field would be cleared automatically.
  • Make it so end user’s cannot close as FIXED: The majority of our UNCO bugs are reported by end-users. And let’s face it, Bugzilla is not friendly towards end-users. Even long-time developers still fight with Bugzilla from time to time. These reporters see a “RESOLVED FIXED” status, think “My bug disappeared when this update came through”, and naturally think “I should mark this FIXED”. They are only trying to be helpful, we don’t give them much else of a chance. We don’t let just anyone mark a bug as NEW, why should we let them mark it as FIXED? Maybe we can change Bugzilla permissions so anyone who does not have CANCONFIRM can’t close a bug a FIXED. I’ve filed Bug 624135 to address this.

I’m sure I’m going to have more ideas and more issues as the months go on, and I plan on trying to expand my ideas further. This is kind of just a first take, I’m just throwing ideas out there. Please, let me know if you have any ideas on ways to improve triage. Let’s brainstorm.

Gearing Up for Summit

With only 3 more weeks until Mozilla Summit 2010, things are getting busy. I received my tickets the other day, and it looks like I’m going to be riding a CRJ to Vancouver,  and an A320 back. As my last ride in a CRJ was very smooth, but rather uncomfortable seating, let’s see how it goes this time 🙂 It also looks like I have to wake up dark and early to leave, but my flight back will be later in the day, which is always nice.

For my time at summit, it has been kinda tough to schedule, seeing st the schedule is not released yet. However, there looks like there will be some breakout sessions on triage which I plan to attend, and some other interesting topics to be discussed. The face-to-face will be great as well. I will be posting pics and updates daily at least (as I am able to) from this blog, so stay tuned.

As for my little status update, until I get back from Summit I will be quite busy. With my day job, volunteering at the County stampede, plus prepping for the trip, not many bugs will be getting squashed. I hope to come after them with a vengeance!

Advertisements