Even More Clarifications
Apparently my last blog post got picked up by both Conceivably Tech and Slashdot, somewhat to my surprise. It also came to my attention that I was a bit too ambiguous in how I worded things, so I wanted to make a few clarifications.
Firefox did not ship with 6,000 bugs. Rather, there are 6,000 UNCO bugs in the Firefox Product on BMO. Not 6,000 real bugs, but rather 6,000 bugs that haven’t been marked as NEW. Out of those 6,000 bugs, there are duplicates, bugs that have already been fixed, bugs that are user error, bugs that are caused by third-party software, and so on. I simply wanted to suggest that we need to do a better job of going through the bugs that are submitted. Without going through the list, it is difficult to determine which bugs are real, and which are invalid.
What is more important to me than the number of bugs in Bugzilla is how fast we respond to the bugs that are submitted as UNCONFIRMED. If we don’t respond to bugs submitted to BMO within a reasonable time frame, the chance of missing a major regression or of something else slipping by is much greater. With the new Rapid Release, it makes it even more important that we strive as hard as we can to get a quick response to end-user bug submissions.
The situation with Triage is not hopeless. I have been in talks over the past few days, and I see a good possibility that Mozilla means business in improving triage. My blog post was a summary of the state of triage today. There is a new “Tell Us More” input tool in the works, which will implement some of my desired changes (such as a separate product for UNCONFIRMED bugs). Several other ideas are being bounced around, so hopefully progress will be made soon.
To those that have given me the title “Community Lead”, I am not a Mozilla Employee, nor did I ever have a “Title”. I am a volunteer and have never been paid or been “appointed” by Mozilla as a “Community Lead”. My opinions are mine only, and are not an official statement from Mozilla or any Mozilla employee.
I’m going to write another blog post in a day or two concerning some more details and clear up the confusion that was apparently in my last blog post.
The life and adventures of Tyler 
now look at the mess you’ve made..
No mess, just a misunderstanding, and a tech news site looking to get a shocker headline.
That’s a bit harsh, don’t you think. He’s allowed to have an opinion. It’s not his fault that the media seem to read different words to what is written and take things out of context.
This is a discussion about Mozilla QA. Most tech news sites have no QA whatsoever, I’m sorry to say. The only place you’ll see tech news more pathetically stumbled through than tech news blogs is on TV.
Personally, I would say the biggest problem with UNCO bugs is the problem that a vast portion are filed by people who simply should not be anywhere near Bugzilla. These bugs often get outright ignored because nobody wants to have to deal with them. Once upon a time I tried to triage these sorts of bugs but after having to deal with way too many horribly confused screaming children (sometimes, completely literally) I just didn’t want to waste my free time on them anymore. Nowadays when I do any triaging I only focus on ones that are mildly coherent and let “someone else” deal with the messy cases. This seems to be a fairly well defined path to fall into for some triagers, and others just ignore many of the messy ones as well, hence the untouched pile. I really wish there was some way I could just shunt clearly misplaced reports to SUMO (or similar) without having to directly deal with people that often spam me with confused complaints. As Gavin said in his blog reply, it isn’t worth the effort to bother with a large pile of these, and in my opinion it would be if we just had better ways to do so.
There’s a special place in hell for Slashdot submitters …
Tyler, I’m curious why you decided to stop contributing in Bugzilla. It seems to me that the UNCO backlog makes your contributions even more valuable.
You also have the option of focusing your triage efforts on a specific component. There are plenty of components small enough for you to “own” as a triager, yet large enough for a developer to be very appreciative of the help. When you “own” triage for a component or type of bug, like I did briefly for crash bugs, you can make your own rules and whiteboard flags.
Unfortunately, while there is a backlog of UNCO bugs (which i actually decreased from 13,000 to around 5,500, it is going back up now, it was alot of closing bugs as INCO), there is little incentive in the Mozilla community outside of QA to make things better. At Summit, there were several proposals, and nothing was done ( not even getting a triage member on the security group so far as I know), and since July 2010, there has been one good improvement, the guided bug reporting form.
Hey Tyler! Did you see the questions on Slashdot?
I did, however, I’ve been a bit busy lately, but I do intend to write a follow up post in a few days that will answer most of those questions.
I think you were honest and correct in your original post, and this post is more or less PR for people that don’t know what you were getting at.
Here is a handy chart: https://bugzilla.mozilla.org/reports.cgi?product=Firefox&datasets=UNCONFIRMED&datasets=NEW&datasets=ASSIGNED&datasets=FIXED&datasets=INCOMPLETE
I agree that there are to many open, new, and unconfirmed bugs, and that lots of what gets submitted is by peeps that shouldn’t be submitting.
However… the bug counts need to go down. Its more or less work that needs to be done that hasn’t been caught up on yet. If it was all labeled for what it really is, it would help find what you are looking for in bug search queries (as opposed to digging through pages of irrelevant results) and help find what is open that is actually a priority.
If Mozilla actually wants people to submit bugs, there needs to be an attempt to actually look at the bugs submitted and figure out what is going on. You actually saw this… and people are just miffed that you are actually calling them out for doing a crappy/lazy job.
You Go Gurl!
For people wondering about the drop in the chart, there was a Graveyard move on 2008-07-31. What that means is that a bunch of components were acknowledged to not actually be part of shipping products, so all bugs in those components were moved to components in corresponding components in products in the Graveyard classification. — This enables people to still see the bug reports (they aren’t deleted), while also enabling people who are searching for bugs that might affect shipping products to not turn up irrelevant bugs.
And at the same time, certain components were moved from the Firefox product to the Toolkit product, in recognition that the components were shared between Firefox and other products (e.g. Thunderbird).
https://bugzilla.mozilla.org/reports.cgi?product=Toolkit&datasets=UNCONFIRMED&datasets=NEW&datasets=ASSIGNED&datasets=FIXED&datasets=INCOMPLETE
We did this during https://wiki.mozilla.org/Summit2008
I’ve always thought that the way to solve this was to simply shut down the entire Mozilla project for a week or so.
Dedicate every single person involved in Mozilla to cleaning up Bugzilla.
Then start things up again.
The issue is not just unconfirmed bugs. It’s bug’s that are no longer relevant, bugs that have long since been fixed, etc. etc.
Bugzilla is simply umanageable at this point.
Actually that would not be a bad idea. We would need to have a game plan and the right tools to use before, but actually it is a really good idea.
We sort of did that in 2008 at the Summit in Whistler, Bugzilla was down (in fact, we did some of the bug cleanup while all the developers were experiencing a power outage).
Although, the number of people doing the work was under a dozen, I believe people overall were happy it was done. — I’m sorry we didn’t have a chance to do the same thing in 2010 (but I wasn’t there…).
In fact, using annual summits as a time to do general cleanup, and to encourage others to do cleanup isn’t a bad thing in general. It’s pretty easy for bugzilla to be configured for a week to refuse bug reports. And it isn’t too hard to send people a random buglist of say 10 bugs and ask them to triage them.
Oh well, 2010 was a missed opportunity.
You raise a valid and important point. From reading the updates and responses to this story (and falling in to the category of a technically aware end user) it seems as though the problem here is similar to the “Emperor has no clothes” and that you are just the first to point out that the process we follow today is not working as intended.
There are several different conventional solutions to this problem.
We could put more people on to the triage work until we break the back-log, then keep a lid on it. This is difficult to do with a largely volunteer community. Even if it could be done to a reasonable degree, it would probably last until the next major release…
Alternatively, we could find a way to simplify the differentiation between the “flames” and the genuine bug reports. When faced with a similar challenge in my workplace, my solution was to introduce an optional set of questions for the submitter to complete. These were written to require the submitter to really stop and think about what they were doing. They did not enforce a technical requirement but they did enforce discipline. We then used software to filter the list of submitted issues so that it was easy for us to address the well-formed responses first.
I am sure that you can think of other ways.
I guess the point I am trying to illustrate is that in certain systems, we go in to a situation knowing that we’re going to get dirty data. We then have to write something capable of filtering out the noise. We then tune that software until we get an acceptable level of false positives.
Spamassassin has been doing this with incredible success for years.
Do you remember that quote that goes, “Life today is a race between software engineers, who are striving to produce more powerful and capable idiot-proof software, and the universe, which is striving to produce bigger idiots. So far, the universe is winning.” ?
That’s our challenge, right there.
So we need to put an “Idiot Detection Test” into the bug report submission process…
I think the new guided bug reporting form on BMO does actually a fairly decent job at making sure Reporters actually want to submit a bug. Of course, people will always ignore the suggestions, but, at least we can try.
One thing that could be done is to ask people to triage 3 bugs before filing their first bug.
You risk getting bad triage, but you explain as part of the filing form that being a bad triager will not earn you points and will cause your bug to get bad treatment — http://en.wikipedia.org/wiki/The_Golden_Rule
The 3 bugs should generally be picked from the same component that the reporter is trying to file their bug in. In ideal cases it acts as a “getting acquainted with” process.
I feel like the same issue is present in many open source projects, definitely in Ubuntu and its’ Launchpad… 😦
Hey man, don’t sweat the hype. And I certainly hope that you don’t feel guilty. You’ve been speaking your mind.
Freedom is the most precious thing in life.
Without it, nothing is worth a damn.
Also remember that there are many, many, individuals in the media and on high profile sites that have turned against Mozilla, primarily for their own agenda to promote Chrome (if indirectly or not) and they jump on any chance to publish something negative about Mozilla.
They’ve been framing Firefox as a slow, outdated memory hog that is just copying Chrome to try and catch up, and they attack Mozilla the organization to try and make it seem as if things are falling apart all over.
They don’t get to sit in on the conversations at Microsoft and Google to see that the same crap goes on there and at every other business and organization.
They also don’t realize that we’re a family and despite issues, we still respect and watch out for each other. Especially when were being attacked collectively, or individually.
I don’t even know you. But I have your back and I know that you don’t mean any ill will towards Mozilla.
Peace bro.
Thanks Ken. Honestly, I never meant this to grow this big. All I wanted was to spark discussions among the triage team, the QA team, and Mozilla leadership, and frankly am a bit confused as to the media response. I think that it partly stems from a misunderstanding on what triage actually is, and what an UNCO bug is. But thanks, I appreciate the encouragement.
Perhaps I should leave a comment for the media saying there is nothing to see here? This is a discussion for triagers about Triage. While I welcome input from other people, please try to understand that when I say “bug” it isn’t a real bug in the software (necessarily), but merely a submission to bugzilla.
You could state that content on your blog may not be reused in any manner without permission. And if they want a statement, they can make an appointment for an interview.
Grabbing quotes and misunderstanding them and/or taking them out of context shouldn’t be a free for all.
By the way, do they even know that anyone can go file a bug to have free beer and pizza coupons provided upon successful product installs?
Hrm, the closest I can recall is a bug I filed about not being able to find a Pizza Hut due to a broken web site — https://bugzilla.mozilla.org/show_bug.cgi?id=102849
Makes sense. I wonder if your follow up will get as much coverage? 🙂
Doh – this was meant to be in reply to: