Links, Lots of Links

I wanted give a bit of input to an idea I have started to hear tossed around which I find quite intriguing. Shutting down BMO for a week or so, and devoting ALL Mozilla’s resources (devs, QA, the community, everyone) to cleaning up BMO. With 5868 UNCO bugs, and 7191 NEW bugs in Firefox alone, we obviously have a bit of need for cleaning up. Then throw in Toolkit (UNCO bugs, NEW Bugs), Thunderbird (UNCO bugs, NEW bugs), Core (UNCO bugs, NEW bugs), and so many other components and products, and really, we do have a mess on our hands.

The solution isn’t as simple as it sounds. If we just shut BMO down now, we will be using resources ineffectively, and without a long-term solution we will be right back at square 1 within a year or two. I whole-heartedly approve of shutting BMO down for a week or so, but I also know we need a multi-pronged approach. If you read my series on why we need to radically change triage, you can read most of my points there, and they are ones I have been saying repeatedly for the past year. I’ll quickly summarize them here though.

  • Create a separate product for UNCO bugs and triage. There is a new Input Tool in the works right now, and it will be using this idea in a more constrained form. I have had some discussions with the people working on it, and hopefully we can get this UNCO product rolled out to cover all bugs submitted for Firefox/Toolkit/Thunderbird/etc. I do see promise of progress here, so this is good news. Once we get this fully implemented, I feel all UNCO and abandoned NEW bugs should be moved into the product.
  • Create a way for Triagers to tell how far a bug is in the triage process, using multi-state flags. This idea was proposed at the Summit 2010 by tmyoung. Unfortunately, a lot of devs didn’t want to have their bugs cluttered up with more flags (which is understandable). By implementing the flags only in the UNCO product, they will serve triagers, and once a bug is moved out of the product, they will be no longer be needed and removed. Hopefully this way we can make both sides happy.
  • We desperately need a way to mark Support and Extension bugs as such. Having a RESOLVED>Support status and a way to automagically open a support issue for the bug will go far to help users feel that we care about them and their issues. The same goes for extensions.
  • We need Bug Triage Guidelines. Something along the lines of what I threw onto the wiki a few months ago (obviously a lot more fleshed out). But something the community, especially new members, can reference.
  • I think that if we implement a lot of the ideas that David Eaves has proposed, we can go far in improving both triage, and BMO in general. I know a few of his suggestions have already been put into production, but I don’t see any of them that would harm the community, and I think they should be thought about again.
  • I’m not sure how practical this is, but perhaps we can create some sort of bot to automatically ping a triager when a bug has gone a certain amount of time without a comment, and to ping reporters when they have not replied to their bug after a certain amount of time. This would eliminate much of the time that is spent “cleaning up”, freeing more time for “real” triage.
  • Hire several official Traigers. To do several things (most of which I’ve already discussed, so I give it in brief here). Organize the community, help new community members, communicate with Devs, do full-time triage work, be advocates for the community in Mozilla.
  • Having a yearly intense cleaning like Mike proposed would also be great, because even with the best of efforts, some bugs will slip through the cracks. The first one I think we should set a week aside for, but subsequent BMO cleaning could probably only be done in 3-4 days.

So I know this is a fairly short summary, but these are by far my most important changes to BMO, and the Triage process that I think need to be made. Some progress is happening, so I look forward to seeing what is done in the near future.

Have a good Labor Day

I am going to be using this long weekend to do a lot of reading, and private discussions on triage, and hopefully come back early next week with some more blog posts. However, in the meantime, I recommend that everyone read Mike Kaply’s blog post on how to deal with the backlog of Bugs on Bugzilla. It is an idea I’ve had before, but never talked about, and I think that once we get a game plan for triage down, it should be taken VERY seriously.

Answering some Questions, and Responding to Comments, Part 1

It has been an interesting week. I published a blog post describing in detail some of my concerns with Triage on August 27th. I intended to simply spark a discussion about some of the finer details of the Triage issues, and explain a few of the reasons why I did and didn’t leave. It was written by a triager for triagers, and as such, it used terms that probably weren’t understood by many outside of the triage and Mozilla community. I would recommend that anyone who wants to a good grounding in how (BMO) works, and what UNCONFIRMED Bugs, Triage, etc. are, read the links at the end of this post. And if anyone needs clarification on the technical details of the What and How of BMO and Triage, I’d be more than happy to write a post on that.

I have gotten quite a few questions and comments regarding triage however, so I wanted to divide them up into a couple main themes that I noticed, and discuss them that way. I will be repeating myself from time to time to hammer home points that are important.

Some Questions and (hopefully) Answers:

  • Should we focus on bringing bugs to zero?

Matthias Versen: “In the beginning i thought that we should try to manage every incoming bug report but i realized later that this isn’t possible. It’s not possible because mozilla is a huge project with millions of users and the target to get all UNCO Firefox bug reports to zero will never happen.”

Gavin Sharp: “I don’t think that focusing on the number of UNCO bugs is useful. We should focus on outcomes, not on driving a number to 0 for the sake of it.”

So, we have the question, what should the goal of triage be? To have zero UNCO bugs? To drive every bug to resolved within a week?

I feel the goal of triage is to filter out the real bugs from the duplicates, invalid and spam bugs in an efficient, professional and through manner. The number of bugs shouldn’t be our emphasis, at least not the main one. We definitely don’t want to repeat the 13,000 bug low that we once had. As I have said, the number of bugs doesn’t matter as much to me as how fast we respond. We have Rapid Release; maybe we should have Rapid Response for triage. I don’t mean closing bugs quickly. I mean not letting a bug languish for 6 months, or even years without a comment.

It would actually be quite an interesting metric to see what the average response time to an UNCO bug is now. Then, as it seems that at least a few of my ideas are being implemented into Triage, to re-measure after they have begun to take effect. While a Rapid Response won’t fix every issue in Triage, by keeping our reporters happy, we will be able to keep a dedicated community willing to contribute again in the future.

What would help our Rapid Response strategy most will be a separate UNCO Bug Product, making bugs that need to be triaged easier to find. Along with that, we need some sort of triage flag, so then we can tell what stage of the triage process a bug is in. Organization, so the community can focus on areas of BMO that haven’t been triaged thoroughly yet. Finally, we really badly need several people to do triage full-time. Community Volunteers who are working in their spare time simply can’t always get to every bug. The community does an awesome job (pretty much the only job, as there are no Official Triagers), but, it isn’t quite enough.

  • We need Increased Communication between Triager and Developer

Steve Fink: “I just feel like there’s a conversation that needs to be happening between triagers and developers, and it isn’t, for the most part.” (I also recommend you read the rest of his comment)

There is, and it isn’t happening. We need to turn triage from Triagers vs. Developers to Triagers & Developers vs. Bugs. Triage and Development have the same end-goal, to make a product better, and to fix bugs. In theory, they make a perfect team. Triage takes incoming bugs, filters out the junk, the spam, the distractions, and hands the workable bugs over to devs, who then fix them. In reality, there is very little communication between the triage community and the developer community. I’m sure a lot of this problem stems from the lack of communication within the triage community itself. I would love to have developers come to the triage group and say “I have X amount of bugs that have to do with printing. Here is the list, would you mind working it a bit.” All this without having to organize a bugday. Having a quick reaction time to changes and trends in the Rapid Mozilla is essential.

We also need to learn from developers how they want their bugs triaged. Certain developers may not like certain methods of triage (“don’t use the closeme tag” for example). That is fine, so long as the triage team is aware. Once a good inter-group communication method is figured out, it would be nice to sit down with the various module owners, and find ways that triage can benefit them, and what they feel are dos and don’ts.

But communication needs to be two way. Steve also said “As a developer, I have an instinctive fear of getting sucked into triage because I know it has an insatiable appetite for time. I’d bet many devs would be willing to help with the overall triage process. But if there’s a hint that it means we have to jump on every vague bug report that a triager forwards to us and work on it until it’s good and dead, we’ll run away and hide under the nearest potted plant.”

Triagers often do need to ping a developer on a bug to get input on if it is a real bug or not. Many developers are good about replying when a long-time triager pings them on a bug, but for new beginners, it can be hard to figure out whom to ping, and once you do, to get a response. Having some sort of an agreement with even just a few developers who will agree to help out the triage group when it is needed would be nice.


I want to try to split this into several chunks that are easy to digest and discuss. So, I’ll be coming out with a few posts in the next couple of days. Let’s try to keep any discussion relevant to the points I have in each post, it will make my life a bit easier when trying to follow-up. I will get to the all in time, don’t worry.

For Further Reading

The Current Mozilla Wiki Page on Triage

Even More Clarifications

Apparently my last blog post got picked up by both Conceivably Tech and Slashdot, somewhat to my surprise. It also came to my attention that I was a bit too ambiguous in how I worded things, so I wanted to make a few clarifications.

Firefox did not ship with 6,000 bugs. Rather, there are 6,000 UNCO bugs in the Firefox Product on BMO. Not 6,000 real bugs, but rather 6,000 bugs that haven’t been marked as NEW. Out of those 6,000 bugs, there are duplicates, bugs that have already been fixed, bugs that are user error, bugs that are caused by third-party software, and so on. I simply wanted to suggest that we need to do a better job of going through the bugs that are submitted. Without going through the list, it is difficult to determine which bugs are real, and which are invalid.

What is more important to me than the number of bugs in Bugzilla is how fast we respond to the bugs that are submitted as UNCONFIRMED. If we don’t respond to bugs submitted to BMO within a reasonable time frame, the chance of missing a major regression or of something else slipping by is much greater. With the new Rapid Release, it makes it even more important that we strive as hard as we can to get a quick response to end-user bug submissions.

The situation with Triage is not hopeless. I have been in talks over the past few days, and I see a good possibility that Mozilla means business in improving triage. My blog post was a summary of the state of triage today. There is a new “Tell Us More” input tool in the works, which will implement some of my desired changes (such as a separate product for UNCONFIRMED bugs). Several other ideas are being bounced around, so hopefully progress will be made soon.

To those that have given me the title “Community Lead”, I am not a Mozilla Employee, nor did I ever have a “Title”. I am a volunteer and have never been paid or been “appointed” by Mozilla as a “Community Lead”. My opinions are mine only, and are not an official statement from Mozilla or any Mozilla employee.

I’m going to write another blog post in a day or two concerning some more details and clear up the confusion that was apparently in my last blog post.

Some Clarification and Musings

As many of you know, I have officially ended my contributions to Mozilla for the time being. Since I published that post, I have been asked by several people why I left, and ways that I think Mozilla could change to be more friendly to Triage. So, I thought I would give a blog post to try to explain a few things, and just maybe spark a bit of discussion on Why triage needs to be improved.

Update: Please read this post in conjunction with this one.

Non Reasons:

First off, I did not leave because of rapid release. I love the idea of rapid release, and I think the recent spurt of posts to the planet on how Rapid Release will be beneficial in the long run does a great job of explaining it. Rapid Release is going to be awesome if done properly. I have always been so frustrated by the continual late releases that hold back awesome new features from the web. I think Rapid Release should have been planned out better before it was put into effect (figuring out a plan for the Enterprise, making the update process smoother BEFORE jumping into it, could have been explained better). But I think that excellent progress is being made on all these fronts, and hopefully by the end of the year, many of the quirks of the system will be ironed out. I wish it had been done earlier, but that is neither here nor there.

Nor did I leave because of a recent conversation concerning the removal of version numbers, at least it was not the main reason I left. I feel that the whole conversation could have been handled much better on both sides of the board (For those who were calling for murders over a SOFTWARE change, you seriously need to get a life. Put down the computer, go camping for a weekend, then come back when you can act reasonably. Even though I firmly believe in Mozilla’s mission, software is not that important to be threatening people’s lives, and frankly, acting like a spoiled 2-year old.) But, it was not the determining factor in my decision. It would be silly to leave a project that I have contributed to for 3+ years over a misunderstanding.

Why I Left

I left because of a general lack of interest in doing anything substantial to improve the Triage process on BMO outside the QA community and a few others. Triage as we know it today is NOT ready to handle the Rapid Release process. While Releng, Devs, QA, SUMO, etc. have been able to transition to the new process relatively smoothly, Triage was caught off guard. And not through a lack of trying. I have pushed for change, and I know others have as well. Change has been made. But it is not enough. I understand that change takes time, and there is always a delay between planning a change, and the implementation. But with Triage, time is our enemy. We currently have 2598 UNCO bugs in Firefox that haven’t been touched in 150 days. That is almost 2600 bugs that have not been touched since Firefox 4 was released. And how many more bugs have been touched, but not really triaged or worked on? Every day this number grows. 27 UNCO bugs have been filed in Firefox alone in the last 24 hours, and that over a weekend.

Triage is Broken. Period.

With the old model of releasing a new major version once a year, triage had a bit more time to go through a massive pile of bugs, to find regressions and issues, and there was a pretty good chance that most bugs would get caught, just because we had time on our side, and we could afford to miss a bug for 6 weeks, because we would most likely get around to it. Or so went the theory. Even this process failed us.

Organization, Community, Politeness and Professionalism

In Spring 2010, we hit roughly 13,000 UNCO bugs in the Firefox product on BMO. 13,000!!! We currently have 5934. While this is an improvement, that is 6000 bugs in Firefox that could be shipping today, and enhancements that could be making the web better (of course it isn’t that high, but the potential is there). This is several thousand contributors that we have told “Thank you for filing a bug report with us. We don’t really care about it, and we are going to let it sit for 6 months and just ask you to retest when you know it isn’t fixed, but thank you anyway. Oh, and Mozilla is run by the community.” Even though nobody means this, that is what we tell an end-user who just submitted their first bug and is ignored. We can’t do this. We just can’t. Now, I realize that it is impossible to touch every bug immediately, but, to those who have gone the extra mile to create a BMO account, file out the bug report form, and navigate BMO emails just to report something they feel is important, don’t we owe at least a little common courtesy?

Perhaps, if a bug isn’t replied to immediately, an automated email could be sent to say “hey, we haven’t forgotten you, we appreciate the bug report. All the Triagers are busy, but here are some steps you can take to test if this bug goes away:” or something along those lines. Building a friendly relationship with our community not only gives them confidence that their report will be taken care of, but it also helps them want to come back.

We also need more coordination as a community so we can touch all these bugs in an effective and professional manner. Having better tools for the Triage community to find bugs that haven’t been replied to within a certain time period. Having better communication between developers and triagers, and between triagers themselves. I want a developer to be comfortable asking the triage group “I have this list of 200 bugs I would like triaged in this way. Thanks”, and it to be done, without a bug day. And I want there to be developers who when they are pinged on a bug by a triager, to reply and assist with the process as needed. Along these lines, THIS IS AWESOME! More of this, lots lots lots more please! I also want the triage group to have an easy way to be able to say “Hey, I noticed that Bug XXXXXX seems to have lots of dupes, let’s keep an eye on it.” Or “I think it is about time that we go through and clean up some old bugs.” With multiple eyes looking at the same list, we can help keep bug numbers down.

I realize that triagers are volunteers, and thus may not be held to the same standards as a Mozilla employee. But I see no reason that they shouldn’t be. As a Triager, I have unique opportunity. I get to be the person that most people will associate with Mozilla, at least at first. This is incredible, and terrifying. Everything I do, and how I treat people’s bugs, is going to influence their perception of Mozilla for a long time. Giving one sentence replies, being rude and thinned skinned gets you nowhere. I have learned this the hard way many times. We need to have a set guideline of how to treat specific bugs, and how to treat their reporters, and somehow ensure that Triagers are following these guidelines. While there are spammers, and people who are just trying to cause problems, the vast majority of BMO bug creators are just trying to make the software they use a little bit better.

We need the Right Tool

Even if we create all the best tools in the world, if we are disorganized, it isn’t going to do any good. We might have 5000 UNCO bugs instead of 6000, but that isn’t an improvement. On the other hand, we can be organized, but if we don’t have good tools, we are going to be trying to cut down a tree with a pocket knife.

Right now, there is no real way to triage except “Here is a list of 1700 bugs. Start at the top and work your way down.” We need a way to mark bugs that need triage (Like Getting Bugs Done and Triage Flags). But we also need to remember, BMO is being used for things it never was created for. BMO is either an end-user bug submission forum, or a developer bug submission database. It can’t be both, but that is what it is trying to do. It worked when it was small, it kinda worked when we took a year or two to release a new versions, it doesn’t work with Rapid Release.

This is why I have envisioned a separate BMO product of UNCO bugs. That way, we can separate End-user bug submission from the development process, at least in the beginning stage. We can make a simplified interface for those who are just starting out with Mozilla. We can add flags to bugs, that while useless for developers, are incredibly useful for Triagers. We can have a different set of rules for the UNCO product than we do for the Developers. While there is no perfect answer, while we can never make everyone happy, we can do what is better for the community as a whole.

I also strongly feel that Mozilla needs at least 1-3 people to just handle Triage on an Official Full-Time capacity. Pretty much all the community Triagers I know have jobs, families, lives outside of Mozilla. Mozilla is just one of their hobbies. While they are passionate about what they do, they don’t have the time it takes to full-time triage. And with Rapid Release, this is what we need. Plus, a Mozilla Employee has a lot more weight at getting changes done, organizing and revitalizing the community, and helping continually improve Triage.


I know this was a long and rambling blog post, and I covered a lot of things with little cohesion. I felt it was important to get my thoughts out in one final attempt to maybe get some traction going on fixing Triage for the good of Mozilla. Without long-term changes, I feel that triage is going to be facing a larger and larger pile of bugs that doesn’t go down, and eventually may just go away out of frustration. And this has nothing to do with the community. When I see what triagers have to go through to do a good job, and they still stick with it, I know that if Triage goes away, it won’t be the fault of the community. It is the only thing keeping it together in fact.  I hope to see some sort of improvement in Triage, and when I do, I will jump right back in. But as it is, I see no use in contributing to a project that seems to have gained little attention from Mozilla. No, there is no glamor in Triage. But neither is there any glamor in 20,000 UNCO bugs.

I am not looking for zero bugs, I am not even looking for less than a thousand. I am looking for timely responses to all or nearly all bugs reported, and better tools and organization in the triage community.

I should also say, for the past two weeks I have been in discussions with several Mozilla Employees and Contributors on how Triage can be improved. I simply wanted to be fair to those in the community that were not aware of all my concerns with Triage, and perhaps to get more feedback. So, this is my in-depth explanation of last week’s post.

(These are my personal opinions, and are not meant in any way to attack Mozilla or any Employee of Mozilla or Contributor. It is simply my frustrations, and personal opinions of what can be done to improve triage. I am not saying my ideas are perfect, but I want them more to serve as a catalyst of HOW to improve triage for the long-term. Please, feel free to contact me for further discussions.)

Goodbye Mozilla

Not all good things can last forever. I’ve seen over the past several months that as Mozilla has drifted further away from the company that it was in 2008, and further from my goals and vision, and that it is best that I depart from the Mozilla community. Rather than hash out all of my issues in a public blog post, I am just announcing that I will no longer be contributing in any way to the Mozilla Community. I will not be receiving anymore notifications on BMO. I will have the same email address, so if anyone ever wants to get in touch with me they can do so. Any patches that I have floating on BMO, feel free to use if there is anything of value (I doubt there is). My blog will be removed from the planet soon, but I will continue to post personal updates and posts on it (there is some big news coming up soon, so stay tuned).

I have many friends in the Mozilla community, and I want to tell you that this is not any grievance that I have with you, but more on the policies and direction that Mozilla is headed in the future, as well as the treatment of the community in recent months. I wish Mozilla luck, and I will continue to watch it. Sometime in the future, I may begin to contribute again. It is just better that after 3+ years, I pursue other, more productive and rewarding contributions with projects that truly appreciate their communities. Thank you Mozilla for a fun past 3 years, and my friends, peers, and mentors, thank you.

Why we Need to Radically Change Bugzilla and Triage, Part 4

I could also title this, What Needs to be Done to Ensure this Works.

I’ll try to summarize in a few short paragraphs.

Mozilla Needs Official Triagers

Instead of relying on a community to handle all the bug triage, Mozilla needs to hire several people to work full-time, who’s job it is to triage. I think that a Firefox/Toolkit, a Core, and a Thunderbird/Seamonkey triager are the minimum. By hiring triagers, Mozilla will help eliminate several problems.

  • The Mozilla triagers will be able to coordinate the community. Instead of relying on individual uncoordinated effort, the Mozilla triage team can focus the community on areas of BMO where their efforts would be most effective at the time.
  • Instead of each person having their own ways to handle different types of bugs (how long should a closeme bug wait before being closed), the Mozilla Triagers could lay out guidelines for all bugs in their product(s). they would be able to coordinate with developers on how they want specific bugs treated, and ensure the community followed the guidelines.
  • Mozilla Triagers would ensure there were always people to provide cover of UNCO bugs. By ensuring there are people who are paid to work on triage, the chances of having 13,000+ bugs would be far less.

The community is a very powerful tool if used effectively. But Mozilla has never learned how to harness the triage community. Bugdays have been of limited effectiveness. They are useful, but not coordinated enough to have long term impact. The QA team tried to assist with Triage, but they are divided between so many projects that devoting alot of time to triage is understandably difficult.

By centrally organizing and assisting triage, the Mozilla Triage team would improve our current triage process dramatically. And that is an understatement. Without Official Mozilla recognition and assistance of triagers, I see the community totally disappearing within two years.

And I know there will be various forms of “why do we need triage” in the comments. If you feel that way, please think about what Bugzilla would look like without the triage community right now. I plan on posting about this shortly, on why we need triage, and why it is so difficult.