End-User Bugs on Bugzilla

Mozilla offers an opportunity normally not available to users of most software. The ability to submit bugs directly to the developers of the software and watch the progression of the bug. With this comes a responsibility normally not faced by software companies. Because users can report these bugs, it means that unless they are looked at, the reporter and other user’s experiencing the issue quickly become frustrated that they are being ignored. The whole benefit of having end-users be able to watch the bug process can bite Mozilla in the tail. It frequently seems that the trend in Bugzilla is to divide bugs into quickly fixed and never fixed.

Bug Species

Bugs reported by regular end-users fall into one of two species.

  1. Not a real Firefox bug. Extension issues, spam bugs, support bugs, etc. These bugs are usually directed to either SUMO or AMO to talk to those who can properly assist.
  2. A real Firefox Bug. These are then split down further.
  • A bug that is already reported, is on a developer’s radar, etc. These bugs are typically resolved relatively quickly. Either as DUPE, FIXED, etc.
  • A bug that has not been reported before, or is not affecting a developer’s “pet project”. These bugs are typically ignored and shoved aside. They may be fixed in later years when the component they are filed against is rewritten, or some developer stumbles against the bug later on and fixes it. These bugs may be edge-case bugs, or really minor issues, but they are still bugs.

Now I realize developers have to have priorities. P1, P2, etc. I’m not expecting every bug to be treated the same. not all bugs were created equals, and there is no way to fix every bug within the first week it is filed. And fixing bugs isn’t very glamorous work. It looks a lot better with a new release to say “Redesigned component X with New Features A, B and C” compared to “Fixed 20 bugs in print preview”.

Scenario

But which makes a greater impact on our users? Ensuring that our users have a solid product with minimal bugs, or giving them all the bling that “the Other Browser” has. I love new Features. I think all the hard work in Firefox 4 like the redesigned Addons Manager, Theme, etc are great and help make a competitive product. But, running through triage I see so many bugs marked as NEW, and then never looked at again (As I commented on in another post). This is disheartening to users. Imagine the scenario:

John is IT tech at a local community bank. He finally convinced the IT Dept. Head to let him run a test run of Firefox on all the bank’s computers. Everything seems to be running great. Until, a few days later, John starts to notice complaints that their proprietary web-based plugin site does not allow the use of navigation keys when the plugin form has focus. This is a major issue for the bank because the tellers are trained to only use the keyboard for sake of speed. Some terminals don’t even have mice, meaning they essentially become trapped once focus is captured by the plugin. With no easy answer to the issue and no patch forthcoming in over 10 years, John is forced to switch to a browser that doesn’t have the same issue. All the great features, speed, and functionality came to nothing when a relatively minor bug caused a major breakdown in the bank’s operations.

All of this could have been avoided if a developer had taken notice of the bug and it had been addressed years before.

Solutions?

Now as someone I respect very much in Mozilla has said “If all bugs were security bugs, they would be fixed very quickly”. And major regressions, and common bugs are frequently fixed very quickly (ie. within a month or so). However, the sad reality is that many bugs reported by end-users go ignored for a very long time. If it is not acceptable for a company making a physical product to have a 2% failure rate, how can we ship a product with over 7000 bugs in NEW, ASSIGNED and REOPENED? I know it is impossible to reduce that number to even 1,000. But, I think we should be able to cut those NEW bugs at least. Maybe what is needed is a developer (or three) who is assigned for a while to go through, reproduce old NEW bugs and fix them. I think that the Papercuts push was a great idea, the unfortunately kinda flopped. Maybe a push like this on every major release of Firefox would help, so long as the follow through was there. Ideas?

Advertisements
  1. Hey … thanks for the writeup.

    Is there an online resource documenting the following:

    1. How to check out the latest Firefox source tree

    2. How to build/debug Firefox (tools, environment, etc)

    3. How to submit bug fixes

    4. How to submit new agreed-upon features, e.g. I submitted a bug report re: missing support for the body.onundo/redo events which are specified in the MDC docs as well as in the html5 spec but AFAIK are not working in FF4 latest build. So if I was to add support for this feature (if/when I get under the hood) how would I go about getting it included in the nightly builds? Is the process for contributing features/bug fixes for Firefox documented somewhere?

    p.s. I have a repeatable crash with the latest build of FF4 for which I filed a Critical bug report (https://bugzilla.mozilla.org/show_bug.cgi?id=639104) … It is entirely repeatable on my 3-year old Macbook but not on the Mac belonging to the developer who is looking into it. The crash happens due to a very obscure and random sequence of unicode characters (including joining and modifier characters) on a twitter.com user page mentioned in the bug report. I doubt that it affects the general stability of FF4 but if I was working on FF4 I would want to investigate it because it might be a more general problem.

  2. Awesome. Thank you. I’ll give the build/debug process a shot and see how far I get with it.

  3. Ah, RC, must time for the usual “omg how can you ship with 7000 bugs!!1” thing. 🙂

    Every piece of software ships with bugs. Go look at any project’s bug tracker. That there are bugs, or even how _many_ bugs is largely irrelevant. The more important question is if the software is improving release-to-release (or, more specifically, how many issues the average user is running into and how severe they are). By all measures I’m aware of, Firefox 4 is a significant improvement over previous releases in that regard. (If you have data to the contrary, I’d love to see it.)

    I think we’re doing a pretty good job of shipping a working product, but you’re right that the experience for users who file bugs can often be not-so-good. It’s a tough problem. The valiant triage from folks like yourself is a huge help, but the process of filing a good report, getting it to the right place, and knowing what’s going on with it can still be a daunting process — especially given the large disparity between the number of bugs being filed and the number of people to process them. I’d think we’d be very open to suggestions for improving this flow!

    Long-lived bugs (like 78414) are usually that way not because they’re being ignored, but because there are complex issues with no good solution… Your “local community bank” example could just as easily go the other way — John installs Chrome, and the bank’s proprietary plugin breaks because it expects to handle Control-W (or whatever) but Chrome doesn’t pass that keystroke to the plugin. There’s a whole side discussion to be had here if that bug was stalled for the wrong reasons (eg perfect being the enemy of the good), but at least w.r.t. the issues in your post I’d say it’s not a great example. 🙂

    Finally, as for Papercuts… It’s not flopped, at all. It’s very much alive, and will continue to be a UX priority for upcoming releases. The difficulty it’s had in the Firefox 4 cycle is that it took form too late to become a core priority (ie, people were already scheduled for other stuff). Also, many of them are significant feature changes, so activity has slowed as Firefox 4’s ship date has approached… We’ve been fixing _bugs_, not adding new features. Which I think you can appreciate! 🙂

    • Justin,
      Yes, I know that we can’t ship with 0 bugs. I’m not saying the 7000 is even too many, what I’m trying to say is that 7000 has been there for a very very long time without changing, which means there are many bugs which have not been looked at or fixed. And I do agree that Firefox 4 is an improvement of 3.6. What worries me is there is nobody who is going through these older bugs, and actually fixing them.
      I actually do have some ideas on how to improve the triage workflow. I would be more than happy to discuss them with you.
      Yes, and my example was just one using a bug that I figured everyone would be familiar with. I can create some other ones if you would like. We can easily sit around and say “john would have found bugs in any browser he used”, which is true, but again, that’s not the point. the point is, he shouldn’t HAVE to find bugs.
      I definitely hope that papercuts does take off, and that it finds a better place in a new release. It was one thing I was seriously looking forward to in Firefox 4 and was sad it didn’t make the cut all the way.

    • “Finally, as for Papercuts… It’s not flopped, at all. It’s very much alive, and will continue to be a UX priority for upcoming releases.”

      Papercuts was originally tagged as a project for Firefox 4.0. Firefox 4.0 is basically done, and last time I counted, less than 20% of the papercuts bugs are fixed (and around half of those, including the big one of making script dialogs not be app-modal, were already under way before Papercuts started). Achieving only 10%-20% of a goal sounds pretty flop-like. If it’s very much alive, then, as you say, some of them are significant feature changes, so they need to be in the plans early. In the planning discussions for the next versions, all I see are more performance work and more new features.

      Unfortunately, being a “UX priority” seems to mean pretty much nothing aside from that the UX team talk about it. The UX team, as such, don’t seem generally to be the ones that write code. They seem to rely on some developer with some spare time chancing across their list of things they want and implementing them.

      If UX priorities don’t translate into development priorities – bugs with blocking flags assigned to people who can fix them, then I don’t see that there is much point in the UX team having priorities, and it certainly looks silly when people keep saying “don’t worry – this is a UX priority” and then nothing happens. 😦

      • “Papercuts was originally tagged as a project for Firefox 4.0. Firefox 4.0 is basically done, and last time I counted, less than 20% of the papercuts bugs are fixed”

        Getting 10-20 papercuts fixed for every release is very good progress. I’m not sure where you got the idea that papercuts were a “Firefox 4” projects, papercuts are by definition a project that will always have remaining issues, and its goal was to give these a name and a bucket people could work from, so they could improve Firefox in a manner that would have a lot of impact for a (mostly, with some exceptions) simple fixes.

        I should definitely publish a blog post on the papercuts that were fixed in 4.0, though — there were many, and they very much make 4.0 into a more polished and less frustrating browser. Work Offline, anyone? 🙂

    • Markus
    • March 6th, 2011

    I guess the answer on how to finally resolve these (or at least the most annoying) bugs is to simply put them on the official development roadmap (as single and explicitly spelled out items). As long as they are only implicitly on the roadmap, nobody will really take care of, as you pointed out. And I think this has also to do with the development process: In the beginning of the development for every major release, people are focusing on new features, and try to get them done. When it’s finally about to fix remaining bugs, people are either still busy with resolving bugs for the newly introduced features, or start to realize that fixing these long outstanding bugs might introduce regressions, which would even further delay the development…

    So I would propose:
    1) Go through open bugs with most user attention/highest annoyance factor
    2) Triage and put on official development roadmap at an early stage

  4. Yeah, what you’re talking about is really closely related to the Suck Less philosophy of software development:

    http://www.codesimplicity.com/post/suck-less/

    I agree with you that those “minor” bugs that stay open for ten years really do need to be dealt with, preferably in priority order of how much they make things suck for people.

    -Max

    • pd
    • March 6th, 2011

    You’re singing my tune. I have some development skills but not enough to dive into the Mozilla ecosystem and fix these bugs. I’ve been reading Planet Mozilla for years on a daily basis waiting for someone to help me bridge my current knowledge and the massive complexity that Firefox seems to be. Maybe I’m just not a good enough developer, I do not really know.

    Maybe the answer is more education and help for potential voluntary devs like me? This is no doubt difficult because the best people to guide new developers into the codebase are also probably the best at hammering away at hardcore code. However surely there is a way?

    Is there one very experienced developer who has good people skills and might be interested in a new role of permanent voluntary developer liaison and recruitment? It strikes me that perhaps such a role could be blended into a new side project with some of the curriculum from the Seneca people. Personally I find nothing more helpful than video of person coding away and commentating on what they are doing. If videos like that could be produced in addition to Seneca curriculum, a hacks.mozilla.org style blog focused on Firefox internals, I think that would be a great start to getting more voluntary developers trained up to work on the overlooked bugs.

    • I really do agree with you that we need to help those who want to begin contributing to Mozilla development more. It is a very closed off system where everyone is expected to automatically know or learn for themselves from the out of date “manuals” on devmo.

      • If you have ideas, send them to the new Community Engagement team. Their entire purpose is to help people start hacking on, and assist people who are currently hacking on, Mozilla.

  5. When I still was mostly a user, I had a few bugs reported that got fixed, some fast, some took some time because they weren’t easy. I also know a good number of bugs reported by users that were fixed in reasonable time.
    So the world is not as black and white as you paint it.

    OTOH, I fully agree that the papercuts idea was great, and I think it mostly flopped because we tried to press too many awesome large features into Firefox 4 and had no time left to care about those papercuts. We surely should pick those up again.

    Oh, and usually, a number of those are nice projects for someone who wants to get acquainted with how to get involved – we still need to get better in holding hands of those people, though.

    • Yes, there are lots of grey, and many bugs do get fixed quickly. But again, must of them are issues that developers are running into themselves, they are major regressions, crashers, etc. The average run of the mill end-user who encounters a minor usability bug and reports it, ill probably have less than a 50-50 chance of that bug begin fixed within 3 months.

    • Frank Burleigh
    • March 6th, 2011

    Amen. A “speculative end-user problem” doesn’t have any obvious “home” in Mozilla’s support infrastructure. I experience BZ as unresponsive and really inappropriate for end-users, so seldom use it and recommend it seldom. s.m.o. is a worthwhile effort but the crowd-sourced effort doesn’t ensure a conversation. The feedback newsgroup, like BZ, does not generate a discussion or a reaction. And the in-program feedback to “input.mozilla.org” locks out nightly users.

    Here’s an example: FF 4 nightlies often beachball on my Mac, especially while starting the add-on Feedly and during daily FF updates. When I use Force Quit, Crash Reporter doesn’t catch the crashes, and the experience is too vague to report anywhere, particularly when I know no one will react to it in BZ.

    • This is a slight digression, but… I disagree a bit with Frank. While I do find bugzilla’s interface hard for non-coders to grok (largely because of a lot of jargon not defined well elsewhere), it is an essential tool to augment MDC’s html, css and javascript references. There is a huge wealth of wisdom in the comments of open and won’t-fix bugs.

      I defy anyone to figure out how to apply a -moz-linear-gradient to a full-screen background in a body tag without reading bug 552698

      What would be useful is a more structured way to make that wisdom more accessible.

    • the_dees
    • March 6th, 2011

    Another very big issue is the review thing of patches.

    Have a look at bug 230350. Patch was posted February 2009, the initial review happend five months later.
    I think the patch author lost interest between those two dates.

    There is a new patch now from another author (and the issue’s now called bug 578534) but for 6 months there’s not the slightest sign of a review.

    Even if the patch was reviewed it probably goes minus again and we’ve got to wait for another author to pop up.

    I realize the pople that suffer from this issue (including myself) are not the target group for the browser anymore. It would still have been nice to know your contributor’s work was acknowledged a tiny bit.

    Yes, this is angry, which I didn’t intend. I feel that it’s just not always easy when you’re part of the mission as a non code producing member.

  6. To make the issues clearer, first let’s imagine that all features were also entered as “bugs” to be fixed. Oh, wait, that already happens! Cool.

    Bug fixes represent a cost. The project has fixed resources. Every bug you fix takes resources away from another bug. So you are essentially arguing that three developers ought to be taken off of fixing other bugs to fix bugs. That’s not going solve anything.

    To break the cycle you need more developers, or tactics to prevent future bugs, or code that is easier for new developers to fix, or similar strategies.

    • And I agree, and I did address that in the blog, that there are priorities we have to work with and we only have so much time. I’m not expecting unlimited resources. I am expecting a little bit more of a targeted effort on fixing minor bugs and issues.

    • “So you are essentially arguing that three developers ought to be taken off of fixing other bugs to fix bugs. That’s not going solve anything.”

      Most bugs that get fixed are actually filed by the developers that fix them (or core developers who work with the people that fix them), so, in this oversimplified proposal, taking those three developers off a new feature could also mean a big reduction in the number of bugs that they file themselves.

      But it would be better to look at this on a higher level than bugs – the suggestion here is really spending more time fixing up existing things, and, as you say, with finite resources, that means less new features.

  7. Alexander Limi :
    Getting 10-20 papercuts fixed for every release is very good progress. I’m not sure where you got the idea that papercuts were a “Firefox 4″ projects

    From misunderstanding what you wrote about it, I think – sorry. Looking back, you did say it was a project starting for Firefox 4, and later highlighted a couple of specific areas that would be targeted for Firefox 4. So that indeed implied that it’s an ongoing thing, but I didn’t get that…

    I should definitely publish a blog post on the papercuts that were fixed in 4.0, though — there were many, and they very much make 4.0 into a more polished and less frustrating browser. Work Offline, anyone?

    The work offline thing wasn’t one that bothered me. Sync making the (modal and system-focus-grabbing) master password prompt appear at a random point a few minutes after starting the browser is extremely annoying (I know it’s by design, to ensure people don’t miss out on syncing which happened before, and obviously only affects the small proportion of people that use a master password and sync. But if you’re going to put up a modal dialog anyway, I’d rather it happened predictably on start up, rather than anything from a few seconds to a few minutes after start up).

    Anyway… before I started ranting about my new pet paper cut, my point was that there are new paper cuts coming in with new features, so the net gain is small than that 10-20, which feels to me like pretty slow progress relative to the total list.

    But you should definitely do a blog post – might help a bit to focus on the fixes rather than the remaining paper cuts.

  8. Great post Tyler!

    However, I fear yours will remain “a voice in the desert”. 😦

    • Frank Burleigh
    • March 8th, 2011

    Richard Milewski :
    This is a slight digression, but… I disagree a bit with Frank. While I do find bugzilla’s interface hard for non-coders to grok (largely because of a lot of jargon not defined well elsewhere), it is an essential tool to augment MDC’s html, css and javascript references. There is a huge wealth of wisdom in the comments of open and won’t-fix bugs.
    I defy anyone to figure out how to apply a -moz-linear-gradient to a full-screen background in a body tag without reading bug 552698
    What would be useful is a more structured way to make that wisdom more accessible.

    I agree. What I’d want from Mozilla is “a way in” that produces a conversation, like a support staff or even an intake function that’s a human. And let me be clear that it’s not just about a user getting attention — it’s about Mozilla not losing that last 10 percent of feedback, like the case I mentioned.

    • Actually I think Mozilla is losing a lot more than 10% of the feedback from non-technical users. Talking to Laura Mesa about what happens to the comments from the Feedback button in beta 4, I learned that Mozilla has millions of responses. It’s not at all clear to me that this firehose of user input is well utilized.

  1. September 7th, 2011

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: