Anatomy of a good bug report

Working on the .NET Framework was an interesting but often difficult time, especially when dealing with vague or incomprehensible bug reports.

Look before you file

Head to Bing, Google, official support sites and bug database if you have access to it (Microsoft Connect, Bugzilla for Firefox etc.) to see if others have run into this issue. Searching for the error message can yield good results but remove elements of the message specific to your project (e.g. class names, property names etc.)

The reasons for this are:

• you may have made a mistake (likely if the software has seen considerable use)
• behave differently to your expectations (the dreaded ‘by design’)
• already be fixed (in the next release and possibly as a patch)
• have an acceptable workaround (why accept a workaround?)

If you are running into problems with developer tools, also check out:

• Official forums (MSDN Forums & ASP.NET Forums for Microsoft tools)
• Unofficial respected sites (StackOverflow)

Consider how likely it is that you’ve discovered a bug given the complexity of what you are doing, how unusual it is and how mature the software is. Attempting something simple on an established piece of software likely means you’ve made a mistake or misunderstood the documentation.

A good bug report

So you decided to go ahead and report the bug (we all need better software after all).

The essence of a good bug report is having just the right amount of information to identify the bug without unnecessary detail.

Let’s break it down.

What happened

The most important element is describing what happened, and there are four major possibilities.

Error message

You told the software to do something, and it displayed an error message instead.

On a good day, the error message lets you know why it can’t do what you asked and lets you know what to do to make it work. Given you’re filing a bug report, it isn’t one of those days.

The error message is likely cryptic, doesn’t tell you how to get what you want or is just plain wrong. We’ll need that message in its entirety so:

• Copy Windows message box contents as text by pressing control-c when it’s in focus
• When there is a lot of text take a screen print (Print Screen on Windows, CmdShift3 on Mac OS X)
• Localized error messages may slow down support responses – switch the app back to their language first

Exceptions

When a piece of a program (called a method or function) can’t do what it claims, it throws an ‘exception’ back up to the piece that asked (called) it. This exception is like an error message but with enough technical detail that travels back up the program until something deals with it (known as a catch).

When you see an exception, then nothing wanted to deal with it.

You see an error message that contains the exception and possibly a list of program pieces that couldn’t deal with it. This is a stack trace, invaluable to the person investigating your report, so include it.

Developers seeing exceptions in their program need to determine if they should be catching that exception or whether it shouldn’t be occurring. Feel free to trim the stack trace to remove your methods if you need to.

Unexpected behaviour

The software should have done what you wanted but did something you didn’t expect instead.

Report what you thought should happen, what the software did instead, how this is different and the reason why you think it should be that way.

Other people may not agree with your change, and in the case of shared programming libraries or frameworks, a fix for you can become a break for others that rely on existing behaviour.

Terminated

The software just vanished from the screen without a trace. Crashed, terminated or unexpectedly quit and perhaps took some of your work with it :(

If you’re really unlucky, the software that crashed is your operating system. This could mean a blue screen of death (BSOD) on Windows, the grey screen on Mac OS X.

Often there are logs left behind that you can examine to identify what went wrong.

On Mac OS X, fire up Console from Application > Utilities and see what you can find. The iPhone, iPod and iPad do this quite often, and iTunes offers to send detailed information to Apple, who may share it with the application developer if it’s not their own.

Windows users should head to the Event Viewer and find the error and any additional messages that appear to relate to it and include these.

Steps to reproduce

Ideally, this is the minimum number of steps to reproduce the error every time. You want this to be as reliable as possible as companies have limited resources and won’t spend days trying to reproduce a low-impact bug.

This repro is a critical step in understanding the scope of the problem and its impact. If you can’t reproduce it in a few steps, there’s always the possibility you’re overlooking some other aspect that could be causing the problem — unexpected data or rogue code elsewhere!

For an application, this may be a data file or manual steps a user must manually perform via the user interface.

For a framework; a small self-contained project with just enough code to reproduce the failing scenario.

Environment

Bugs are often sensitive to their environment, and you should always include the version number of the software you are using as well as pertinent platform details,  including:

• What operating system, version & service pack
• What processor architecture (x86, x64, IA-64)
• What language & locale your machine is running in (e.g. US English (en-US), Brazil Portuguese (pr-BR))

In the specific case of Visual Studio and .NET bugs:

• What version of Visual Studio you are using (including any service packs)
• What processor architecture you are compiling for
• What language and compiler version you are using (e.g. C# compiling for 4.0)

You may need to include details for your desktop or developer machine and details of the server it is connecting to if any are involved.

What you’ve tried

Chances are you tried several things before filing a bug report. Let us know if you tried:

• Alternate routes through the user interface
• Entering data in a different format or order
• A different computer or environment

Letting them know this means they can avoid suggesting things you’ve already tried or waste time trying them too.

[)amien