Anatomy of a good bug report
Working on the .NET Framework was an interesting but often difficult time especially when dealing with vague or incomprehensible bug reports.
Look before you file
Head to Bing, Google, official support sites and bug database if you have access to it (Microsoft Connect, Apple Radar, Bugzilla for Firefox etc.) to see if others have run into this issue. Searching for the error message can yield good results but remove any elements of the message specific to your project (e.g. class names, property names etc.)
Why accept a workaround?
Do not underestimate the value of a solid workaround.
A patch is significant effort for both sides and for re-distributable components like .NET Framework getting the components to the right place can be tough. You’ll need to deploy to your team, build, test and staging environments and, if you distribute software, to all your customers.
This can be especially painful when strictly controlled environments are shared with others and ops teams are hesitant about putting patches or hotfixes into production. Factor in processor architectures, operating systems and language support and weigh it up.
The reasons for this are:
- you may have made a mistake (likely if the software has seen considerable use)
- behave differently to your expectations (the dreaded ‘by design’)
- already be fixed (in the next release and possibly as a patch)
- have an acceptable workaround (why accept a workaround?)
If you are running into problems with developer tools also check out:
- Official forums (MSDN Forums & ASP.NET Forums for Microsoft tools)
- Unofficial respected sites (StackOverflow)
Consider how likely it is you’ve discovered a bug given the complexity of what you are doing, how unusual it is and how mature the software is. Attempting something simple on an established piece of software likely means you’ve made a mistake or misunderstood the documentation.
A good bug report
So you decided to go ahead and report the bug (we all want better software after all).
The essence of a good bug report is having just the right amount of information to identify the bug without any unnecessary detail.
Let’s break it down.
The most important element is describing what happened and there are four major possibilities.
You told the software to do something and it displayed an error message instead.
On a good day the error message lets you know why it can’t do what you asked and lets you know what to do to make it work. Given you’re filing a bug report it isn’t one of those days.
The error message is likely cryptic, doesn’t tell you how to get what you want or is just plain wrong. We’ll need that message in its entirety so:
- Copy Windows message box contents as text by pressing control-c when it’s in focus
- When there is a lot of text take a screen print (Print Screen on Windows, CmdShift3 on Mac OS X)
- Localized error messages may slow down support responses – switch the app back to their language first
When a piece of a program (called a method or function) can’t do what it claims it throws an ‘exception’ back up to the piece that asked (called) it. This exception is like an error message but with enough lot of technical detail that travels back up the program until something deals with it (known as a catch).
When you see an exception then nothing wanted to deal with it.
You will see an error message that contains the exception and possibly a list of program pieces that couldn’t deal with it. This is called a stack trace and is invaluable to the person investigating your report so include it.
Developers seeing exceptions in their own program need to determine if they should be catching that exception or whether it shouldn’t be occurring. Feel free to trim the stack trace so your own methods aren’t included but too much is better than too little.
The software should have done what you wanted but did something you didn’t expect instead.
Report what you thought should happen, what the software did instead, how this is different and the reason why you think it should be that way.
Other people may not agree with your change and in the case of shared programming libraries or frameworks a fix for you can become a break for others that rely on existing behavior.
The software just vanished from the screen without a trace. Crashed, terminated or unexpectedly quit and perhaps took some of your work with it :(
If you’re really unlucky the software that crashed is your operating system. The blue screen of death (BSOD) on Windows, the grey screen on Mac OS X.
Often there are logs left behind you can examine to identify what went wrong.
On Mac OS X fire up Console from Application > Utilities and see what you can find. The iPhone, iPod and iPad do this quite often and iTunes will likely offer to send detailed information to Apple who will hopefully share it with the application developer if it’s not their own.
Windows users will want to head to the Event Viewer and find the error and any additional messages that appear to relate to it and included these too.
Steps to reproduce
Ideally the minimal number of steps to reproduce the error every time. You want this to be reliable as possible as companies have limited resources too and won’t spend days trying to reproduce a low-impact bug.
This can be a very important step in understanding the scope of the problem and it’s impact. If you can’t reproduce it in a minimal number of steps there’s always the possibility you’re overlooking some other aspect that could be causing the problem – bad data or rogue code elsewhere!
For an application this may be a data file or a number of manual steps a user must manually perform via the user interface.
For a framework a small self-contained project file with a minimal amount of code to reproduce the failing scenario.
Again here, if you can make the steps clear and ideally in the language of the company producing the software your bug report is not going to hit extra delays.
Bugs are often sensitive to their environment and you should always include the version number of the software you are using as well as pertinent platform details, including:
- What operating system, version & service pack
- What processor architecture (x86, x64, IA-64)
- What language & locale your machine is running in (e.g. US English (en-US), Brazil Portuguese (pr-BR))
In the specific case of Visual Studio and .NET bugs:
- What version of Visual Studio you are using (including any service packs)
- What processor architecture you are compiling for
- What language and compiler version you are using (e.g. C# compiling for 4.0)
You may need to include details for your desktop or developer machine and details of the server it is connecting to if any are involved.
What you’ve tried
Chances are you tried several things before filing a bug report. Let us know if you tried:
- Alternate routes through the user interface
- Entering data in a different format or order
- A different computer or environment
Letting them know this means they can avoid suggesting things you’ve already tried or waste time trying them too.