MARKETPLACE

Census: Web-based Defect/Issue/Bug Tracking Tool
Census is a customizable web-based tool for bug tracking, defect tracking, support calls, and helpdesk issues. Includes SourceSafe integration and configurable workflow. Windows XP look and feel on the Web. Bug tracking demo available online.

A Guide To Evaluating a Bug Tracking System

Next Page

White Paper

By Stephen Blair, MetaQuest Software

Published: October, 2004

Abstract

Evaluating a bug tracking system requires that you understand how specific features, such as configurable workflow and customizable fields, relate to your requirements and your current bug tracking process. This article provides tips and guidelines for evaluating features, and explains how these features fit into a defect tracking process.

Contents

• Introduction
• Adaptability
• Bug Change Histories
• Customizable Fields
• Notifications
• Ease-of-Use
• Security
• Reports and Metrics
• Workflow
• Version Control Integration
• Web-based Client

Introduction

Before you start evaluating bug tracking systems, make sure you identify your requirements for the system. Understanding these requirements will help you build a list of features that you can use to guide your evaluations.

To identify your bug tracking requirements, take a look at your current bug tracking process.

• What are the different roles and responsibilities of the people who will use the system?
• What is your workflow for managing and resolving bugs? Identify the steps in the process,and determine who is responsible for each step.
• What information do you need to track for each bug?
• What reports and metrics do you need?
• Do you need to provide different levels of access to different users?

Once you identify your requirements for the system, you can translate the requirements into a feature list Table 1 provides an example of a feature list that could be used to evaluate a bug tracking solution

Table 1. List of features to evaluate

The rest of this document provides tips and guidelines for evaluating these features. Hopefully these guidelines will help you choose a bug tracking system that meets your requirements. Remember, the purpose of a bug tracking system is to support your process, not to impose its own.

Adaptability

If you want to track more than just bugs, make sure the bug tracking system can be adapted to track other types of issues (such as support calls, test cases, or purchase orders). A system that is designed specifically for bug tracking can be hard to adapt, so look for a system that provides pre-built templates for tracking different issue types. Also, as you evaluate the various features of the tracking system, look for any hardwired "bug-tracking" terminology or functionality. Fields, queries, reports, notifications, workflow: everything should be adaptable. Otherwise you'll end up trying to fit a round peg into a square hole (for example, trying to fit trouble tickets into bug reports).

For example, if a tool provides a default bug-tracking workflow that you can't replace completely, you won't be able to implement workflows for other issue types.

Evaluation Checklist:

• Pre-built templates for different issue types (support calls, test cases,...)
• No hardwired bug-tracking terminology or functionality

Bug Change Histories

Consider whether you want to track changes to bugs. Some tools maintain change histories (audit trails), which allow you to trace who did what to an issue and when (for example, who raised the priority of a bug).

If the tool provides a change history, look at what information it records. For example, if you change the priority, does the change history simply say "Defect modified" (not so good) or "Priority modified" (better), or does it say "Priority changed to High" (best)? Some tools offer a sort of compromise, where the tool adds "Defect modified" to the revision history and allows the user to add a comment such as "Changed priority to High after speaking to customer".

A related feature is something called timestamping. In tools that support timestamping, information entered into text fields such as Description, Resolution, and Workaround are time and date stamped with the User ID of the individual who entered the text. This time stamping provides a history of the changes to important text fields.

Evaluation Checklist:

• Changes to issues are automatically logged to revision history
• Change history indicates exactly what was changed (for example, Priority changed from Medium to High)
• Change history can be enabled/disabled on a field-by-field basis
• Changes to text fields be timestamped

Customizable Fields

You need to be able to customize the fields used to collect defect information. Otherwise, you’ll end up collecting only the information the tool thinks you should track, and using the tool’s terminology instead of your own.

Fields

Most bug tracking tools come with a predefined set of fields. You probably can use some of these fields as is, and others by simply changing field labels and drop-down list values to match your terminology. For example, if you already classify defects by severity and priority, and the tool includes predefined Severity and Priority fields, you just need to change the severity and priority levels to match those you already use. If you cannot reuse a predefined field, you should be able to delete it, or a least hide it so that users never see it.

To collect all the information you want to track, you may also need to add new fields. The tool should be flexible enough to allow you to put new fields anywhere you want. If you cannot rearrange fields, or if all new fields are put at the bottom of a page or on a separate tab, the usability of even the simplest "bug form" will suffer.

Field relationships

Consider also whether you want to define field relationships. Field relationships can help you keep drop-down lists short and uncluttered. For example, by creating a relationship between the Product and Version fields, you can display only the version numbers of the selected product (instead of displaying all version numbers of all products). Similarly, for software bugs, you can display only the software components and filter out documentation and hardware components.

Some tools also allow you to select a value from a drop-down list based on the values selected from other lists. For example, you could automatically assign bugs based on the selected product and component.

Custom views and field sets

Based on the user's role and the task at hand, do you want to display different sets of fields? The ability to display custom views allows you to reduce UI clutter and present only the fields necessary for the task at hand. For example, a custom view for logging new bugs wouldn't need to include the fields used by QA and Engineering to add information during the resolution process.

Evaluation Checklist:

• Change field labels and drop-down lists to match your company’s terminology
• Custom fields - add and remove fields as required
• Control over which fields are required and which are optional
• Customizable field layout
• Add, remove, and rename tabs
• Define field relationships
• Create custom views

Notifications

Notifications are really a workflow feature, because they help you track and manage bugs. For example, notifications can keep team members informed about important changes to bugs, such as changes in priority or addition of new information (such as how to reproduce the problem). Notifications can also help automate the defect management process. For example, you can notify development managers when new bugs are submitted, and developers when bugs are assigned to them.

When evaluating a bug tracking tool, look closely at what kind of notifications you can send. Most bug tracking tools support a fixed, standard set of notifications:

• Bug added
• Bug edited
• Bug status changed
• Bug assigned

Obviously there’s a potential for a spam-like deluge of notifications, especially if you enable "Bug edited" notifications. So look for the ability to define notifications for specific changes to specific fields. For example, can you send a notification when someone changes the priority of a bug? Or when your most important customer reports a "must-fix" bug?

If you want to allow customers to submit bugs directly into the system, check if the tool allows you send notifications to the customer (e.g., send the customer a confirmation, notify customer when bug is fixed).

Finally, look at what information goes into a notification message, and how easy (or hard) it is to customize the contents. Some tools offer a predefined list of possible content, such as summary, full details, change history, or all of the above.

Evaluation Checklist:

• Automatic e-mail notifications for comment events (new bugs, changes to bugs, status changes, bug assignments)
• Support for custom events (send notifications for specific changes to specific fields by specific users, for example, notify the developer when a tester modifies the Repro Steps fields)
• Send notifications to users or customers (where customers don’t have a user account for the tool)
• Easily customizable notification message contents

Ease-of-Use

Is the interface consistent and easy to navigate? Web-based applications can often be difficult to navigate, especially if they behave more like a sequence of HTML pages than an application. Some Web applications are constantly reloading pages because responding to user input requires a round trip between the Web server and the user's browser. This constant reloading of pages can make an application difficult to work with, because the application doesn't behave the way users expect an application to behave - like a Windows application. When a Web application behaves like a Windows application, it’s easier to adopt into your development process. There's less learning curve, and less resistance to change.

Another important consideration is the user interface text; is it user-friendly, simple, and clear?

Finally, and maybe most important, is the interface easy to use? Try evaluating how easy is to perform common tasks, such as logging a new bug or searching for keywords in common text fields. Is it obvious how to do these tasks? And how many mouse clicks does it take? Frequently-used features should never be more than a few mouse clicks away.

Evaluation Checklist:

• Interface is easy to navigate
• Easy to enter new bugs
• Easy access common features (requires only a few mouse clicks)
• User interface text and messages are easy to understand

Next Page

Download a Free Guide to Evaluating a Bug Tracking System

Triage eSupport Solution Triage streamlines the process of resolving technical support problems, reduces the total cost of support, and enhances the support experience.

Simple to install and highly customizable desktop management. Purchase only the modules you need and run from a central MMC console. Modules include, Inventory, Distribution, Metering, HelpDesk, Diagnostics and Remote Control.

Plug and Browse
Plug and Browse is a network switcher that allows you to create network profiles for different locations.

MARKETPLACE

Census: Web-based Defect/Issue/Bug Tracking Tool
Census is a customizable web-based tool for bug tracking, defect tracking, support calls, and helpdesk issues. Includes SourceSafe integration and configurable workflow. Windows XP look and feel on the Web. Bug tracking demo available online.