Issue Tracking - Evaluating Systems

A Guide To Evaluating An Issue Tracking System

White Paper

By Stephen Blair, MetaQuest Software

Published: October, 2004

Abstract

Evaluating an issue tracking system requires that you understand how specific features, such as configurable workflow and customizable fields, relate to your requirements and your current issue tracking process. This article provides tips and guidelines for evaluating features, and explains how these features fit into a defect tracking process.

Security

Security is typically based on user accounts, user groups, and user group permissions. Look for an issue tracking system that provides different levels of permissions:

Field-level permissions control whether users can edit, or even view, fields.
Bug-level permissions control access to bugs by users.
Feature-level permissions control access to specific features, such as reporting or user account administration.
Project-level permissions determine which users can access which projects/databases.

There are two approaches to providing field-level security. One approach is to specify on a fieldby- field basis which user groups have permission to edit the field. Other tools, such as Census from MetaQuest Software, take a different approach, and allow you to define custom Web views of the bug database, where each custom view includes different fields. For each custom view, you can then specify which user groups that have permission to access the view.

For example, a customer view would include only the fields required to submit a new bug, while a developer view would include all fields. Customers would not be allowed to access the developer view, while developers would be allowed to access either view. Bug-level security is usually accomplished by limiting the available queries.

Evaluation Checklist:

User groups and user group permissions
Field-level security
Bug-level security
Feature-level security
Project-level security
View-level security (define multiple views of a project and then grant access permissions)

Reports and Metrics

An issue tracking system should allow you to quickly gather the information you need for staff meetings (bug listings and printouts), as well as provide more detailed metrics to help you make decisions.

Look for tools that provide both distribution metrics (metrics that break bugs down by category or classification: for example, issue age by severity) and trend metrics (metrics that show changes over time: for example, defect arrival rate). You may also require the ability to build your own custom reports, preferably using an industrystandard reporting tool such as Crystal Reports.

Evaluation Checklist:

Bug listing reports, summary and detailed
Distribution reports (cross-tab or chart)
Trend (time-based) reports
Web-based reporting
Library of predefined reports
Export reports and metrics
Support for industry-standard reporting tools such as Crystal Reports

Workflow

An issue tracking tool should automate and enforce your process for managing and resolving bugs. The tool should provide a configurable workflow that allows you to define the steps in your process and the order of the steps.

Workflow is typically modeled as a series of states, such as New, Fixed, and To be Verified. To support your process, you’ll need to be able to add and remove workflow states, as well as define the allowable transitions between states. For example, between Fixed and Closed you may want to add a required Verify Fix state, to ensure that an issue is never closed until after QA verifies the fix.

You should also be able to control which users are allowed to move bugs between states. For example, you may want only members of the QA group to be able to move bugs from Verify Fix to Closed. Granting workflow permissions allows you to enforce accountability and responsibility throughout the process.

While workflow is based primarily on states, you may also want your workflow to depend on other bug attributes, such as product or defect type, or even on user group membership. For example, you may need different workflows for software, hardware, and documentation issues. Finally, you should also expect an issue tracking tool to support different workflows for different projects.

Evaluation Checklist:

Default workflow
Configurable workflow: define custom states and transitions
Enforceable workflow: specify possible transitions and who has permission to make the changes
Separate workflows for different projects, different issue types

Version Control Integration

The starting point for any worthwhile version control support is the ability to link bugs to source code. Linking bugs to source code means developers can document a bug by linking to the related source files (which is especially important if the bug is deferred). These links will also allow you to track the work done on a bug.

Next, verify the version control operations that can you perform from inside the issue tracking system. You should be able to perform common operations such as checking files in and out, viewing file histories, and getting different versions of a file. The ability to check files in and out is key, particularly for Web-based issue tracking systems. Not all Web-based systems support the check in and check out operation, which limits the usefulness of the version control integration. Web-based check outs/ins help to improve development workflow and save time. If an issue tracking system has both a Windows client and a Web client, make sure you can perform all version control operations through both interfaces.

Evaluation Checklist:

Link bugs to source code
Remote, web-based access to Visual SourceSafe databases
Perform basic version control operations through the issue tracking system
Web-based check outs and check ins

Web-based Client

In addition to evaluating the usability (see Ease-of-Use on page 6) of a Web-based client, you should also evaluate the implementation of the client.

Architecture

Web-based clients can be implemented in a number of ways:

Internet Information Server (IIS) applications. An IIS application is a Visual Basic application that lives on a Web server and responds to requests from the browser.
- An IIS application uses a combination of ASP and HTML to present its user interface and uses compiled Visual Basic code to process requests and respond to events in the browser.
- IIS applications offer good performance. They also offer easy customization and branding, because the user interface is driven by HTML templates stored outside of the DLLs.
Internet Server API (ISAPI) applications. ISAPI was developed by Microsoft to address the resource problems of CGI. ISAPI uses Dynamic Link Libraries (DLLs) that are customized to return specific HTML. The IIS Web server loads the DLL the first time a request is received and then keeps the DLL in memory to service other requests. This minimizes the overhead associated with executing such applications many times.
- ISAPI applications offer good performance. However, they are often difficult to customize or brand, because all the HTML that makes up the user interface is emitted by a DLL.
Common Gateway Interface (CGI) applications. CGI is a standard way for a Web server to pass a Web user's request to an application and then forward the results back to the user.
- One of the problems of CGI is that every request from the browser starts a new process on the Web server. Consequently CGI applications are resource hungry, and can impact the performance of your Web server.
Active Server Pages (ASP) applications. ASP is a server-side scripting environment for IIS that combines HTML pages, script commands, and COM components.
- ASP performance can be hindered by the fact that ASP is driven by script languages that must be interpreted at run time.

Application standards

Does the Web-client behave like an application, or like a series of pages? Many Web applications are page-based, like a Web site, and are basically a series of HTML page generated on the server. Every interaction with the user requires a round-trip to the server and a page reload. Something as simple as running a predefined query, or switching to another tab, can require a page reload.

For example, suppose you log on to your issue tracking system to review the defects submitted yesterday. One looks familiar so you decide to do a keyword search for related or duplicate issues. In a page-based tool, this loads a new page, replacing the list of new defects you were looking at. After you type the text you want to search for and click Ok, another page is loaded, this time listing the defects that matched your query. And so it goes, back and forth between different pages as you search, scan lists, and read details.

Contrast this with the behavior of a Web application whose interface follows familiar Windows standards. In this type of Web client, the keyword search, the list of issues, and the issue view/edit form are all part of a unified interface. You don't load another page to define the query, running the query updates the list in place, and the defect you're viewing doesn't disappear just because you ran a query.

Customer communications

If you want to allow customers to submit bugs or to monitor issue status through a Web-based interface, check that the tool includes a Web-based client that customers can use. The customer submit interface should allow an unlimited number of customers to submit bugs.

Evaluation Checklist:

Web-based client implemented using best technology
Web-based client follows familiar desktop UI standards
Web-based client for customers

About the Author

Stephen Blair is a programmer-writer at MetaQuest Software, developers of the Census issue tracking system as well as solutions for e-support, application self-healing, and desktop management.

Web site: http://www.metaquest.com
Telephone: 514 341-9113
Fax: 514 341-4757
E-mail: info@metaquest.com

Previous Page