Sunday, December 4, 2011

Before you evaluate a test management tool, please read this.

Testing requires a lot of coordination between business analysts team, development team and the testing team. When the product is live, testing team needs to coordinate with implementation and support team as well. Teams with size more than 5 members, cannot be so easily managed with spreadsheets and emails alone. There needs to be a centralized system that can control various activities in testing and fixing. There are many test management tools in the market. This section will touch the important features that one must look for, in a test management tool.

Manage Requirements

Testing, in short, is to ensure whether the product does what the customer wants. The starting point is what the customer wants and this is nothing but the set of requirements given by the customer. The first feature of every test management tool is to manage requirements as well. Without managing requirements, we cannot ensure what the customer wants and what we deliver. The tool must provide features to enter requirements, its short description and long description, type of requirements such as functional, performance, UI, legal etc. The tool must ensure that the proper version details of the requirements are maintained. The user must be in a position to attach any documents with a requirements. More importantly the requirements have to be managed in a hierarchical fashion - i.e. the parent requirement child requirement relationship must be maintained. It will be great if the tool provides some sort of tracking mechanism whether the requirement is reviewed or not. Also, if there are multiple products among which the requirements are shared, we must be in a position to tie requriements to one or more products.

Requirements and their variations may change from release to release. Hence the tool needs to provide a way to map a release with one or more requirements. Whenever a requirement is modified, there must be a history of what got changed and who modified at what time. This is essential from the compliance point of view. Many tools provide a mechanism to associate a priority with requirement for satisfying the same in subsequent releases.

Since many companies are already used to maintaining the requirements in documents and spreadsheets, it will be of great use, if the tool provides an import feature to directly import the requirements from a document or spreadsheet to the tool's repositiory. People feel comfortable in free hand typing rather than data entry in an application. This can greatly help users to accumulate past requirements in the projects. One requirement may be dependent on another requirement or a set of requirements. Some tools provide a mapping of dependencies between requirements.

Manage Test Scenarios and Cases

For every requirement, we may arrive at one or more test cases and scenarios. Using the tool, the testers will start documenting what to be tested and how to test the same. So the test case description, pre conditions, test steps, expected results, any data attachments to the test case, test case type such as smoke test, performance test, install test etc. are to be stroed in the tool. Here again, an import facility from a spreadsheet or document to the tool is greatly desired. This can minimize a whole lot of data entry time.

The test cases must also be managed hierarchically. The history of any modifications done to the test cases and scenarios must be stored automatically. The very important feature is to tie the requirements to the associated test cases. This is required to ensure traceability. This in turn ensures the functional coverage of the product. This also helps in doing an impact analysis in case of a change in requirements.

Manage Test Executions

The test execution happens in terms of test rounds. Some companies call this as test iteration or test cycle or test set. By principle, every roud must execute all available test cases. But practically, based on the changes made in the product, a specific set of test cases are planned and executed for any given test round. The tool must provide a feature to define a test round with its effective start and end dates and a facility to map what test cases will be part of that round. Also for tracking purpose, for each test case we must assocaite a tester who is supposed to execute the same.

Once the test cases are allocated to testers, the testers must execute the same and provide the status in the tool. When the test was started, when it ended, the status of pass or fail for each test step must be provided in the tool. This will clearly show the work in progress and the test manager will come to know where the project stands in testing. If a tester executes the same test multiple times, the status of each run must be stored as history.

Defect Tracking

The resultant effect of testing is the list of defects found on the product. We have already seen elaborately on problem reporting and tracking. The use of the tool should make bug tracking simple, easy and efficient. The system must provide data entry screen for bugs and its associated attachments such as screenshots. The tester must be able to do search of bugs to see if it has been already duplicated. The bug cycle across various status codes must be a configurable one. The bug tracking system must trigger automatic notifications between testing team and development team to ensure that the people are aware of the status of every single bug. Also the bugs must be mapped to test cases and test rounds so that we can do further analysis.

Data Analytics

The test management tool's efficiency comes out in its reporting and charting features. The stakeholders in the project require various reports from the system. It may be a document like vertical report or a spreadsheet like tabular report. The reporting engine of the tool must provide a variety of read-made reports and configurable reprots. For every report, the user must be able to use filters and sorting order. Also, the user must be in a position to choose a field to be or not to be displayed in report. This kind of a reporting feature is an essential one for any enterprise wide application.


  • List of bugs reported today
  • Summary of bugs based on status
  • List of tests executed in this week with status
  • List of requirements not mapped to test cases

Also, in reports, we require simple reports that are a list of data presented to the user. Also we require trend reports. Trend reports provide a clue on how things changed over a period of time. For example, how many tests executed day-wise in the last 5 weeks or how many bugs were kept open in the last 10 days. Any report that provides data for a time period to show progress, is a good piece of information for the management.

More than the textual reports, the tool must provide a pictorial view of the data. These are charts. The tool must provide at least bar charts and pie charts with possibly 2D and 3D modes. These will be based on the data fed to the system by various users. If we plot all defects based on status, we can see a set of bars and we can easily find which bar is tall and take corrective actions. A graph hits the eyes better than text. A set of charts grouped together is called a dashboard. If the user can see a variety of graphs on a single panel, then it can help the user to quickly compare and contrast each parameter. This kind of analytics help management to take actions based on the statistics.

Also the tool must provide a feature to plot one parameter over another parameter. For example, we must be able to generate a graph that groups items based on defect severity; within each group we want further grouping based on status of the bugs.

Administration Features

Since the test management tool is usually an enterprise wide application, there must be clear boundaries within the tool to create various projects and group projects under various business units. For each project we must allocate users. These users must be further associated with their roles such as developer, tester, manager, business analyst etc. The purpose of any enterprise wide application is to bring consistency in usage, control in usage, visibility of information and to facilitate data analysis. Since the templates in the tool is used by all users, the consistency is established. The tool is used by all users in the organization that provides visibility to the management on what happens at this moment, within a few clicks.

The tool must provide access rights to various forms and fields to various user roles. This way, we can set that these users can enter the form, these users can modify this field, these users cannot see this form etc. This ensures that the users cannot misuse the tool and the built-in security of the tool must take care of this control mechanism.

Also, in many projects we may require additional data to be captured that are not provided by the tool in its templates. We must be able to add new fields on the fly to desired projects in desired screens only. This will make the tool more configurable and adaptable to any client and any project.

The tool must have provisions to send automatic email notifications upon certain fixed events and configurable events. For example, we must have a feature to automatically send email if a show stopper bug is raised. This will reduce the time latency in project coordination.

If you need a tool, that solves the above items, try QAMonitor.



  1. How QAMonitor is different from QC or other Test Management tool?

  2. How about data security in QAMonitor?

  3. How many users I can create using QAMonitor?

  4. What is the technical specs of QAMonitor?

  5. Security:

    1. QAMonitor is hosted on Amazon cloud that is fully protected by firewall.
    2. Users are contained within enterprise. Each enterprise database and webapp is different.
    3. Users are controlled by access rights across all screens.

    Technical Specs of QAMonitor:

    1. UI is done using ExtJS, jsp
    2. Business rules are done in Java
    3. Database is MYSQL
    4. Webserver is Tomcat
    5. Jasper reports and Chartdirector are used for dashboards.