What you get and not get in open source tools - Top 5 items

Open source movement is amazing. Linux, FireFox, MySQL, Open Office etc etc. They had really changed the way people started looking at software. They save around $5-8bn every year for customers, cumulatively. There are always doubts and anti-campaign on open source software. Here we try to give the top 5 items that you do get and that you do not get in open source. Take it or drop it.

You get this. 1. Many to most of open source software are free. Open source need not always be free, beware. So you do not have to worry about licensing, budgeting etc. You get it fee, the biggest saving!

You do not get this. 1. Usually the upgrades are not that quick. Except products like FireFox or OpenOffice, you do get stable releases very late. Their nightly builds need not be stable. So you may start feeling "Will I get this feature or not?".

You get this. 2. A community support. You can see content to help in blogs, forums. A mindset of not being alone.

You do not get this. 2. 90-95% of open source products do not have phone or live support. You need to be prepared to wait, before you get a solution, if you face problems.

You get this. 3. You get source code. How it is built - you can see it inside out.

You do not get this. 3. You do not get the supporting specs, design documents, test cases for that in most of the cases. What can you and I do with the source code Linux or MySQL? At least I do not even need the source code; I do not know about you.

You get this. 4. You may get many islands of tools. For anything, you get a tool. For bug tracking - you get one tool, for task tracking you get one tool. You name it, you get it.

You do not get this 4. You will not get integrated ones. They come with some catch. If you want an integrated task, test and bug tracking system, like QAMonitor, you may not get it.

You get this. 5. A complete stack to build an end to end application. A UI platform, an operating system, a backend programming language, a web server, a database, a report engine, and IDE etc. But each one may be provided by different communities.

You do not get this 5. Time tested products. There are products that you can name like FireFox, Linux etc. At the same time, many open source products are dead within a decade; those development communities vanished. End of the day, everyone wants to make money. If an open source product does not make money, it is going to be obsolete. They must either get donations or they must come up with paid support or premium versions etc. to make money. It is not a charity.

Before you evaluate a test management tool, please read this.

Testing requires a lot of coordination between business analysts team, development team and the testing team. When the product is live, testing team needs to coordinate with implementation and support team as well. Teams with size more than 5 members, cannot be so easily managed with spreadsheets and emails alone. There needs to be a centralized system that can control various activities in testing and fixing. There are many test management tools in the market. This section will touch the important features that one must look for, in a test management tool.

Manage Requirements

Testing, in short, is to ensure whether the product does what the customer wants. The starting point is what the customer wants and this is nothing but the set of requirements given by the customer. The first feature of every test management tool is to manage requirements as well. Without managing requirements, we cannot ensure what the customer wants and what we deliver. The tool must provide features to enter requirements, its short description and long description, type of requirements such as functional, performance, UI, legal etc. The tool must ensure that the proper version details of the requirements are maintained. The user must be in a position to attach any documents with a requirements. More importantly the requirements have to be managed in a hierarchical fashion - i.e. the parent requirement child requirement relationship must be maintained. It will be great if the tool provides some sort of tracking mechanism whether the requirement is reviewed or not. Also, if there are multiple products among which the requirements are shared, we must be in a position to tie requriements to one or more products.

Requirements and their variations may change from release to release. Hence the tool needs to provide a way to map a release with one or more requirements. Whenever a requirement is modified, there must be a history of what got changed and who modified at what time. This is essential from the compliance point of view. Many tools provide a mechanism to associate a priority with requirement for satisfying the same in subsequent releases.

Since many companies are already used to maintaining the requirements in documents and spreadsheets, it will be of great use, if the tool provides an import feature to directly import the requirements from a document or spreadsheet to the tool's repositiory. People feel comfortable in free hand typing rather than data entry in an application. This can greatly help users to accumulate past requirements in the projects. One requirement may be dependent on another requirement or a set of requirements. Some tools provide a mapping of dependencies between requirements.

Manage Test Scenarios and Cases

For every requirement, we may arrive at one or more test cases and scenarios. Using the tool, the testers will start documenting what to be tested and how to test the same. So the test case description, pre conditions, test steps, expected results, any data attachments to the test case, test case type such as smoke test, performance test, install test etc. are to be stroed in the tool. Here again, an import facility from a spreadsheet or document to the tool is greatly desired. This can minimize a whole lot of data entry time.

The test cases must also be managed hierarchically. The history of any modifications done to the test cases and scenarios must be stored automatically. The very important feature is to tie the requirements to the associated test cases. This is required to ensure traceability. This in turn ensures the functional coverage of the product. This also helps in doing an impact analysis in case of a change in requirements.

Manage Test Executions

The test execution happens in terms of test rounds. Some companies call this as test iteration or test cycle or test set. By principle, every roud must execute all available test cases. But practically, based on the changes made in the product, a specific set of test cases are planned and executed for any given test round. The tool must provide a feature to define a test round with its effective start and end dates and a facility to map what test cases will be part of that round. Also for tracking purpose, for each test case we must assocaite a tester who is supposed to execute the same.

Once the test cases are allocated to testers, the testers must execute the same and provide the status in the tool. When the test was started, when it ended, the status of pass or fail for each test step must be provided in the tool. This will clearly show the work in progress and the test manager will come to know where the project stands in testing. If a tester executes the same test multiple times, the status of each run must be stored as history.

Defect Tracking

The resultant effect of testing is the list of defects found on the product. We have already seen elaborately on problem reporting and tracking. The use of the tool should make bug tracking simple, easy and efficient. The system must provide data entry screen for bugs and its associated attachments such as screenshots. The tester must be able to do search of bugs to see if it has been already duplicated. The bug cycle across various status codes must be a configurable one. The bug tracking system must trigger automatic notifications between testing team and development team to ensure that the people are aware of the status of every single bug. Also the bugs must be mapped to test cases and test rounds so that we can do further analysis.

Data Analytics

The test management tool's efficiency comes out in its reporting and charting features. The stakeholders in the project require various reports from the system. It may be a document like vertical report or a spreadsheet like tabular report. The reporting engine of the tool must provide a variety of read-made reports and configurable reprots. For every report, the user must be able to use filters and sorting order. Also, the user must be in a position to choose a field to be or not to be displayed in report. This kind of a reporting feature is an essential one for any enterprise wide application.

Examples:

• List of bugs reported today
• Summary of bugs based on status
• List of tests executed in this week with status
• List of requirements not mapped to test cases

Also, in reports, we require simple reports that are a list of data presented to the user. Also we require trend reports. Trend reports provide a clue on how things changed over a period of time. For example, how many tests executed day-wise in the last 5 weeks or how many bugs were kept open in the last 10 days. Any report that provides data for a time period to show progress, is a good piece of information for the management.

More than the textual reports, the tool must provide a pictorial view of the data. These are charts. The tool must provide at least bar charts and pie charts with possibly 2D and 3D modes. These will be based on the data fed to the system by various users. If we plot all defects based on status, we can see a set of bars and we can easily find which bar is tall and take corrective actions. A graph hits the eyes better than text. A set of charts grouped together is called a dashboard. If the user can see a variety of graphs on a single panel, then it can help the user to quickly compare and contrast each parameter. This kind of analytics help management to take actions based on the statistics.

Also the tool must provide a feature to plot one parameter over another parameter. For example, we must be able to generate a graph that groups items based on defect severity; within each group we want further grouping based on status of the bugs.

Administration Features

Since the test management tool is usually an enterprise wide application, there must be clear boundaries within the tool to create various projects and group projects under various business units. For each project we must allocate users. These users must be further associated with their roles such as developer, tester, manager, business analyst etc. The purpose of any enterprise wide application is to bring consistency in usage, control in usage, visibility of information and to facilitate data analysis. Since the templates in the tool is used by all users, the consistency is established. The tool is used by all users in the organization that provides visibility to the management on what happens at this moment, within a few clicks.

The tool must provide access rights to various forms and fields to various user roles. This way, we can set that these users can enter the form, these users can modify this field, these users cannot see this form etc. This ensures that the users cannot misuse the tool and the built-in security of the tool must take care of this control mechanism.

Also, in many projects we may require additional data to be captured that are not provided by the tool in its templates. We must be able to add new fields on the fly to desired projects in desired screens only. This will make the tool more configurable and adaptable to any client and any project.

The tool must have provisions to send automatic email notifications upon certain fixed events and configurable events. For example, we must have a feature to automatically send email if a show stopper bug is raised. This will reduce the time latency in project coordination.

If you need a tool, that solves the above items, try QAMonitor.

Visit http://www.freeqamonitor.com.