What you get and not get in open source tools - Top 5 items

Open source movement is amazing. Linux, FireFox, MySQL, Open Office etc etc. They had really changed the way people started looking at software. They save around $5-8bn every year for customers, cumulatively. There are always doubts and anti-campaign on open source software. Here we try to give the top 5 items that you do get and that you do not get in open source. Take it or drop it.

You get this. 1. Many to most of open source software are free. Open source need not always be free, beware. So you do not have to worry about licensing, budgeting etc. You get it fee, the biggest saving!

You do not get this. 1. Usually the upgrades are not that quick. Except products like FireFox or OpenOffice, you do get stable releases very late. Their nightly builds need not be stable. So you may start feeling "Will I get this feature or not?".

You get this. 2. A community support. You can see content to help in blogs, forums. A mindset of not being alone.

You do not get this. 2. 90-95% of open source products do not have phone or live support. You need to be prepared to wait, before you get a solution, if you face problems.

You get this. 3. You get source code. How it is built - you can see it inside out.

You do not get this. 3. You do not get the supporting specs, design documents, test cases for that in most of the cases. What can you and I do with the source code Linux or MySQL? At least I do not even need the source code; I do not know about you.

You get this. 4. You may get many islands of tools. For anything, you get a tool. For bug tracking - you get one tool, for task tracking you get one tool. You name it, you get it.

You do not get this 4. You will not get integrated ones. They come with some catch. If you want an integrated task, test and bug tracking system, like QAMonitor, you may not get it.

You get this. 5. A complete stack to build an end to end application. A UI platform, an operating system, a backend programming language, a web server, a database, a report engine, and IDE etc. But each one may be provided by different communities.

You do not get this 5. Time tested products. There are products that you can name like FireFox, Linux etc. At the same time, many open source products are dead within a decade; those development communities vanished. End of the day, everyone wants to make money. If an open source product does not make money, it is going to be obsolete. They must either get donations or they must come up with paid support or premium versions etc. to make money. It is not a charity.

Before you evaluate a test management tool, please read this.

Testing requires a lot of coordination between business analysts team, development team and the testing team. When the product is live, testing team needs to coordinate with implementation and support team as well. Teams with size more than 5 members, cannot be so easily managed with spreadsheets and emails alone. There needs to be a centralized system that can control various activities in testing and fixing. There are many test management tools in the market. This section will touch the important features that one must look for, in a test management tool.

Manage Requirements

Testing, in short, is to ensure whether the product does what the customer wants. The starting point is what the customer wants and this is nothing but the set of requirements given by the customer. The first feature of every test management tool is to manage requirements as well. Without managing requirements, we cannot ensure what the customer wants and what we deliver. The tool must provide features to enter requirements, its short description and long description, type of requirements such as functional, performance, UI, legal etc. The tool must ensure that the proper version details of the requirements are maintained. The user must be in a position to attach any documents with a requirements. More importantly the requirements have to be managed in a hierarchical fashion - i.e. the parent requirement child requirement relationship must be maintained. It will be great if the tool provides some sort of tracking mechanism whether the requirement is reviewed or not. Also, if there are multiple products among which the requirements are shared, we must be in a position to tie requriements to one or more products.

Requirements and their variations may change from release to release. Hence the tool needs to provide a way to map a release with one or more requirements. Whenever a requirement is modified, there must be a history of what got changed and who modified at what time. This is essential from the compliance point of view. Many tools provide a mechanism to associate a priority with requirement for satisfying the same in subsequent releases.

Since many companies are already used to maintaining the requirements in documents and spreadsheets, it will be of great use, if the tool provides an import feature to directly import the requirements from a document or spreadsheet to the tool's repositiory. People feel comfortable in free hand typing rather than data entry in an application. This can greatly help users to accumulate past requirements in the projects. One requirement may be dependent on another requirement or a set of requirements. Some tools provide a mapping of dependencies between requirements.

Manage Test Scenarios and Cases

For every requirement, we may arrive at one or more test cases and scenarios. Using the tool, the testers will start documenting what to be tested and how to test the same. So the test case description, pre conditions, test steps, expected results, any data attachments to the test case, test case type such as smoke test, performance test, install test etc. are to be stroed in the tool. Here again, an import facility from a spreadsheet or document to the tool is greatly desired. This can minimize a whole lot of data entry time.

The test cases must also be managed hierarchically. The history of any modifications done to the test cases and scenarios must be stored automatically. The very important feature is to tie the requirements to the associated test cases. This is required to ensure traceability. This in turn ensures the functional coverage of the product. This also helps in doing an impact analysis in case of a change in requirements.

Manage Test Executions

The test execution happens in terms of test rounds. Some companies call this as test iteration or test cycle or test set. By principle, every roud must execute all available test cases. But practically, based on the changes made in the product, a specific set of test cases are planned and executed for any given test round. The tool must provide a feature to define a test round with its effective start and end dates and a facility to map what test cases will be part of that round. Also for tracking purpose, for each test case we must assocaite a tester who is supposed to execute the same.

Once the test cases are allocated to testers, the testers must execute the same and provide the status in the tool. When the test was started, when it ended, the status of pass or fail for each test step must be provided in the tool. This will clearly show the work in progress and the test manager will come to know where the project stands in testing. If a tester executes the same test multiple times, the status of each run must be stored as history.

Defect Tracking

The resultant effect of testing is the list of defects found on the product. We have already seen elaborately on problem reporting and tracking. The use of the tool should make bug tracking simple, easy and efficient. The system must provide data entry screen for bugs and its associated attachments such as screenshots. The tester must be able to do search of bugs to see if it has been already duplicated. The bug cycle across various status codes must be a configurable one. The bug tracking system must trigger automatic notifications between testing team and development team to ensure that the people are aware of the status of every single bug. Also the bugs must be mapped to test cases and test rounds so that we can do further analysis.

Data Analytics

The test management tool's efficiency comes out in its reporting and charting features. The stakeholders in the project require various reports from the system. It may be a document like vertical report or a spreadsheet like tabular report. The reporting engine of the tool must provide a variety of read-made reports and configurable reprots. For every report, the user must be able to use filters and sorting order. Also, the user must be in a position to choose a field to be or not to be displayed in report. This kind of a reporting feature is an essential one for any enterprise wide application.

Examples:

• List of bugs reported today
• Summary of bugs based on status
• List of tests executed in this week with status
• List of requirements not mapped to test cases

Also, in reports, we require simple reports that are a list of data presented to the user. Also we require trend reports. Trend reports provide a clue on how things changed over a period of time. For example, how many tests executed day-wise in the last 5 weeks or how many bugs were kept open in the last 10 days. Any report that provides data for a time period to show progress, is a good piece of information for the management.

More than the textual reports, the tool must provide a pictorial view of the data. These are charts. The tool must provide at least bar charts and pie charts with possibly 2D and 3D modes. These will be based on the data fed to the system by various users. If we plot all defects based on status, we can see a set of bars and we can easily find which bar is tall and take corrective actions. A graph hits the eyes better than text. A set of charts grouped together is called a dashboard. If the user can see a variety of graphs on a single panel, then it can help the user to quickly compare and contrast each parameter. This kind of analytics help management to take actions based on the statistics.

Also the tool must provide a feature to plot one parameter over another parameter. For example, we must be able to generate a graph that groups items based on defect severity; within each group we want further grouping based on status of the bugs.

Administration Features

Since the test management tool is usually an enterprise wide application, there must be clear boundaries within the tool to create various projects and group projects under various business units. For each project we must allocate users. These users must be further associated with their roles such as developer, tester, manager, business analyst etc. The purpose of any enterprise wide application is to bring consistency in usage, control in usage, visibility of information and to facilitate data analysis. Since the templates in the tool is used by all users, the consistency is established. The tool is used by all users in the organization that provides visibility to the management on what happens at this moment, within a few clicks.

The tool must provide access rights to various forms and fields to various user roles. This way, we can set that these users can enter the form, these users can modify this field, these users cannot see this form etc. This ensures that the users cannot misuse the tool and the built-in security of the tool must take care of this control mechanism.

Also, in many projects we may require additional data to be captured that are not provided by the tool in its templates. We must be able to add new fields on the fly to desired projects in desired screens only. This will make the tool more configurable and adaptable to any client and any project.

The tool must have provisions to send automatic email notifications upon certain fixed events and configurable events. For example, we must have a feature to automatically send email if a show stopper bug is raised. This will reduce the time latency in project coordination.

If you need a tool, that solves the above items, try QAMonitor.

Visit http://www.freeqamonitor.com.

A Test Manager's Wish List - Top 5 Items

When we talk to a test manager, first he/she complains; after some point, their wish list starts coming out. The test manager wants something that is not there now, but can be implemented across teams; yet the test manager feels that it may not happen in near future. Looks like never-ending chase!

1. One single system for all my test assets. But branded tools are costly; my management rules those out when I give a quote. They do not understand my problem of dealing with so many spreadsheets and emails!

What is the real problem here? It takes time to get even the simplest metrics. Also it always gives doubt whether the data in the spreadsheets are 100% correct or not. The recurring pain of manually reviewing the same and making charts out of it, drains my energy at EOD; oh,,I have a con-call tonight to attend and present these metrics once again!

2. Status of test execution and bug fixes must be fully visible to all. Every day I need to collate all status reports, (btw, I have become a super user of copy-paste technique!). So many status mails and sugar-coated wordings by my test leads to protect their team members, sluggish fix updates by developers...I feel I do not have full control on the testing status. If I have a system that tells me, what tests are executed till the last minute, I will be the happiest person in this world.

3. Ready charts and reports with just 1 click. I know my boss appreciates one specific report and chart; if that reaches his/her inbox every night, I am assured of a great appraisal. The test management system that I have does not have that report; hence I export data, massage it and make the report and chart. I am good type-setter as well.

4. Automatic Alerts when something goes very bad. I do too many things. I cannot keep track of everything. But my blackberry or mobile alerts me when a new alert email comes. If build is not deployed or a show stopper happens or 2 testers do not update execution status for last 2 hours, I must be notified. Can someone do that for me?

5. Traceability of tests and requirements. Holy grail of testing. I must know coverage at any given point of time. People write 1000 test cases, but my business team says the test coverage is poor. Again I need some magic medicine.

There is a clear solution for this. Softsmith heard these lines so often from so many people and developed QAMonitor. This tool is aimed at satisfying these top 5 wish list items for FREE to micro organizations with less than 25 employees and at a throw away price to small and medium enterprises.

Please visit http://www.freeqamonitor.com.

Simple steps to improve testing process

Every time, we meet customers, we do hear that every stakeholder in the project conveys the message "there are gaps in our process", "we do not follow processes seriously", etc. etc. What surprises me is that, their senior management listens to that with a smile! If there is a process problem in crediting their salary, will they tell the same reason?

Here are some simple steps to improve testing process, if you feel that you are not happy with your current process.

1. Foremost complaint is "We do not get proper requirements specs from any teams!". In this case, let the testers go one extra mile. Someone has to bell the cat. Let the testing team simply write use cases or features list after dev team gives demo or build; let the testing team ask for green signal on the piece of specs that they wrote. Put the blame on the business analyst and project manager. Someone will review and get back to you. Take it from me. If you do this a few times, you can see a positive attitude change in your dev team. As a good side effect, a few of your testers can easily grow as business analyst.

2. If your testing team members do assume a lot on specs and write bad tests or write unwanted tests, do this. For every change request or build release instructions, ask every tester to come up with just 1 clarification needed. Reward a good one. This will motivate the team. If testers stop asking clarifications, there is something wrong, for sure. Give a target, every requirement or CR must have at least 1 clarification needed.

3. Testers do not write test cases. They try only adhoc tests. If this is your problem, try this medicine. Ask every tester to first write "what to test" - kind of test conditions or test scenarios. For 1 week or 2 weeks, do not ask them to write test cases. Make them to write only test conditions. This must not take much time. You review the same and point out where they make mistakes. This will give a jump start for testers who do not write test documentation. Once they get to the normalcy of writing test conditions, load them with test cases as well.

4. Bug reports do not have proper information. If this is the problem, here is a simple solution. Ask every tester to talk about the bugs they found, the next day and not on the same day. They might have forgotten many details about the bug; then you ask logical questions and make them understand what they miss in bug reports. They will realize and change.

5. Because of so many problems, test leads do not present proper status. In this case, spreadsheets and emails will not solve the problem. Use a test management tool. Some of the tools are HP's Quality Center, Rational IBM Test Manager or Softsmith's QAMonitor.

Imagine, you send good status to your management with charts like this?

For more information on this tool, QAMonitor, please visit www.freeqamonitor.com.

QAMonitor - Oct 2011 Upgraded Features

We are happy to communicate that QAMonitor has now more features. All these features are requested by various users from various companies. Thanks to all of them who provide value adding feedback to us.

1. Issues can now be saved as draft. Later, a senior member may review the defect and approve. Till then it is not counted as part of regular defects for the dev team.
2. You can choose to generate requirement id by yourself in manual entry mode or you can set that to auto-generate id mode.
3. You can choose to generate test case id by yourself in manual entry mode or you can set that to auto-generate id mode.
4. All new icons in many pages. These enhance the look and feel of the product.
5. Mail settings has now 2 options. You can use your company's SMTP server or Amazon AWS SES mail settings.
6. Login panel will appear if session times out
7. Forgot password screen will have captcha feature enabled.
8. Users can be removed from project if the user has not done any transactions in the project and nothing is assigned to the user
9. Groups can be removed from project if the group has no users assigned to it
10. About 20 bug fixes to enhance usability
    

New Features - 03-July-2011

Dear QAMonitor Users,

We are glad to announce that we had upgraded the product with amazing new set of features. Thanks to all those who provided value-adding feedback for feature enhancements. Here is the list for your reference.

1. QAMonitor is now compatible with Microsoft IE 7+ and Firefox 3.6+
2. A simple text based metrics board soon after you login.
   
3. Cut Copy Paste of tasks within the task grid
4. Frameview to see requirements with full details (like reading layout in MS Outlook)
   
5. Frameview to see test cases with full details
6. Frameview to see defects with full details
7. Rich edit text box with format icons for Requirements and Defects
8. Create new defect window right from test execution window
   
9. Simplified defect cycle and email notification settings
10. Many new reports
11. Minor bug fixes

We pledge to ensure the product meets your needs and we continuously improve the usability of the product.

Regards,

QAMonitor Team.

How adaptive is QAMonitor?

Dinosaurs did not adapt to the environment and hence they perished. Cockroaches did that trick well and still living. Same for any tool.

Recently we received a set of questions from customers - but most of those questions followed the same tone - "How can I make the tool to adapt to my project?". I am used to some names and the tool does not have those, what to do?

The UI must show the exact labels that the users are familiar with. Else, they may feel that the tool is not capturing the data they want to.

Here is the simple 10 minutes exercise you need to do.

1. What fields you want to capture in requirements, test case, tasks and defects? First list them out.
2. Login as eadmin to your enterprise, Go to settings, go to System Defined Fields.
3. Check whether the list of fields you have is there in the respective module such as defects.
4. For example, you need a field called "Importance" and there is a field called "Priority".
5. In that screen, change the label of the field Priority to Importance.
6. This way, there may be fields already available, but you call it by one name and the tool calls it by another name; just rename in the tool.
7. Log out and relogin; go to the project that you configured. Click on new defect. You will see the same field names that you wanted. This is half the battle won.
8. Mark those fields that you have in list that are not in the tool.
9. For example, you want a field called Browser as a list box, with values MS IE, FireFox, Chrome.
10. Login as eadmin to your enterprise, Go to settings, go to User Defined Fields.
11. In the defects module, add a new field, with field label as Browser and type as single selection list box and add these values as comma separated ones.
12. Repeat the above step for all the fields that are not in the tool.
13. Log out, relogin and go to the project. Create a new defect.
14. Now you will see those new fields that you want, right in front of you.

This way, you can make the tool to adapt to your templates.

Why do people feel it hard, to use a tool? - Part 2

Pain number 6. When we start, the tool was internally built and it was simple to use. Within 6 months, so many people wanted so many features and rules, and it became difficult for me to use. So by doing too many customization, people make simpler things more complex.

Pain 7. Just one manager is not happy about the tool; he/she starts telling bad stories about the tool to others and that creates a bad mindset. So after sometime, people start moving away from the tool.

Pain 8. People keep doing comparison against the best tool in that category with the tool your project uses. A wish list of people kills the spirit. To travel 2 miles to reach office, you do not need a race car that moves from 0 to 70 miles per hour in 7 seconds.

Pain 9. A few people want to try a new tool; for what? They want that jazzy name of the tool to appear in their resume! So the real intention is not to use the tool, but to just see the features and move out.

Pain 10. Ultimate fear of being tracked! Am I being policed? The more details I put in the tool, can act against me. So log only good stuff and let all bad data go to hell. When project is in crisis, and management generates a report, the report looks cool and green with no escalations. But that is not true and everyone knows that.

A tool can be anything from simple text editor to IDE to load testing tool or tracking tool. But people find a way to evade it. But ultimately if the wish matures to need and need turns to be a pain, and the pain cannot be solved without this tool, people do not care which tool they use - they start using it and they stay happy!

Why do people feel it hard, to use a tool?

Many times, you may hear the words from your management "We need specific metrics before we take a decision". But, from where we will get those numbers? The simple answer is right in front of our eyes - we have those numbers; the only difference is that those numbers are in different form. Someone has to collect, collate and present. Who is that magician to do that? Can that person relentlessly do that, what are the chances of making mistakes?

Simple rule. When you cannot rely on humans, rely on tools. They do not get tired or they do not at least ask salary hikes. When one SME company wants to have a tools - they get too many options from open source. When we did survey, many people said that, even though those tools are free, we need to spend a lot of time in setting up, configuring etc. Support is not a happy part of those tools. So the first pain is getting a tool and setting it up.

Second pain comes on usability. Are those tools designed with the best usability? Some techies did that - features are great, but the UI is not good, reports are not upto the expectations... No one wants to spend more than 2 hours to train or retrain their employees on a new tool.

Third pain. When we ask a question in the forum, answer comes 1 week after. By the time, the answer comes, the problem is either solved in one or the other way or it is forgotten. Hence the support loses the gas. People can wait for max 24 hours, and not more than that.

Fourth pain. This is one of the biggest pains you can ever think of. Addiction of people to emails and spreadsheets. When they are so used to these 2, making them to enter some stuff on a neat UI is not that easy. It is like making your kid to take broccoli soup.

Fifth pain. There is a not a big community behind the tool to support from peer users. This is an important aspect. People get results from other people who already faced the problems. They trust the words of these people, more than the original author who coded the tool!

Pain will continue in further posts...