Software testing is an indispensable part of the software development process, aimed towards improving the quality and accuracy of the software. Designing extensive test cases trying to cover every nook and corner of the functionality is the fundamental of all software testing practices. Failure to create elaborate, all-encompassing test scripts, introduce loop holes in the SDLC that provides a backdoor for bugs to travel to the production system.
The Cost of fixing a bug in a real-life scenario depends on the stage when they are discovered. Once into production, costs can be as extensive as recalling an entire batch of a product. The most epic examples of massive product reversals are Toyota motor’s faulty gas pedals and Ford’s cruise control product recalls which cost the companies millions of dollars.
Measure what is measurable and make measurable what is not so.
Galileo Galilei
What are Testing Metrics?
Metrics are used as a standard of measurement, to provide a qualitative and quantitative analysis of the process/product being evaluated. These can be used to further improve the processes being executed. Metrics need to be easily measurable and actionable to realign with the existing production processes being used. Similarly, Software Testing Metrics are created to provide quantitative and qualitative evaluation of the SW development process and the Testing Strategy that is being implemented.
Testing Metrics are broadly classified based on 2 objectives -
- Process Based Metric: Used to improve the processes involved in testing
- Product Based Metric: Used to measure the product quality, usability, performance and provide a feedback on the defects detected.
A project does not involve all available Testing Metrics to evaluate their process. Identification of the metric depends on the project’s testing strategy and the Development Methodology being used. The most common Testing Metrics in use are listed below.
Test Coverage
Coverage can be segregated over 3 areas of the project; based on the customer requirements, the logical paths of the code and code coverage. Are your test scripts evaluating all the customer requirements provided before the project can be evaluated based on Test Design coverage.
TEST DESIGN COVERAGE = Total number of requirements mapped to Test Cases / Total number of Requirements
Automation Test Coverage is another parameter which evaluates the number of manual test cases that could be executed as automated scripts.
AUTOMATION COVERAGE = Number of manual test cases automated / Total number of test cases
Test Execution productivity Metric
It defines the efficiency with which test cases are executed per hour.
TEST EXECUTION PRODUCTIVITY = Total No of test cases executed/Efforts in Hours spent for execution of Test Cases
Defect Removal Efficiency
The defect removal efficiency (DRE) is a measure of the development team’s ability to remove defects before a code base is deployed to production environment. It is the comparison between the number of defects found before the code was delivered to production and the ones found after the code went live. The higher the DRE the better the testing strategy in place.
DEFECT REMOVAL EFFICIENCY = Number of Defects detected before Production / Total number of Defects Detected
Defect Acceptance Metric
Once the defects are identified and directed back for resolution, the percentage of these defects which are accepted as valid by the development team, constitutes the Defects Acceptance Percentage.
DEFECT ACCEPTANCE = Number of valid defects / Total number of defects identified
This can further be branched out as the number of defects that were rejected.
DEFECT REJECTION = Number of defects that were Invalid / Total number of defects identified
Going beyond the values generated by the Metric, each finding should be sent through a Root Cause Analysis Cycle (RCA). RCA is a process designed to understand the causes of deviations in metrics and coming up with alternatives/modifications in the process. An RCA document should include WHY the defect wasn’t caught during the different stages of Testing and WHAT we could do to avoid this in future. RCA helps in contributing to the improvement of the SDLC processes by restructuring the leaky areas of the Testing Lifecycle.
With the emergence of DevOps the trend towards automating most of the testing methodologies is on the rise. You now have access to a pool of tools that would help you automate and streamline almost all branches of testing and provide you with a comprehensive report for evaluation of the Testing Metrics. This further helps you evaluate your testing strategies and channelize the inputs to your CI/CD processes.
Using Automated testing methodologies, Clarion has helped numerous clients reduce both Effort and Cost they put into testing. Having an experience of over 18 years in the industry, we expertise in advance tools like Selenium (with Java, .Net, Python), TestNG, Maven, Protractor, Appium, JS Frameworks, JMeter, etc.
For further reading here is link to a book that will provide you with elaborate information on Agile testing methodologies - Introduction to Agile Testing
