Post on 21-Dec-2015
Validating and Improving Validating and Improving Test-Case EffectivenessTest-Case Effectiveness
Author: Yuri ChernakAuthor: Yuri Chernak
Presenter: Lam, Man TatPresenter: Lam, Man Tat
AgendaAgenda
Some definition that need to be Some definition that need to be understandunderstand
How verification and validation is doneHow verification and validation is done
Overview of how in process validation of Overview of how in process validation of test cases is donetest cases is done
Case-studyCase-study
ConclusionConclusion
Functional testingFunctional testing
By focusing on a system's functionalityBy focusing on a system's functionality
Looking for as many defects as possibleLooking for as many defects as possible
Test plan documents and test case Test plan documents and test case specifications are important deliverablesspecifications are important deliverables
Test CasesTest Cases
Critical for effective testingCritical for effective testing
Testers use test-case specifications does Testers use test-case specifications does not guarantee that systems are sufficiently not guarantee that systems are sufficiently testedtested
VerificationVerification
First verify test case specifications at the First verify test case specifications at the end of the test design phaseend of the test design phase
Reviews or inspections – evaluate test Reviews or inspections – evaluate test case specifications for their correctness case specifications for their correctness and completenessand completeness
Test cases that passed verification could Test cases that passed verification could have weak failure-detecting abilityhave weak failure-detecting ability
ValidationValidation
Can process as soon as testers have Can process as soon as testers have executed all test casesexecuted all test cases
Determines whether the test cases were Determines whether the test cases were sufficiently effective in finding defectssufficiently effective in finding defects
If test cases effectiveness has not proved If test cases effectiveness has not proved satisfactory, it is not too late to analyze the satisfactory, it is not too late to analyze the reasons and correct the test processreasons and correct the test process
Test case effectiveness metric introducedTest case effectiveness metric introduced
Test case effectiveness metricTest case effectiveness metric
Mainly use for online client server systemsMainly use for online client server systems
Certain number of defects are always Certain number of defects are always found as a side effectfound as a side effect
Assume that the more defects test cases Assume that the more defects test cases find, the more effective they arefind, the more effective they are
Define as the ratio of defects found by test Define as the ratio of defects found by test cases (Ntc) to the total number of defects cases (Ntc) to the total number of defects (Ntot) reportes during the test cycle(Ntot) reportes during the test cycle
Test case effectiveness metric Test case effectiveness metric Cont’Cont’
TCE = Ntc / Ntot *100%TCE = Ntc / Ntot *100%
TCE metric evaluates test cases from the test TCE metric evaluates test cases from the test cycle perspective, which provides in process cycle perspective, which provides in process feedback to the project team on how well a test feedback to the project team on how well a test suite has worked for testerssuite has worked for testers
Compare the actual TCE for a given project with Compare the actual TCE for a given project with a baseline value (such as previous successful a baseline value (such as previous successful project, author experience suggests 75 %)project, author experience suggests 75 %)
Test case effectiveness metric Test case effectiveness metric Cont’Cont’
If the TCE value is at baseline or above, If the TCE value is at baseline or above, we can conclude that the test cases have we can conclude that the test cases have been sufficiently effective in test cyclebeen sufficiently effective in test cycle
If the TCE value falls below the baseline, If the TCE value falls below the baseline, the higher is the risk of user dissatisfactionthe higher is the risk of user dissatisfaction
Improving test case effectivenessImproving test case effectiveness
If in process validation finds test case If in process validation finds test case effectiveness to be less than acceptable, effectiveness to be less than acceptable, the project team should analyze the cause the project team should analyze the cause and identify areas that need to be improveand identify areas that need to be improve
Author’s proposed improvement Author’s proposed improvement framework was based on IBM’s “Test case framework was based on IBM’s “Test case escape”escape”
IBM’s “Test case escape”IBM’s “Test case escape”
Define as “product defects that a particular Define as “product defects that a particular test failed to find, but which were found in test failed to find, but which were found in a later test, or by a customer [in a later test, or by a customer [in production]production]
Author’s proposed improvement Author’s proposed improvement frameworkframework
Define as software defects that a given Define as software defects that a given suite of test cases failed to find but that suite of test cases failed to find but that were found as a side effect in the same were found as a side effect in the same test cycletest cycle
Rather than compare the Ntot with the Rather than compare the Ntot with the whole processwhole process
Author’s proposed improvement Author’s proposed improvement frameworkframework
1- Understand and document the test 1- Understand and document the test process used by the project teamprocess used by the project team
2- Make assumptions about factors that 2- Make assumptions about factors that affect test casesaffect test cases
3- Gather defect data3- Gather defect data
4- Identify main factors4- Identify main factors
5- Implement corrective actions5- Implement corrective actions
Understand, document the test Understand, document the test processprocess
Test planning – definition of the scope, Test planning – definition of the scope, objectives, and approach to testingobjectives, and approach to testing
Test design – the design of the test casesTest design – the design of the test cases
Test preparation and execution – Test preparation and execution – preparation of the test environment preparation of the test environment executing test cases and finding defectsexecuting test cases and finding defects
Test evaluation and improvement – Test evaluation and improvement – analyzing the results of testinganalyzing the results of testing
Make assumptionMake assumption
Once understands and documents the test Once understands and documents the test process, the project team should analyze process, the project team should analyze each phase and identify factors that can each phase and identify factors that can affect test case effectivenessaffect test case effectiveness
Lets look at some exampleLets look at some example
Example of making assumptionExample of making assumption
Test planning – if the functional Test planning – if the functional specifications do not completely define specifications do not completely define functional features, the test plan document functional features, the test plan document will not be complete either. So, the test will not be complete either. So, the test cases will not completely cover a system’s cases will not completely cover a system’s functionalityfunctionality
Example of making assumption 2Example of making assumption 2
Test design – a common example could Test design – a common example could be lack of negative test cases (such as be lack of negative test cases (such as abnormal conditions by using either invalid abnormal conditions by using either invalid data input or the wrong user actiondata input or the wrong user action
Test design – test case specifications Test design – test case specifications could be simply incorrectcould be simply incorrect
Gather defect data and perform Gather defect data and perform analysisanalysis
Defect tracking systemDefect tracking system
Able to identify which defects were found Able to identify which defects were found as result of test cases and which were as result of test cases and which were found as a side effectfound as a side effect
Once selected, defects should be Once selected, defects should be classified according to one of the factors classified according to one of the factors based on the analysisbased on the analysis
Gather defect data and perform Gather defect data and perform analysis cont’analysis cont’
Analysis is used to evaluate each test Analysis is used to evaluate each test case and understand why the test suite case and understand why the test suite missed the corresponding defect during missed the corresponding defect during the test executionthe test execution
Example: A lack of negative test cases in Example: A lack of negative test cases in is a common cause of missed defects. is a common cause of missed defects. Can you see which category it will be Can you see which category it will be under?under?
Identify the main factorsIdentify the main factors
At this point, all test case escapes have At this point, all test case escapes have been classified according to their been classified according to their respective causesrespective causes
Build a “Pareto chart” which displays Build a “Pareto chart” which displays frequency bars in descending order, frequency bars in descending order, convenient for analyzing (will be an convenient for analyzing (will be an example later)example later)
Implement corrective actionsImplement corrective actions
After identification of the main causes of After identification of the main causes of test case escapes, the project team should test case escapes, the project team should implement corrective actions and repeat implement corrective actions and repeat the test executionthe test execution
Example of corrective actionsExample of corrective actions
Rework functonal specifications and test Rework functonal specifications and test case specificationcase specification
Use a traceability matrix to ensure Use a traceability matrix to ensure complete coverage of business rulescomplete coverage of business rules
Use checklists to design test case Use checklists to design test case specificationspecification
Implement training of testers on test Implement training of testers on test executionexecution
Case studyCase study
Banking application intended for external Banking application intended for external clientsclients
Project team consisted 10 developers and Project team consisted 10 developers and 3 testers3 testers
183 defects reported183 defects reported
71 found by test case escapes71 found by test case escapes
112 found by test case execution112 found by test case execution
Case study cont’Case study cont’
TCE = 61%TCE = 61%
Lower than the acceptable level of 75%Lower than the acceptable level of 75%
Project team performed the test process Project team performed the test process improvement according to the frameworkimprovement according to the framework
Pareto chart created for analysisPareto chart created for analysis
Case study cont’Case study cont’
Analysis of the distribution showed Analysis of the distribution showed incomplete test design and incomplete incomplete test design and incomplete functional specification to be the main functional specification to be the main factors causing missed defects.factors causing missed defects.
Project team correcting and completing Project team correcting and completing the test design and functional specificationthe test design and functional specification
Case study cont’Case study cont’
As a result, 48 additional defects found As a result, 48 additional defects found before releasing the productbefore releasing the product
By the end of the second month the By the end of the second month the number of production, only 23 defects was number of production, only 23 defects was foundfound
Indicated sufficient effectiveness of test Indicated sufficient effectiveness of test progress, user should be satisfied with the progress, user should be satisfied with the system qualitysystem quality
ConclusionConclusion
This technique intended to give project This technique intended to give project teams better visibility into test process teams better visibility into test process effectiveness effectiveness beforebefore their system are their system are released into productionreleased into production
Help evaluate each test case and Help evaluate each test case and understand why the test suite missed the understand why the test suite missed the corresponding defect during test executioncorresponding defect during test execution
ConclusionConclusion
Help testers revise and improve the test Help testers revise and improve the test suitesuite
In other word, it helps testers find In other word, it helps testers find additional defects and deliver a better additional defects and deliver a better software productsoftware product