Test Coverage - SE 3S03€¦ · What Test Coverage Does I Finds areas of requirements not...

Test Coverage

Zahra Ali

Test Coverage

RequirementsCoverage

Structural Coverage

ArchitecturalCoverage

Advantages

Disadvantages

RequirementsTraceability Matrix

Types of TraceabilityMatrix

Summary

1/34

Test CoverageSE 3S03

Zahra Ali

Department of Computing and SoftwareMcMaster [email protected]

Week of Feb 29, 2016

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

2/34

Outline

Test CoverageRequirements CoverageStructural CoverageArchitectural Coverage

Advantages

Disadvantages

Requirements Traceability MatrixTypes of Traceability MatrixSummary

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

3/34

Test Coverage

I The measure of the effectiveness of the testing byproviding data on different items.

I Amount of testing performed by a set of test cases

I 100% coverage does not mean 100% tested

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

4/34

What Test Coverage Does

I Finds areas of requirements not implemented by a set oftest cases

I Helps to create additional test cases to increasecoverage

I Identifies a quantitative measure of test coverage, whichis an indirect method for quality check

I Identifies meaningless test cases that do not increasecoverage

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

5/34

What Test Coverage Does

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

6/34

Coverage Measure

I The basic coverage measure

where the coverage item is whatever we have been ableto count and see whether a test has exercised or usedthis item.

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

7/34

Size of the software

I Requirements, Structure and Architecture are threesoftware abstractions which are used to develop andbuild the software

I With these we can determine the completeness criteriawith an ease

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

8/34

Requirements Coverage

I Has the software been tested against all requirementsfor the normal range of use?

I Has the software been tested against all requirementsfor abnormal or unexpected usage?

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

9/34

Techniques of Requirements Coverage

I Normal Range Coverage of Software Requirements.Sunny day tests

I Normal Range Coverage of Software Design

I Robustness Coverage of Software Requirements. Rainyday tests

I Robustness Coverage of Software Design

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

10/34

Structural Coverage

I Has each element of the software been exercised duringtesting?

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

11/34

Techniques of Structural Coverage

I Code Statements

I Decisions

I Conditions and Decisions

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

12/34

..Recall White Box Testing

I Structural coverage is white box testing

I Statement coverage

I Edge coverage/Branch Coverage

I Condition coverage

I Path coverage

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

13/34

Architectural Coverage

I Have the actual control and data links been utilizedduring testing?

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

14/34

Techniques of Architectural Coverage

I Data path part of the CPU where data signals flow

I Control unit guides data signals through data path

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

15/34

How do we perform Test Coverage Analysis?

I Test Coverage can be implemented by Static testingtechniques.

I These test techniques include peer reviews, codeinspections and code walkthroughs.

I We can convert the adhoc defects into test cases andanalyze test coverage.

I We can use code level tools and automation to achievetest coverage at unit level.

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

16/34

How do we perform Test Coverage Analysis?

I We can use test management tools to performfunctional test coverage which will establish traceabilitybetween, requirements, defects and test cases.

I We can use Bi- directional traceability matrix to achievetest coverage.

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

17/34

Advantages of Test coverage Analysis

I We can prevent defect leakage using Test coverageanalysis.

I Defects can be prevented by this early stage ofapplication life cycle.

I We can keep time, resources, cost, scope under controlwith this technique.

I The test coverage analysis can determine the decisionpoints and important path made in the applicationwhich helps us to increase the test coverage.

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

18/34

Advantages of Test coverage Analysis

I We can check the paths of the code which are nottested.

I We can determine the quality of the testing we areperforming.

I We can find the gaps in requirements, test cases anddefects at an early level and code level.

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

19/34

Disadvantages of Test Coverage

I There is no measurement of the part of the softwarethat is not coded. Thus it has coverage only of thewritten part.

I If there is a specification which is not implemented orthere might be a specification which is now removed,then this technique will not tell anything about it. Itsays only about the structure code which is alreadypresent.

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

20/34

Requirements Traceability Matrix

I Requirements Traceability Matrix establishes a way tomake sure we place checks on the coverage aspect. Ithelps in creating a snap shot to identify coverage gaps.

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

21/34


I Testers start writing their test scenarios/objectives andeventually the test cases based on some inputdocuments Business requirements document,Functional Specifications document and Technicaldesign document (optional).

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

22/34


Lets suppose, the following is our Business requirementsdocument (BRD):

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

23/34


The below is our Functional Specifications document (FSD):

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

24/34


Based on the above two input documents, as the QA teamwe came up with the below list high-level scenarios for us totest.

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

25/34


For each test scenario you have at least 1 or more test cases.So, include another column and write the test case IDs asshows below:

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

26/34


I The test team has somehow missed considering theExisting user functionality.

I The Existing user functionality has been deferred tolater or removed from the applications functionalityrequirements. In this case, the TM shows aninconsistency in the FSD or BRD which means that anupdate on FSD and/or BRD documents should be done.

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

27/34


The below version of the Traceability Matrix is generallyprepared during or after test execution:

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

28/34

Types of Traceability Matrix

I Forward traceability: This matrix is used to checkwhether the project progresses in the desired directionand for the right product. It maps requirements totest cases.

I Backward or reverse traceability:It is used to ensurewhether the current product remains on the right track.Used to verify that we are not expanding the scope ofthe project by adding code, design elements, test orother work that is not specified in the requirements.Itmaps test cases to requirements.

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

29/34

Types of Traceability Matrix

I Bi-directional traceability (Forward+Backward):This traceability matrix ensuresthat all requirements are covered by test cases. Itanalyzes the impact of a change in requirementsaffected by the defect in a work product and vice versa.

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

30/34

Advantages of using RTM

I Ensuring 100% test coverage

I Showing requirement document inconsistencies

I Displaying the overall defect/execution status withfocus on business requirements.

I If a certain business and/or functional requirement wereto change, a TM helps estimate or analyze the impacton the QA teams work in terms of revisiting/reworkingon the test cases.

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

31/34

Disadvantages of not using RTM

I More defects in production poor or unknown testcoverage.

I Discovering bugs later in the development cycleresulting in more expensive fixes.

I Difficulties planning and trackingprojects.

I Misunderstandings between different teams over projectdependencies, delays, etc.

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

32/34

About RTM

I A TM is not a manual testing specific tool, it can beused for automation projects as well. For an automationproject, the test case ID can indicate the automationtest script name.

I It is also not a tool that can be used just by the QAs.The development team can use the same to mapBRD/FSD requirements to blocks/units/conditions ofcode created to make sure all the requirements aredeveloped.

Test Coverage

Zahra Ali

Test Coverage


Structural Coverage


Advantages

Disadvantages



Summary

33/34

Summary

I The measure of the effectiveness of the testing byproviding data on different items.

I Does not guarantee 100% accuracy of the software

I Requirement traceability matrix

Test Coverage

Zahra Ali

Appendix

References

34/34

References I

William E. Perry;Effective Methods for Software Testing

Test Coverage - SE 3S03€¦ · What Test Coverage Does I Finds areas of requirements not...

Documents

Transcript of Test Coverage - SE 3S03€¦ · What Test Coverage Does I Finds areas of requirements not...