Software Testing part II (white box)santucci/SW_Engineering/Material/05_A_Test_PartII.pdf ·...
Transcript of Software Testing part II (white box)santucci/SW_Engineering/Material/05_A_Test_PartII.pdf ·...
4. White box testing
• White-box (or Glass-box) testing: general characteristics
• Statement coverage• Decision coverage• Condition coverage• Decision and condition coverage• Coverage of a linearly independent set of paths
2
White-box testing: general characteristics
• Also called– Glass-box testing– Structural testing (compared to functional)
• Exploits the control structure, that means the control or flow graph of programs in order to design test cases
3
White-box testing: general characteristics cont.d
• Allows to test parts of the program– With black-box is not possible
• Allows to cover systematically every part of the program– Often errors are found in special cases
4
Statement coverage
• It is the easiest criterion for a reasonable white-box test
• Necessary but not sufficient• Goal
– To identify a set of test cases sufficient to exercise all statements at least once
6
Statement coverage cont.d
The test case{x != 0; y = any}
Covers the statements
Do you see any problem ?7
Java example:
/*A*/ float x = InOut.readFloat(),y = InOut.readFloat();
/*B*/ if (x != 0)/*C*/ y = y+10;/*D*/ y = y/x;/*E*/ System.out.println(x + ' ' + y);
A
B
C
D
E
x == 0x != 0
Control flow graph for the example
Statement coverage cont.d
• Possible problem– Even in a case of a program without iterations, the
execution of every statement does not guarantee that all possible paths are exercised
• Possible negative consequence:– In previous java example program the error of division
by 0 is not identified (statement D)
8
Decision coverage
• Properly includes statement coverage• Vice versa does not hold• Focuses on all possible decisions in a program • Decisions are in statements
– if, switch, while, for, do• Goal
– To identify a set of test cases sufficient for guaranteeing that each decision will have value “true” at least once and value “false” at least once
9
Decision coverage cont.d
10
Java example:
/*A*/ float x = InOut.readFloat(),y = InOut.readFloat();
/*B*/ if (x != 0)/*C*/ y += 10;/*D*/ y = y/x;/*E*/ System.out.println(x + ' ' + y);
x == 0Test cases1. {x=20; y=30}2. {x=0; y=30}
Cover decisions hence statementsThe second case identifies the error of division by zero in statement D
A
B
C
D
E
x != 0
Decision coverage cont.d
• Possible problem• If decisions are composed of several conditions
(AND, OR), decision coverage can be not sufficient
11
Java example:/*A*/ float z = InOut.readFloat(),
w = InOut.readFloat();t = InOut.readFloat();
/*B*/ if (z == 0 || w > 0)/*C*/ w = w/z;/*D*/ else w = w + 2/t;/*E*/ System.out.println(z+' '+w+' '+t);
What is the problem here?
Decision coverage cont.d
• Test cases:– {t = 0; z= 5 ; w= 5} decision TRUE– {t = 0; z= 5 ; w= -5} decision FALSE
• Cover decisions• The risk of division by zero in D is identified
(thanks to t = 0 that is not due to decision coverage strategy !!!! )• The risk of division by zero in C is not identified (z!=0)• We need a criterion that considers the decision’s
conditions (i.e., the atomic Boolean expression components)
(z == 0 || w > 0)
12
Condition coverage
• Does not properly include decision coverage• It is not properly included by decision coverage• Focus on all possible conditions in a program• Conditions are the components of Boolean
expressions • Goal:
– To identify a set of test cases sufficient for guaranteeing that every condition (atomic Boolean expression) included in the program’s decisions have value “true” at least once and value “false” at least once
13
Condition coverage cont.d
14
Test cases{t=0;z=0;w=‐5} 1st condition=T, 2nd condition=F{t=0;z=5;w= 5} 1st condition=F, 2nd condition=TCover conditions
What problem do such case tests make emerge?
Java example:/*A*/ float z = InOut.readFloat(),
w = InOut.readFloat();t = InOut.readFloat();
/*B*/ if (z == 0 || w > 0)/*C*/ w = w/z;/*D*/ else w = w + 2/t;/*E*/ System.out.println(z+' '+w+' '+t);
Condition coverage
• Test cases– {t=0;z=0;w=-5} 1st condition=T, 2nd condition=F– {t=0;z=5;w= 5} 1st condition=F, 2nd condition=T
• Covers conditions• The risk of division by zero in C is identified • The risk of division by zero in D is not identified as
D is never exercised• Decision is always “true”
15
Decision and condition coverage
• Properly includes– Decision coverage
– And, as a consequence, statements coverage
– Condition coverage• Goal
– To identify a set of test cases sufficient for guaranteeing that
• Each decision is “true” at least once and “false” at least once• All conditions composing decisions is “true” at least once and
“false” at least once
16
Decision and condition coverage cont.d
17
Test cases{t=0;z=0;w=5} 1st condition=T, 2nd condition=T, decision=T{t=0;z=5;w=‐5} 1st condition=F, 2nd condition=F, decision=FCover decisions and conditions and identify both division by zero in
statements C and D(thanks to t = 0 that is not due to decision and condition coverage
strategy !!!! )
/*A*/ float z = InOut.readFloat(),w = InOut.readFloat();t = InOut.readFloat();
/*B*/ if (z == 0 || w > 0)/*C*/ w = w/z;/*D*/ else w = w + 2/t;/*E*/ System.out.println(z+' '+w+' '+t);
Summary of test cases so far
18
Test case
t z w cond.1
cond.2
decis. Finds error in C?
Finds error in D?
C1 0 0 5 T T T yes no
C2 0 5 5 F T T no no
C3 0 0 -5 T F T yes no
C4 0 5 -5 F F F no yes
Decision and condition coverage
19
Coverage criterion
Associated test cases
Finds error in C?
Finds error in D?
Decision C2 + C4 no yes
Condition C2 + C3 yes no
Decision and condition
C1 + C4, orC2 + C3 + C4
yes yes
Which one is the best?
Methodological issue
• There are different combination of test cases that guarantee decision and condition coverage but it is not always trivial to find one
• We need a simple and effective method for identifying test cases that cover both decisions and conditions
20
Control flow graph’s paths coverage
• Based on– Criteria of decision and condition coverage– Program’s control flow graph
• Goal – To provide a simple method for guaranteeing decision
and condition coverage
21
Control flow graph
22
while (A) {if (B) { C }else { D }E
}F;
Example:V(G) = 7 ‐ 6 + 2 = 3V(G) = 3
node
edges
region
Predicate nodes
A
B
CD
E
F
Cyclomatic number•V(G) = E ‐ N + 2 (E = number of edges, N = number of nodes)•V(G) = R (R = number of regions)
Control flow graph
23
Sequence ofstatements
Simple if
until
whilecase
What about the for statement? It must be simulated through a while…
Control flow graph without conditions
24
A
B
CD
E
False True
The control flow graph should be more detailed
/*A*/ float z = InOut.readFloat(),w = InOut.readFloat();t = InOut.readFloat();
/*B*/ if (z == 0 || w > 0)/*C*/ w = w/z;/*D*/ else w = w + 2/t;/*E*/ System.out.println(z+' '+w+' '+t);
Control flow graph with complex decisions
25
if (a || b) x;
else y;
b==F
...
a
b
xy
...
a==Ta==F
b==T
if (a && b) x;
else y;
...
a
b
yx
...
a==Fa==T
b==Fb==T
Control flow graph for the program
26
A
B
CD
E
False True
A
z
w
CD
E
== 0!= 0
<= 0 > 0
Control flow graph Without conditions
Control flow graph with conditions
/*A*/ float z = InOut.readFloat(),w = InOut.readFloat();t = InOut.readFloat();
/*B*/ if (z == 0 || w > 0)/*C*/ w = w/z;/*D*/ else w = w + 2/t;/*E*/ System.out.println(z+' '+w+' '+t);
Control flow graph, conditions and decisions
• Boolean values (“true” and “false”) associated with conditions and decisions have corresponding edges
• To design a set of test cases such that all edges of the control flow graph are traversed implies condition and decision coverage
27
Paths of the control flow graph
• The set of test cases corresponds to a set of paths such that every edge is traversed at least once
• The number of paths sufficient for covering all arcs is always equal or less than the cyclomaticcomplexity
28
For the previous example
29
Three test casesC2. {t=0; z=5; w= 5}C3. {t=0; z=0; w= ‐5}C4. {t=0; z=5; w=‐5}Cover all edges, hence decisions and conditions
V=7‐6+2=3The three paths 1. A‐z‐w‐C‐E (C2) 2. A‐z‐C‐E (C3)3. A‐z‐w‐D‐E (C4)
/*A*/ float z = InOut.readFloat(),w = InOut.readFloat();t = InOut.readFloat();
/*B*/ if (z == 0 || w > 0)/*C*/ w = w/z;/*D*/ else w = w + 2/t;/*E*/ System.out.println(z+' '+w+' '+t);
A
z
w
CD
E
== 0!= 0
<= 0 > 0
How to choose paths
• Pragmatic rule– Experiments show that the the number of errors
increases with increasing of the cyclomaticcomplexity
– Choose a number of paths equal to the cyclomaticcomplexity
– You have a correlation between number of test cases and code complexity
– Which paths?
30
Linearly independent paths:Basis Path Testing
• A good criterion for choosing paths is based on the concept of linear independency
• A maximal set of linearly independent paths is called a basis– It is NOT unique
• A basis contains a number of paths equal to the cyclomatic complexity
• Intuitively, every path in the control graph can be obtained as a linear combination of paths of a basis
• By choosing a basis, a certain extent of reliability is guaranteed with respect of combinations of errors that hide each other
31
Example of paths
32
while (A) {if (B) { C }else { D }E
}F;
Paths. A‐F. A‐B‐D‐E‐A‐F. A‐B‐C‐E‐A‐F
A
B
CD
E
F
1
2 3
4 5
67
Arcs/paths 1 2 3 4 5 6 7
0 0 0 0 0 0 1 1 1 0 1 0 1 1 1 0 1 0 1 1 1
The matrix rank is 3 (the maximum) { } is a basis
Note that A‐B‐D‐E‐A‐B‐C‐E‐A‐F covers all edges BUT…
Remarks• Every path corresponds to a binary row vector (0/1)• Vice versa does not hold• A matrix representing paths has maximum 2E
distinct rows (E = n° of edges)• McCabe has proved that for each graph G:
1. The matrix rank cannot be greater than V(G) (cyclomatic complexity)
2. There exists always a matrix made of paths with rank equal to V(G)
• A basis is any set of paths (rows) with maximum rank
• Note: MacCabe’s result is purely topological:– /*B*/ if(5>7)– /*C*/ i=9;– It is not possible to traverse Edge B-C
• In other cases is necessary to re-iterate the cycles in order to traverse all edges
33
A
B
CD
E
F
1
2 3
4 5
67
Edges traversable only by iterations
34
/*A*/ int z = … ; int t=0;/*B*/ while (t<100)/*C*/ if (t > 20) /*D*/ t=t+90;/*E*/ else t= t+z*z;/*F*/ System.out.println(t);
A
B
C
ED
K
t>=100t<100
t<=20t>20
F
B‐FC‐DRequire at least one iteration in order to be traversed{z=10} A‐B‐C‐E‐K‐B‐F{z= 5} A‐B‐C‐E‐K‐B‐C‐D‐K‐B‐F
SoA‐B‐FA‐B‐C‐D‐K‐B‐Fare not traversable
Fake node
Additional considerations
• Paths coverage method does not consider explicitly loops iterations
• In some cases it is necessary to perform iterations in order to traverse all edges
• It might be interesting to consider some iterations in any case but the number of test cases increases exponentially
• It is important to decide the type of iterations
35
Pragmatic choices for loops
• To limit the number of iterations to n– It is called “loop coverage”
• To execute only certain loops• To limit the number of paths to be traversed based
on weighted edges and function to be maximized– Probability of execution– Resources occupancy
• To limit the number of paths by identifying the paths that define and use program variables (Data Flow testing)– Definition of a variable value– Use of such value in a test
36
Data Flow Testing
37
...A int x,y,a,b;B scanf(“%d %d”,&x, &y);C a=x;D b=yE while (a!=b)F if(a>b)G a=a‐b;H else b=b‐a;I printf(“%d”,a);....
D
E
I
F
a!=b
C
B
HG
a==b
a<=ba>b
For x and y it is not needed to execute the loop (loop coverage = 0)For the definition of a and b the loop has to be iterated once (loop coverage = 1)For the definition and usage of a and b the loop has to be iterated twice (loop coverage = 2)
5. Testing strategies
• Testing and development process• Verification vs. Validation• The V-model• Test levels• Test types• Maintenance testing
38
Testing and development process 1
• The adopted life cycle has a big impact on the testing that is carried out: testing does not exist in isolation
• Test activities are highly related to software development activities
• The way testing is organized must fit the development life cycle or it will fail to deliver its benefit
• If a fully documented software development life cycle is required, the testing must be fully documented
39
Verification vs. Validation
• In every development life cycle, a part of testing is focused on verification testing and a part is focused on validation testing
• Verification:– Evaluation of a work product, component or system to determine
whether it meets the requirements set– Answers the question:
• Is the deliverable built according to the specification?• Validation:
– Evaluation of a work product, component or system to determine whether it meets the user needs and requirements
– Answers the question: • Does the deliverable fit for purpose, e.g., does it provide a solution to
the problem?
40
Good practices for testing
Good testing practices apply to any life cycle model• For every development activity there is a
corresponding testing activity• Each test level has test objectives specific to that
level• The analysis and design of tests for a given test level
should begin during the corresponding development activity
• Testers should be involved in reviewing documents as soon as drafts are available in the development cycle
41
V-model
• The V-model was developed to address some of the problems experienced using the traditional waterfall approach
• Defects were being found too late in the life cycle, as testing was not started until the end of the project
• The V-model provides guidance to begin testing as early as possible in the life cycle
42
03Test.43
RequirementsAnalysis
RequirementsReview
SystemDesign
DesignReview
ArchitectureDesign
ArchitectureReview
ModuleDesign
ModuleReview
Coding
Acceptance testPlanning
Functional testPlanning
Integration testPlanning
Acceptance Testing
SystemTesting
IntegrationTesting
Unit testPlanning
UnitTesting
Design Planning Testing
V-model
Internal activities
Activities performed with the userReview
Four test levels1. Unit testing
– Searches for defects in and verifies the functioning of software components • e.g. modules, programs, objects, classes etc., that are separately testable
2. Integration testing – Tests interfaces between components, interactions to different parts of a
system such as an operating system, file system and hardware or interfaces between systems
3. System testing – Concerned with the behavior of the whole system/product as defined by the
scope of a development project or product – The main focus of system testing is verification against specified requirements
4. Acceptance testing – validation testing with respect to user needs, requirements, and business
processes conducted to determine whether or not to accept the system
44
1: Unit testing
• Also known as component, module, and program testing
• Searches for defects in, and verifies the functioning of software– e.g. modules, programs, objects, classes, etc. that are
separately testable• Can be done in isolation from the rest of the system• Stubs and drivers are used to replace the missing
software and simulate the interface between the software components in a simple manner
45
Stub and Driver
• A stub is called from the software component to be tested
• A driver calls a component to be tested
46
A
B
A
Stub
Driver
B
1: Unit testing (cont.)
• Typically occurs with access to the code being tested
• Can be supported by the development environment– such as a unit test framework or debugging tool – See for example JUnit (http://www.junit.org/)
• Usually involves the programmer who wrote the code
• Defects are fixed as soon as they are found, without formally recording the incidents found
47
Junit test case example
@Testpublic void testClassX_methodY(){
ClassX a = new ClassX();assertEquals(4, a.methodY(3, 1));
}
50
Test-driven development
• To prepare and automate test cases before coding
• Highly iterative • Based on cycles of developing test cases, then
building and integrating small pieces of code, and executing the component tests until they pass
• Used in Extreme Programming
51
2: Integration testing
• Focus: – interfaces among components – Interactions with different parts of a system
• an operating system, file system and hardware
– Interfaces between systems• Within the NASA Apollo project, interface errors were the
75% of all errors !
52
Different levels of integration testing
• Component integration testing– Testing the interactions among software components and
is done after unit testing• System integration testing
– Tests the interactions between different systems and may be done after system testing
– In this case, the developing organization may control only one side of the interface, so changes may be destabilizing
– Worst case: a series of systems that run on different platforms
• The greater the scope of integration, the more difficult it becomes to isolate failures to a specific interface
53
Two extreme approaches
• Big-bang integration testing– All components or systems are integrated simultaneously – After that everything is tested as a whole– Advantage: everything is finished before integration testing starts– Disadvantage: time-consuming and difficult to trace the cause of
failures with this late integration• Incremental integration testing
– All programs are integrated one by one– A test is carried out after each step– Advantage: defects are found early in a smaller assembly when
it is relatively easy to detect the cause– Disadvantage: it can be time-consuming since stubs and drivers
have to be developed and used in the test
54
Incremental testing approaches
• Top-down – Testing takes place from top to bottom, following the
control flow or architectural structure• e.g., starting from the GUI or the main menu
– Components or systems are substituted by stubs• Bottom-up
– Testing takes place from the bottom of the control flow upwards
– Components or systems are substituted by drivers• Functional incremental
– Integration and testing takes place on the basis of the functions or functionality, as documented in the functional specification
55
3: System testing
• Is concerned with the behavior of the whole system/product• It may include tests based on
– Risks and/or requirements specification – Business processes – Use cases, or other high level descriptions of system behavior – Interactions with the operating system and system resources
• Is most often the final test on behalf of development• Should investigate both functional and non-functional
requirements of the system• System testing requires a controlled test environment with
regard to control of the software versions (configuration management)
• The test environment should correspond to the final target or production environment as much as possible
58
4: Acceptance testing
• It comes after the system testing • Responsibility of the user or customer• Answer questions:
– Can the system be released? – What, if any, are the outstanding (business) risks? – Has development met its obligations?
• Executed in a test environment that isrepresentative of the production environment
• Focused on a validation type of testing• It is not necessarily the final level of testing
– E.g. a large-scale system integration test may come after the acceptance of a system
59
Test types: target of testing
• Test types are a means of clearly defining the objective of a certain test level for a program or project
• A test type is focused on a particular test objective• For example
– Testing of a function to be performed by the component or system
– A non- functional quality characteristic, such as reliability or usability
– The structure or architecture of the component or system – Testing changes, confirming that defects have been fixed
(confirmation testing, or re-testing) and looking for unintended changes (regression testing)
• Depending on its objectives, testing will be organized differently
60
ISO 25010 quality characteristics
ISO 25010 (ex 9126) defines eight quality characteristics and the subdivision of each quality characteristic into a number of sub-characteristics1. Portability2. Maintainability3. Security4. Reliability5. Functional suitability6. Performance efficiency7. Compatibility8. Usability
61
ISO 25010
Functional testing
• The function of a system (or component) is what it does• This is typically described in a requirements specification, a
functional specification, or in use cases• It is difficult to test against undocumented and implicit
requirements– some functions that are assumed to be provided
• Functional tests are based on functions described in documents or understood by the testers
• Functional tests may be performed at all test levels– e.g. test for components may be based on a component
specification• Functional testing considers the specified behavior and is
often also referred to as black-box testing– Even if black-box testing includes also non-functional testing
62
Functionality testing based on ISO 25010
• Functions are those that satisfy stated or implied needs
• The objective is to guarantee the existence of a set of functions and their specified properties
• Functional suitability– Completeness– Correctness– Appropriateness
63
Functionality testing based on ISO 25010 (ex 9126)
• functional suitability: • degree to which a product or system provides functions
that meet stated and implied needs when used under specified conditions
• functional completeness: • degree to which the set of functions covers all the
specified tasks and user objectives• functional correctness:
• degree to which a product or system provides the correct results with the needed degree of precision
• functional appropriateness: • degree to which the functions facilitate the
accomplishment of specified tasks and objectives
64
Non-functional testing
• Quality characteristics or non- functional attributes of the system
• How well or how fast something is done• We need to measure on a scale of measurement
– E.g., time to respond • Non-functional testing is performed at all test
levels
65
Non-functional testing based on ISO 25010
• Portability • Maintainability• Security• Reliability• Performance efficiency• Compatibility• Usability
66
Structural testing
• Is often referred to as white-box or glass-boxbecause we are interested in what is happeninginside the box
• It can occur at any test level– It tends to be mostly applied at unit and integration
testing and generally is less likely at higher test levels (e.g., acceptance testing)
• Good tool support to measure code coverage– At component level, and to a lesser extent at (component)
integration testing– Code coverage: the percentage of executable elements,
e.g., statements or decision outcomes that have been exercised
67
Confirmation and regression testing
• Testing of changes• If you have made a change to the software, the
way it functions, the way it performs (or both) and its structure have changed
• We look at the specific types of tests relating to changes, even though they may include all of the other test types
68
Confirmation testing (re-testing)
• A test run on a code that has been changed in order to fix a defect
• The test is the same that was run before and that causes the defect to emerge
• The test is executed in order to confirm that the defect has been fixed
• It is important to ensure that the test is executed in exactly the same way as it was the first time– Using the same inputs, data and environment
69
Confirmation testing: question
• If the test now passes does this mean that the software is now correct?
• We now know that at least one part of the software is correct– Where the defect was
• The fix may have introduced or uncovered a different defect elsewhere in the software
• The way to detect these unexpected side-effectsof fixes is to do regression testing
70
Regression testing
• Involves executing (all) test cases that have been executed before• The intent is checking that the system has not regressed
– It does not now have more defects in it as a result of some changes• It is common for organizations to have what is usually called a regression
test suite or regression test pack– They are designed to collectively exercise most functions in a system
(but not test each one in detail)– It is appropriate to have a regression test suite at every level of testing– All of the test cases in a regression test suite would be executed every
time a new version of software is produced and this makes them ideal candidates for automation
– Maintenance of a regression test suite should be carried out so it evolves over time in line with the software
71
Maintenance testing
• Once deployed, a system is often in service for years or even decades
• During this time the system and its operational environment is often corrected, changed or extended
• Testing that is executed during this life cycle phase is called maintenance testing
• Note that maintenance testing is different from maintainability testing, which defines how easy it is to maintain the system
72