Software Testing part I - dis.uniroma1.itsantucci/SW_Engineering/Material/05_A_Test_PartI.pdf ·...
Transcript of Software Testing part I - dis.uniroma1.itsantucci/SW_Engineering/Material/05_A_Test_PartI.pdf ·...
Software Testing part I
Lecturer: Giuseppe Santucci
Software testing
1. Basic concepts
2. Test that does not use the computer
3. Black box testing
4. White box testing
5. Test strategies
Test.2
1. – Basic concepts
• An example
• Test definition
• Testing principles
03Test.3
Test the following program
A Java program that:
• reads from the file "foo.txt", which is
generated by another program, three
integers, corresponding to the sides of a
triangle
• Print on the screen the triangle type:
– isosceles,
– scalene, or
– equilateral
03Test.4
Prepare some files to test it
// File Triangolo.java
import java.io.*;
public class Triangolo {
public static void main (String[] args) throws IOException {
String nome_file = “foo.txt";
FileInputStream istream = new FileInputStream(nome_file);
BufferedInputStream bis = new BufferedInputStream(istream);
int first = InOut.readInt(bis),
second = InOut.readInt(bis),
third = InOut.readInt(bis);
bis.close();
System.out.println(“Sides: "+first+' '+second+' '+third);
if (first == secondo && second == third)
System.out.println("Equilateral");
else if (first== second || second == third)
System.out.println("Isosceles");
else
System.out.println("Scalene");
}
}
Test.5
Did you considered the following cases?
1. A valid scalene (2,5,10 is wrong)
2. A valid equilateral
3. A valid isosceles (2,2,4 is wrong)
4. Three permutations of an isosceles (3,3,4 / 3,4,3 / 4,3,3)
5. One side equal to 0
6. A negative side
7. 3 integers >0 and one is the sum of the others two (1,2,3)
8. Test 7 permutations (1,2,3 / 1,3,2 / 3,1,2)
Test.6
Did you considered the following cases?(2)
9. 3 integers >0 and one is less than the sum of the
others two (2,5,10)
10.Test 9 permutations
11. 3 0s
12.One side as real
13.Only two sides
14.The file does not exist or it is not readable
15.Did you provide for each test the foreseen output?
Test.7
Results
According to Myers, [1979], on the basis of
the answers got from experienced
programmers, the media of Yes answers to
the previous questions is less than 8 (out of
15)
Testing, even a trivial program, is not a trivial
activity
Test.8
Moreover
• Myers does not consider the test in which the file
contains more than 3 integers
• or maxint ...
Test.9
• Revisions and other inspection activities are able to detect errors, but are not enough
• Detecting errors typically costs 30-40% of the total project (but the percentage may be much higher)
• testing is the process of using a program with the specific intent to find errors before delivery it to the end-user
Test.10
Testing definition
Test.11
Psychological issues
From the psychological point of view, it is a
destructive phase (all other phases, e.g., design,
are constructive):
1. the test is to run a program in order to discover
errors
2. a case testing is of good quality if it has a high
probability of discovering errors
3. a test case is successful if it found previously
unknown errors
03Test.12
General testing goals
Testing is the design and execution of test cases able
to detect errors:
Of different types
In a systematic way
As fast as possible
Using a predefined amount of resources
Testing has not the goal of demonstrating that the
software is error-free
Test.13
Testing limits
Testing cannot demonstrate that the software is
error-free
Example: Java program about triangles
One or more inputs: files with 3 integers (not
enough)
Number of different inputs: 32 bit integers
232232232=296
Time to execute all of them (brute force):
1 Million test/second (overestimated!)
276 seconds 251 years (about 1015)
Test.14
Testing principles
We have to define the output of a test case before
executing it
Ideally, we should get someone else to check our
work : another person is more likely to find the
errors
1. Psychological issues: finding error in own work
is a destructive process
2. Test cases will be driven by the knowledge of
the program structure
NOTE: it is not true for debugging
Test.15
Testing principles (2)
Test results have to be carefully analyzed
Test cases must be designed:
• For non valid and non foreseen
conditions
• For valid and foreseen conditions
Testing has to check if:
• The program does not what it is supposed
to do
• The program does what it should not to do
Test.16
Testing principles (3)
Avoid “use and thrown away” use cases
• You lose your investment
• You have to reinvent the tests
• As a consequence, tesing a new release of the
sw is less rigorouse
Do not plan test cases assuming (hoping?) that
you will not find errors! You need to allocate time
and resources for debugging
Design tests in advance (e.g., when requirements
are ready, see XP)
Testing principles (4)
Test.17
Number of found errors
Probability to find
other errors
The probability of the existence of errors in a
program grows with the number of errors already
found. The errors tend to cluster in certain sections
of code (Pareto principle: 80% of the errors were
due to 20% of code)
03Test.18
Testing principles (5)
Test cases rise from and must be connected
with user requirements
We start from the single functions (unit test) till
encompass the overall program structure
Brute force test is not possible testing is a
creative and stimulating activity
Test.19
Testing principles (6)
• Some empirical rules:
• A good test case is not redundant compared
to the previous ones (you risk to waste time
and resources)
• It is important to determine categories of test
cases
• A good test case is neither too easy nor too
complex
03Test.20
Design the SW with testing phase in mind
Testability: the sw is easy to test. It depends on
Observeability: test results are observable (e.g., internal
states and variables, the source code is available, …)
Controllability: testing can be automated. It is possible to fill
variables from outside and inspect them
Decomposability: level of modularity : it is possible to
independently test modules
Simplicity: only mandatory functions are implemented (not
gold plating), agreed usage of standard coding, modular
architecture
Pesticide paradox
• If the same tests are repeated over and over
again, eventually the same set of test cases will
no longer find any new bugs
• To overcome this pesticide paradox
– the test cases need to be regularly reviewed and
revised
– new and different tests need to be written to exercise
different parts of the software or system to potentially
find more defects
03Test.21
Overall testing strategies
Three different objectives:
• Highlight a pre-fixed number of defects, minimizing the cost
• To make the best use of a fixed budget
• Looking for the global minimum of the function of cost
03Test.22
2. – Testing without a computer
• Inspection and walkthrough
• Common errors
03Test.23
“Human based testing”
• Static code analysis• Such techniques are not substituting computer
based techniques and have to be used when the code is not running
• Experience shows that such techniques are very effective: – It seems that psychological aspects are not affecting
such a process– The sooner you discover an error the less the cost
03Test.24
03Test.25
Inspection e walkthrough
• the sw is examined by a group of people (3-5), including
the author(s)
• involved people have (typically) viewed the product before
the meeting
• the programmer comments, line by line, the program
• significant sets of input are selected
• The sw is simulated on each set
• the aim must be to find mistakes, not correct them (testing,
not debugging)
3. Black box testing
03Test.26
• Introduction
• Equivalence classes coverage
• Boundary value analysis (BVA)
03Test.27
Introduction
• Functional test (vs structural test)
• Uses the definition of the input-output behavior of programsto design test cases
• General principle: the design of test cases occurs
disregarding the internal structure of the sw and reasoning
only on requirements (test cases (should) must be designed
after requirement specification)
03Test.28
Introduction (2)
• (Typically, but not always) performed after thewhite box testing (not alternative butcomplementary to it)
• if properly conducted, can discover manyfunctional failures
• may also highlight the failure of some non-functional requirement (e.g., inadequateperformance)
Equivalence classes
Equivalence classes
• It is a common sense approach to testing
• Most testers practice it informally even though
they may not realize it
• The idea behind the technique is to partition a
set of input values into equivalence classes
– The system should handle them equivalently
Equivalence partitioning
• The equivalence-partitioning technique requires that
we need to test only one input value from each
equivalence class
• We are assuming that all the input values in one
class will be treated in the same way by the software
• If one input value in a class works, we assume all of
the input conditions in that partition will work
• Conversely, if one of the input value in a class does
not work, then we assume that none of the input
conditions in that class will work
Example 1/5
• A savings account in a bank earns a different rate of
interest depending on the balance in the account
• In order to test the software that calculates the
interest due, we can identify the ranges of balance
values that earn the different rates of interest
– A balance in the range $0 up to $100 has a 3% interest
rate
– A balance over $100 and up to $1000 has a 5% interest
rate
– Balances of $1000 and over have a 7% interest rate
Example 2/5
• We would identify four equivalence classes
– Three valid equivalence classes and one invalid class
• Important:
– Not only we test what is in our specification
– We also think about things that haven't been
specified
– In this case we have thought of the situation where
the balance is less than zero
– We haven’t identified an invalid class on the right
Invalid class Valid (for 3% interest) Valid (for 5%) Valid (for 7%)
-? -- -$0.01 $0.00 -- $100.00 $100.01 -- $1000.00 $1000.01 -- ?
Example 3/5
• Do you identify additional classes?
Invalid class Valid (for 3% interest) Valid (for 5%) Valid (for 7%)
-$0.01 -- $0.00 $0.01 -- $100.00 $100.01 -- $1000.00 $1000.01 -- ?
Example 4/5
• Do you identify additional class?
• What about non-numeric numbers /format
errors?
• Overflow?
Invalid class Valid (for 3% interest) Valid (for 5%) Valid (for 7%)
-$0.01 -- $0.00 $0.01 -- $100.00 $100.01 -- $1000.00 $1000.01 -- ?
Example 5/5
• When we say a class is invalid, it does not mean that it represents a value that cannot be entered by a user or a value that the user is not supposed to enter
• It just means that it is not one of the expected inputs for this particular field
• The software should correctly handle values from the invalid class – E.g. By replying with an error message such as Balance must be
at least $0.00
• Note also that the invalid class may be invalid only in the context of crediting interest payments.– E.g. An account that is overdrawn will require some different
action.
Identifying equivalence classes
• Distinguish between valid and invalid classes
• Classes are identified by carefully analyzing
input conditions
• From a sentence in the specification derive one
or more valid or invalid classes
Identifying Equivalence classes
• List the classes
• Each class has an identification code
• Useful criteria for partitioning
– Intervals of values
– Number of values
– Sets of values
– Constraints
Intervals of values 1/2
• If an input condition specifies an interval of
values, identify
– One valid class
• Values within the interval
– Two invalid classes
• Values below the minimum boundary
• Values above the maximum boundary
Intervals of values 2/2
Condition Valid class Invalid classes
Customer code
between 1 and 999
1<= Customer code<= 999
1
Customer code< 1
Customer code> 999
2
3
Number of values 1/2
• If an input condition specifies the number of
values, identify
– One valid class
• For a number of values between the minimum and the
maximum specified
– Two invalid classes
• One for values lower than the minimum
• One for values greater than the maximum
Number of values 2/2
Condition Valid class Invalid classes
The number of product must not exceed 6
1<= Number of products<= 6
4
Number of products< 1
Number of products> 6
5
6
Where this 1 comes from?
Constraints 1/2
• If an input condition specifies something like
must be, identify
– One valid class
• For elements that satisfy the constraint
– One invalid class
• For elements that brake the condition
Constraints 2/2
Condition Valid class Invalid class
The type of the product must be “Manufactured”
Product type= “Manufactured”
7
Product type!= “Manufactured”
8
Set of values 1/2
• If an input condition specifies a set of values,
identify
– A number of valid classes equal to the set cardinality
• If the element of the set are treated in the same way by the
system they can be grouped in the same partition
– One invalid class
• For elements that are not in the set
Set of values 2/2
Condition Valid class Invalid class
The type of customer
can be either “old” or “new”
Customer type=“new”
9 Customer type not in {“old”, “new”}
11
Customer type=“old”
10
Test cases design
• Valid classes
– To design a number of test cases sufficient to cover
all valid classes
– Constraint: to maximize the number of classes
covered by each test case (to minimize the number of
test cases)
• Note: sometimes such criterion may be in
contrast with easy localization of error
47
Valid classes: example
48
Conditions Valid Invalid
Equivalence classes
1. Customer code between 1 and 999
2. Number of products must not exceed 6
3. Type of product must be “Manufactured”
4. Customer type can be either “old” or “new”
1 <= Customer code<= 999 (1)
1<= Number of products<= 6 (4)
Product type= “Manufactured” (7)
Customer type=“new” (9)
Customer type=“old” (10)
Customer code<1 (2)
Customer code>999 (3)
Number of products< 1 (5)
Number of products> 6 (6)
Product type= “Imported” (8)
Customer type != {“new”, “old”} (11)
Test case:
Customer code = 5
Number of products = 4
Type of product =“Manufactured”
Customer type = “new”
Covers classes (1) (4) (7) (9)
Valid classes: example
49
1. Customer code between 1 and 999
2. Number of products must not exceed 6
3. Type of product must be “Manufactured”
4. Customer type can be either “old” or “new”
1 <= Customer code<= 999 (1)
1<= Number of products<= 6 (4)
Product type= “Manufactured” (7)
Customer type=“new” (9)
Customer type=“old” (10)
Customer code<1 (2)
Customer code>999 (3)
Number of products< 1 (5)
Number of products> 6 (6)
Product type= “Imported” (8)
Customer type != {“new”, “old”} (11)
Test case:
Customer code = 5
Number of products = 4
Type of product =“Manufactured”
Customer type = “old”
Covers classes (10)
Conditions Valid Invalid
Equivalence classes
Test cases design cont.d
• Invalid classes
– To design a number of test cases sufficient to cover
all invalid classes
– Constraint: each test has to cover one and only one
invalid class
• In some cases that is not possible and the test is designed
minimizing the number of covered invalid classes
• Invalid classes have to be covered individually in
order to avoid errors that hide each other
50
Invalid classes: example
51
1. Customer code between 1 and 999
2. Number of products must not exceed 6
3. Type of product must be “Manufactured”
4. Customer type can be either “old” or “new”
1 <= Customer code<= 999 (1)
1<= Number of products<= 6 (4)
Product type= “Manufactured” (7)
Customer type=“new” (9)
Customer type=“old” (10)
Customer code<1 (2)
Customer code>999 (3)
Number of products< 1 (5)
Number of products> 6 (6)
Product type!= “Manufactured” (8)
Customer type != {“new”, “old”} (11)
Partition
covered(2) (3) (5) (6) (8) (11)
Customer code 0 1000 7 7 7 66
Number of
products 1 1 0 9 3 4
Type of products Manufactured Manufactured Manufactured Manufactured Imported Manufactured
Customer type New New New New New recent
Conditions Valid Invalid
Equivalence classes
Equivalent classes for the triangle
program
• Criterion: interval of values– Every side has a positive value -> the interval has only the
minimum boundary
• Valid class(VCi):– VC1: 0 < a <= 100 VC2: 0< b <= 100 VC3: 0 < c <= 100
• Invalid classes(ICi)– IC4: a <= 0 IC5: b <= 0 IC6: c <= 0
– IC7: a > 100 IC8: b > 100 IC9: c > 100
• Note: – 100 is an arbitrary value, we could use MAXINT
– It is important to always define a maximum boundary• E.g., by using values that can be represented
52
Equivalent partitions for the triangle program
cont.d
• Criterion: number of values
– The txt file must contain exactly three integers
– I.e., min number = max number
• Valid class
– VC10: exactly three integers
• Non valid classes
– IC11: more than three integers
– IC12: less than three integers
53
Equivalent partitions for the triangle program
cont.d
• Criterion: set of values
– There are three types of triangles
• Valid classes
– VC13: equilateral triangle
– VC14: isosceles triangle
– VC15: scalene triangle
• Non valid class
– IC16: it is not a triangle
54
Equivalent classes for the triangle
program cont.d
55
• Criterion: constraints
– The txt file contains integers
• VC17 : the file contains integers
• IC18 : the file does not contain integers
– The file exists
• VC19 : the file exists
• VC20 : the file does not exist
Test cases for the “triangle program”
56
Test cases Covered equivalence classes Output
1) 4, 4, 4 VC1, VC2, VC3, VC10, VC13, VC17,
VC19
Equilateral triangle
2) 2, 2, 1 VC14 Isosceles triangle
3) 3, 4, 5 VC15 Scalene triangle
4) -3, 4, 5 IC4 Err: a non-positive
5) 3, -4, 5 IC5 Err: b non-positive
6) 3, 4, -5 IC6 Err: c non-positive
7) 150, 4, 5 IC7 Err: a greater than max
8) 3, 150, 5 IC8 Err: b greater than max
9) 3, 4, 150 IC9 Err: c greater than max
10) 3, 4, 5, 2 IC11 Err: too many input data
11) 3, 4 IC12 Err: too few input data
12) 1, 1, 5 IC16 Err: non-valid triangle
13) 3.0, 4, 5 IC18 Err: a is not integer
14) No file IC20 Err: the file does not exist
Boundary value analysis
03Test.57
Boundary value analysis (BVA)
• It is based on testing at the boundaries between
classes
• Note that we have both valid boundaries (in the
valid equivalence classes) and invalid
boundaries (in the invalid equivalence classes)
• Practical experience showed that boundary
value analysis lead to beneficial results during
testing
Boundary value analysis (BVA) cont.d
• You have to consider values:
– Directly on the boundaries
– Immediately above them
– Immediately below them
• of equivalence classes of input and output
Boundary value analysis (BVA) cont.d
• It means that we are looking for values “close” to one
boundary (not a formal definition)
• BVA differs from equivalence classes partitioning in two
aspects
– You choose one or more values around each
boundary as representatives of the equivalence
partition
– Test cases are designed by considering also results
(output)
• Such technique requires a good knowledge of the
application domain and some creativity
Boundary value analysis (BVA) cont.d
Guidelines:
1. Interval of values specified by an input conditionEquivalent classes:
– Valid, for values that equal the interval’s boundaries and internal values “close” to the boundary
– Invalid, for values immediately below the minimum and above the maximum of the interval
2. Number of values specified by an input conditionEquivalent classes:
– Valid, for the minimum and maximum of the values and internal values “close” to the minimum and maximum
– Invalid, for numbers immediately below the minimum value and immediately above the maximum value
Boundary value analysis (BVA) cont.d
3. Repeat number 1. for each analogous output
condition
4. Repeat number 2. for each analogous output
condition
5. If input and/or output conditions include an
ordered set of elements you have to consider
also the first and the last elements of the set
6. Ask yourself if there are input conditions that
generate boundaries
Boundary value analysis (BVA) cont.d
• The output domain
• You have to carefully analyze the output domain considering that
– It is not always the case that boundaries of the input
domain correspond to the boundaries of the output
domain as they are determined by different
circumstances
– It is not always possible to generate a result out of
the boundaries of the output valid domain
Example of constraint
Requirements
Equivalence partitions
Valid Invalid
Sides are integers andthe sum of two facets isgreater than the thirdone
A + B > C A + B <=C
Boundary: A+B-1=C
Test cases
t1) A=4, B=4, C=6 : OK
t2) A=3, B=4, C=6 : OK
t3) A=3, B=2, C=6 : Error
[
C A+B-1
values
t3 t2 t1
BVA on IF THEN conditions
• We consider BVA on an If Then condition when it
contains disequations
• IF speed>50 THEN gear>2 (Valid partition)
• IF speed>50 THEN gear<=2 (Non valid partition)
– We negate ONLY the Then part !
• Valid partition:
– Minimum speed value that makes speed>50 TRUE : 51
– Minimum gear value that makes gear>2 TRUE: 3
65
BVA on IF THEN conditions
• Speed 50,51,52
• Gear 2,3,4
• Nine test cases error message!
• 50,2 50,3 50,4 51,2 51,3 51,4 52,2 52,3 52,4
66
x x x
x x x
Remarks
• Test cases designed by applying BVA catch very
common errors
• Random generation of test cases does not
identify such errors
• BVA is one of the most useful techniques in test
case design
• It seems easy, almost banal but it is not!
– Boundary conditions may be very subtle