här
-
Upload
softwarecentral -
Category
Documents
-
view
217 -
download
0
Transcript of här
Applied Software Testing
A crash course in applying SW testing strategies
Birgitta Lindström
2
Overview
Introduction exercise (30 min)
Test Strategies (50 min) White-box Black-box
Assignment introduction (10 min)
3
Exercise
Implement a program that classifies triangles given
Input: 3 parameters that represent the length of the edges Example : Triclass 10 10 10
Output: One of four classes Equilateral Triangle (liksidig) Isosceles Triangle (likbent) Triangle Error
10 10
10
4
You should
Write the program on a paper Use pseudo code
Similar to some well-known sequential programming language…
Challenge a friend Exchange programs Try to come up with a set of test cases
that makes your friends program fail
5
Recap : Purpose of testing
Find faults! Provoke failures
Deviations from Specification or expected behavior
Gain confidence in product quality Remember : Testing is NOT proving
6
Testing methodology STEP 1 : Test Case Generation
Select test cases according to strategy
Test case is input and expected outcome
Input is e.g., input values, abstract parameters, sequences of button pushes, flag values, DBS contents, etc…
Outcome includes output (external behavior) but often also internal behavior (e.g., traced states)
Expected outcome is always taken from “specification”
STEP 2: Test Case Execution
Run the program with selected test-cases (i.e., the test suite)
STEP 3: Analysis
Compare expected outcome with actual outcome
Decide whether actual outcome is correct or not
7
Test criteria
We cannot test everything ! Unfortunately…
Test requirements Specific demands that must be satisfied or
investigated during testing Execute a specific source code statement Check if a particular error code can be produced
Test criterion Specifies class of test requirements
Execute all source code statements Check if all error codes can be produced
8
Test coverage
Expressed in context of test criterion
Example
Total #test requirementsCoverage =
Fulfilled #test requirements
#Executed statements = 922Total #statements = 1000Statement Coverage = = 92,2%
9
Testing strategies
Structural strategies (white-box) Implementation based
Statement coverage Decision coverage MC/DC
Functional strategies (black-box) Specification based
Equivalence partitioning Boundary-value analysis Combination strategies
10
Statement coverage
Each statement should be executed by at least one test case
Minimum requirement for code coverage
int silly_func(int a, int b){ int i=4; if (a<b) { a=a*14; } if (a<=b){ b=42; } else { i=a-b; } return i*b; }
silly_func(1,2)
silly_func(1,17)
11
Decision coverage
Also called ‘branch coverage’
All decisions with true and false value
“Subsume” all statements coverage
int silly_func(int a, int b){ int i = 4; if (a<b) { a=a*14; } if (a<=b){ b=42; } else { i=a-b; } return i*b; }
silly_func(1,2)silly_func(1,17)
true
false
true
true
silly_func(5,1)
false
false
12
Modified Condition/Decision Coverage (MC/DC)
For each variable x in a logical expression B, let x decide the value of B try x=false try x=true
Example:
if(x1 and x2) {S
}
Used for safety critical applications
x1 x2 x1 and x2
true true true
false true false
true true true
true false false
B
(x1 and true)
x1 decides B :
(true and x2)
x2 decides B :
13
Using white-box test strategies1 Test Case Generation (e.g. statement coverage)
1 Choose initial input data (e.g., random)
2 Execute and check coverage
3 Select a requirement (e.g. a statement) you have not yet covered
4 Select input data that will fulfill the uncovered requirement (e.g. execute that statement)
5 Iterate from 2 until desired code coverage level is reached
6 Minimize test suite (if possible)
7 For each test case, derive expected outcome from specification
2 Test Case Execution and analysis (see slide 6)
14
Using black-box strategies
Input parameter modeling Identify input parameters
Input values, sequences of button pushes, mouse movements, sensor signals, etc
System state, flags, DBS content, cache content, etc Abstract parameters based on functionality and behavior
Equivalence Partitioning Divide each parameter into classes
Boundary Value Analysis Define parameter values for closer analysis
Generic representation of test problems Combine classes with a combination Strategy
All Combinations Each Choice Pair-Wise Base Choice
ClassParam
1 2 3
A
B
C
15
Equivalence partitioningFor each parameter Identify classes of values based on
specification Assumption: if one input from class s leads
to a failure, then any input from class s will lead to the same failure
Chose a representative value from each class (e.g., by random)
Form test cases with the representative values according to a criterion (e.g., pair-wise)
16
Example : Partitioning a parameterA command-line program : SquareInput : A numberOutput : The square roots of that number
10^6 to 10^999 1 to 10^60
Class 1Negative
Class 2Zero
Class 3Small int
Class 4Large int
-10^999 to -1
17
Example: Partitioning with several parameters
Type Integer Real String
Size Small Large Zero
Negative Yes No
Square Yes No
18
Example : Representative valuesA command-line program : SquareInput : A numberOutput : The square roots of that number
17676500 001 420
Class 1NegativeIntegers
Class 2Zero
Class 3Small integers
Class 4Large integers
-5003
Test-CasesSquare -5003Square 0Square 42Square 17676500001…
19
Boundary value analysis
Identify boundaries in input and output
For each boundary value : Select one value from each side of
boundary (as close as possible)
Form test cases with the chosen values
20
Example : Boundary value analysis
10^999 0-10^999
Class 1NegativeReal
Class 2Zero
Class 3Small real
Class 4Large real
Test-CasesSquare -10^999Square -0,0000001Square 0Square 0,0000001…
10^6
21
Combination strategies
Equivalence partitioning and boundary analysis give representative values for input parameters
Test case often contain more than one input parameter
Combination strategies help to combine parameter values!
22
Example : A direct combination of input parameters
Triclass( , , )
100 001 420
-1257.0089 1876.897897896786875812343.089876
Class 1 Class 2 Class 3 Class 4
Class 5 Class 6 Class 7
-1532
23
Example : A direct combination of input parameters
Triclass( , , )
100 001 420
-1257.0089 1876.897897896786875812343.089876
Class 1 Class 2 Class 3 Class 4
Class 5 Class 6 Class 7
-1532
24
Example : A direct combination of input parameters
Triclass( -1532 , , )
100 001 420
-1257.0089 1876.897897896786875812343.089876
Class 1 Class 2 Class 3 Class 4
Class 5 Class 6 Class 7
-1532
Class 1
25
Example : A direct combination of input parameters
Triclass( -1532 ,43.089876, )
100 001 420
-1257.0089 1876.897897896786875812343.089876
Class 1 Class 2 Class 3 Class 4
Class 5 Class 6 Class 7
-1532
Class 1 Class 6
26
Example : A direct combination of input parameters
Triclass( -1532 ,43.089876, 100 001)
Class 1 Class 4Class 6
Sometimes we write this as
Denoting the equivalence classes rather than the particular values
Test case
A B C
TC1 1 6 4
27
Generic test case representation
Test case
A B
TC1 1 1
TC2 1 2
TC3 1 3
TC4 1 4
This allows us to define tests sets that aregeneral for all testing problems that have the same number of parameters and values
For example, two parameters with three and four classes always give following twelve test cases,assuming all combinations
Test case
A B
TC5 2 1
TC6 2 2
TC7 2 3
TC8 2 4
Test case
A B
TC9 3 1
TC10 3 2
TC11 3 3
TC12 3 4
28
Decoding generic test cases
Print_strX Input : String and Integer Output : A string containing each letter in the
original string X times
Class # Parameter class Representative value
1 Empty string “”
2 String with white-space “w h i t e space”
3 String without white-space “yeaha”
Test case
A B
TC1 1 3print_strX(“”,)
29
Decoding generic test cases
Print_strX Input : String and Integer Output : A string containing each letter in the
original string X times
1230002994210
Class 1NegativeIntegers
Class 2Zero
Class 3Small integers
Class 4Large integers
-64566
Test case
A B
TC1 1 3print_strX(“”, 421 ) print_strX(“”,
30
Testing Criteria for Combination Strategies
Assume any function with 3 parameters A, B, and C with 2, 2, and 3 values
For example : my_func(bool A, int B, string C);
Parameter Value 1 Value 2 Value 3
A true false
B Positive Integers Negative Integers
C Empty string String without white-space
String with white-space
31
Testing Criteria for Combination Strategies
Assume any function with 3 parameters A, B, and C with 2, 2, and 3 values
For example : my_func(bool A, int B, string C);
Parameter Value 1 Value 2 Value 3
A true false
B Positive Integers Negative Integers
C Empty string String without white-space
String with white-space
TC : 2, 2, 1 my_func(false,-21,””)
32
Combination strategy : All Combinations
Assume any function with 3 parameters A, B, and C with 2, 2, and 3 values Gives
2*2*3 = 12 TC Grows quickly
E.g 3*3*4 = 36
A B C
TC1 1 1 1
TC2 1 1 2
TC3 1 1 2
TC4 1 2 1
TC5 1 2 2
TC6 1 2 3
TC7 2 1 1
TC8 2 1 2
TC9 2 1 3
TC10 2 2 1
TC11 2 2 2
TC12 2 2 3
Problem : Very expensive !
33
Combination strategy : Each choice
Each parameter value (class) should occur in at least one test case
Test case
A B C
TC1 1 1 1
TC2 2 2 2
TC3 any any 3
my_func(true,341,””);
my_func(false,-4622,”hej”);
my_func(false,4622,”hej Monika”);
Problem : Few test cases / combinations poor testing…
34
Combination strategy: Pair-wise
Each pair of chosen values occurs in at least one test case
Efficiently generated by ‘Latin squares’
Can detect failures due to specific pairs of input values
1 2 3
3 1 2
2 3 1
Latin square
ARow
BCol
CValue
TC1 1 1 1
TC2 1 2 2
TC3 1 (3) 3
TC4 2 1 3
TC5 2 2 1
TC6 2 (3) 2
TC7 (3) 1 2
TC8 (3) 2 3
TC9 (3) (3) 1
Problem : What if we have more than 3 parameters ?
Row=2
Column=1
Value=3
35
A mathematical property that is exploited is “orthogonality”. Two Latin Squares are orthogonal if the contents in each intersection is unique. Orthogonal Latin Squares preserve the pair-wise coverage criterion.
1 2 3
1
12
2
3
3
1 2 3
3
23
1
1
2
11 22 33
13
1223
21
31
32
Combination strategy: Pair-wise
36
Combination strategy: Pair-wise
Each pair of chosen values occurs in at least one test case
Efficiently generated by latin squares
Can detect failures due to pairs of input values
1 1 2 2 3 3
3 2 1 3 2 1
2 3 3 1 1 2
2 Orthogonal Latin squares
ARow
BCol
CValue 1
DValue 2
TC1 1 1 1 1
TC2 1 2 2 2
TC3 1 3 3 3
TC4 2 1 3 2
TC5 2 2 1 3
TC6 2 3 2 1
TC7 3 1 2 3
TC8 3 2 3 1
TC9 3 3 1 2
37
Combination strategy : Base Choice
For each parameter, define a base choice, i.e., the most likely value
Let this be base test case and vary one value at a time until all other classes has been chosen
In the example, let (1, 2, 2) be base choices
A B C
TC1 1 2 2
TC2 2 2 2
TC3 1 1 2
TC4 1 2 1
TC5 1 2 3
38
Equivalence partitioning and combination strategy problems
If equivalence partitioning are based on independent input parameters A lot of functionality may never be tested
For example : What is probability that the triclass function
finds an error in classifying a equilateral triangle ?
Success of combination strategies depend a lot on how equivalence partitioning is done Parameters must sometimes be more abstract Combine properties on combined inputs instead of
independent classes of input values !!
39
Example : Combination testing of Triclass
“Parameter” “values”
a+b>c True False
# sides of equal length 3 2 0
Value Ranges 0< >0 &&<=65536
>65536
Number of decimals 0 >0 &&< 5
>5
a b
c
40
Example : Combination testing of Triclass
“Parameter” “values”
a+b>c True False
# sides of equal length 3 2 0
Value Ranges 0< >0 &&<=65536
>65536
Number of decimals 0 >0 &&< 5
>5
a b
c
TCn = (1,2,2,1) e.g. Triclass(5,5,9)
41
Example : Combination testing of quickSort
“Parameter” “values”
File exists True False
Size of file 0kb 1Mb 2gb
Type of rows Numbers Strings Numbers +Strings
Number of similar signs from beginning
0 5 all
TCn = (1,2,2,1) e.g. quickSort testfile1221.txt
A command-line program : quickSortInput : A text file with rows ending with ;Output : A file with sorted rows
42
Using black-box test strategies
1. Test Case Generation1. Do equivalence partitioning or use some other
method to find good parameters values
2. Select a combination strategyE.g. Pair-wise or base-choice
3. Generate test cases using the selected methodIn the generalized format
4. Select input data within the specified classes for each parameter value
5. For each test-case, derive expected output from specification
2. Test Case Execution (see slide 6)