Unit 2: Functional Testing Reading: PJR Ch. 5
-
Upload
softwarecentral -
Category
Documents
-
view
746 -
download
0
description
Transcript of Unit 2: Functional Testing Reading: PJR Ch. 5
Unit 2:Functional Testing
Reading: PJR Ch. 5
References:Software Testing and Analysis: Process, Principles, and Techniques
(Pezzè and Young) Why Programs Fail (Zeller)
Software Testing: Principles and Practices(Desikan and Ramesh)How We Test Software at Microsoft (Page, Johnston, and Rollison)
Functional Testing Functional testing uses the specification
(formal or informal) to systematically create tests
Timely Can write tests before code is written
Effective Finds faults that can elude other approaches Best for catching “missing logic” bugs
Widely applicable Any description of program behavior serving as spec Any level of granularity from unit to system testing
2
Functional Testing Goals Problem: Cannot test all possible input combinations Need to choose tests…
Intelligently Systematically Thoroughly
Key idea: Partition the input space into equivalence classes. All inputs within the equivalence class have the same
functionality. Sufficient to run one test in each equivalence class (not
really). Can be tricky to divide the input space into equivalence
classes. Better to err on the side of too many tests than too few.
3
Systematic Functional Testing
4
Failure (valuable test case)
Th
e s
pace
of
poss
ible
in
pu
t valu
es
No failure
Functional Testing Approach1. Identify independently testable features in the
program.2. Find equivalence classes for each feature
identified in step 1.3. Apply boundary value analysis to create one
or more test case specifications for each equivalence class.
4. Select other potential interesting values based on previous experience, intuition.
5. Apply combinatorial analyses to create test case specifications that combine the independently testable features.
5
Independently Testable Features Decompose subject under test into
independently testable features (ITF). For functions:
Each parameter may be an ITF. Closely coupled parameters may be considered as
one ITF. For integration testing:
Each unit is an ITF. For system testing:
ITFs are exposed through user interfaces or APIs. ITFs refer to different “phases” of the program.
6
Class Problem: Testable Features Consider a multi-
function calculator What are the
independently testable features?
7
Equivalence Classes Need to divide the input space for each ITF
into equivalence classes. Based on the desired / specified functionality
of the subject under test. Examples:
Computer science students need to meet with their advisor if their GPA is less than 2.75 or if they are in their first year at SU.
Income tax rate is dependent on income. Volume discount if you buy more than 1000 units.
8
Class Problem: Next Date The next date
program accepts a date as an input and returns the next date.
It only works on dates in the years 1700-3000.
What are equivalence values for each of the three inputs?
9
Boundary Value Analysis
For ordered inputs, history suggests that values near the boundary are most error prone.
What are good values for an input that expects integers from -10 to 10?
10
Possible test case
Boundary Value Analysis For each boundary n, want to have three cases:
n – 1, n, n +1 For a range of values from x to y, have at least
six cases: x – 1, x, x + 1, y – 1, y , y + 1
Some values are outside of the range: May also be part of another equivalence class
(worry about redundancy later) May be illegal - should check to make sure it’s illegal May be impossible to test
Example: drop-down menu Is there another way to enter the value?
11
Boundaries Boundary analysis can be applied to more
situations than handling numeric data. Examples?
12
Hidden Boundaries What about “unbounded” data types?
Integers with no specified range Size of “growing” data structures
What about hidden boundaries? The size of the window in a GUI Data type limits Memory limits Operating system limits
13
Special Values Within an equivalence class, we have
identified values near the boundary. However, we don’t want to necessarily stop there.
Should have at least one test in the middle. Sometimes the boundaries are either trivial or
handled differently in code. Using prior experience and intuition, look at
other values likely to cause problems: 0 and negative numbers values of a different type (if possible) n +1 digit numbers when only n digits are needed other values based on the subject under test
paying close attention to inputs considered invalid
14
Class Problem: Next Date
15
Month Day Year
Catalog Based Testing Catalogs capture the experience of test
designers by listing important cases for each possible type of variable
A catalog lists kinds of elements that can occur in a specification
Each catalog entry is associated with a list of generic test case specifications
Example: catalog entry Boolean two test case specifications: true, false label if applicable only to: input, output, both
16
A simple catalog (part I) Boolean
True in/out False in/out
Enumeration Each enumerated value in/out Some value outside the enumerated set in
Integer large negative number in/out small negative number in/out 0 in/out small positive number in/out large positive number in/out
17
A simple catalog (part II) Range L ... U
L-1 in L in/out A value between L and U in/out U in/out U+1 in
Numeric Constant C C in/out C–1 in C+1 in Any other constant compatible with C in
Non-Numeric Constant C C in/out Any other constant compatible with C in Some other compatible value in
18
A simple catalog (part III) Sequence (such as a sorted linked list)
Empty in/out A single element in/out More than one element in/out Maximum length (if bounded) or very long in/out Longer than maximum length (if bounded) in Incorrectly terminated in
Traversal for item P in list P occurs at beginning of list in P occurs in interior of list in P occurs at end of list in P does not occur in list in
Insertion of item P in list P inserted at beginning of list out P inserted in interior of list out P inserted at end of list out
19
Combinatorial Testing Programs / units generally have more than
one input.
Test cases need to take care of combinations of different inputs. A program may fail on a particular combination of
inputs
Combinatorial testing: systematically generate combinations to be tested Uses the representative values for each ITF.
20
Example: Next Date How many different combinations are there?
Month: Day: Year: Total:
How could you legitimately reduce the number of combinations?
21
Introducing Constraints Generally each invalid input case needs to be
executed once (with all other inputs valid) Overkill to test things like April 0, 3005
Some combinations of valid input values are invalid Example: February 31 Suffices to test each of these once
Look for uninteresting cases – cases that can be grouped together. Example: The year does not matter except in Feb. & Dec.
Common constraints: Error or invalid conditions Input x is ignored when input y is 0 “Empty” data structure
22
Pairwise Testing Even when intelligent constraints are applied, the
number of resulting combinations can be too large.
Pairwise Testing: generate combinations that efficiently cover all pairs of classes Rationale: most failures are triggered by single values or
combinations of a few values. Efficient – fewest number of test cases Covering pairs reduces the number of test cases, but
reveals most faults N-way Combinatorial Testing: generate all n-
tuples of input combinations N=2 for pairwise testing
23
Combinatorial Testing of Configurations A common use of combinatorial testing is for
configuration testing: Operating system Environment variables Other pieces of software
Example: Browser could be “IE” or “Firefox” Operating system could be “Vista”, “XP”, or
“Windows 7” Systematically generate combinations to be tested
IE on Vista, IE on XP, Firefox on Vista, Firefox on OSX, ...
24
Example: Pairwise Testing How many tests are needed such that each
pair of inputs is tested at least once? OS: XP, Vista, Windows 7 Browser: IE, Firefox Version of Library X: 1.0, 1.1, 1.2, 2.0 Has program Y installed: Yes, No
25
Example: One possible solution
26
Library X OS Browser Program Y?
1.0 XP IE Y
1.0 Vista Firefox Y
1.0 Windows 7 Firefox N
1.1 XP Firefox Y
1.1 Vista Firefox N
1.1 Windows 7 IE N
1.2 XP IE N
1.2 Vista IE Y
1.2 Windows 7 Firefox Y
2.0 XP Firefox N
2.0 Vista IE N
2.0 Windows 7 IE Y
Pairwise Testing Not always the case where the number of
combinations is the product of the number of cases of the two inputs with the most cases.
In general, generating a list of combinations is hard to do manually. Too many combinations Too many constraints to contend with
Process can be automated using tools Tradeoff: manually specifying constraints vs. more
tests
27