Post on 31-Dec-2015
Software Engineering : A Practitioner’s Approach
Software Testing Strategies
Dr. Thomas E. HicksComputer Science Department
Trinity University
Thanks To Ian Sommerville & Roger Pressman For Much Of The Slide Content
Chapter 13
Software Testing
Software Testing is the process of exercising a program with the specific intent of finding errors prior to delivery to the end user.
Testing is the exposure of a system to trial input to see whether it produces the correct output
What Testing Shows 4 Things!
1. errors1. errors
2. requirements conformance2. requirements conformance
3. performance3. performance
4. an indication4. an indicationof qualityof quality
What is a “Good” Test?
1. A Good Test has a high probability of finding an error
2. A good test is not redundant.
3. A good test should be “best of breed”
4. A good test should be neither too simple nor too complex
Test Case Design
""BugsBugs lurklurk inin cornerscorners andand congregatecongregate atat boundariesboundaries ..." ..."
BorisBoris BeizerBeizer
OBJECTIVEOBJECTIVE
CRITERIACRITERIA
CONSTRAINTCONSTRAINT
to uncover errorsto uncover errors
in a complete mannerin a complete manner
withwith aa minimumminimum ofof efforteffort andand timetime
Software Testing: The Basics of the Trade (Kindle Edition) by Gaia Asher
The Art of Software Testing, Second Edition by Glenford J. Myers, Corey Sandler, Tom Badgett, and Todd M. Thomas
Software Quality Approaches: Testing, Verification, and Validation: Software Best Practice 1 (Software Best Practice, 1) by Michael Haug, Eric W. Olsen, and Luisa Consolini
Professional Software Testing with Visual Studio 2005 Team System: Tools for Software Developers and Test Engineers (Programmer to Programmer) by Tom Arnold, Dominic Hopton, Andy Leonard, and Mike Frost (Paperback - Sep 11, 2007)
Software Testing (2nd Edition) by Ron Patton
Lessons Learned in Software Testing by Cem Kaner, James Bach, and Bret Pettichord
Foundations of Software Testing: Certification by Dorothy Graham, Erik van Veenendaal, Isabel Evans, and Rex Black
Testing Computer Software, 2nd Edition by Cem Kaner, Jack Falk, and Hung Q. Nguyen
Software Testing Foundations: A Study Guide for the Certified Tester Exam, 2nd Edition by Andreas Spillner, Tilo Linz, and Hans Schaefer
Pragmatic Software Testing: Becoming an Effective and Efficient Test Professional by Rex Black
Managing the Testing Process: Practical Tools and Techniques for Managing Hardware and Software Testing by Rex Black
Software Testing Fundamentals: Methods and Metrics by Marnie L. Hutcheson
Systematic Software Testing by Rick D. Craig and Stefan P. Jaskiel
A Practitioner's Guide to Software Test Design by Lee Copeland
41.
Software Testing: A Craftsman's Approach, Third Edition by Paul C. Jorgensen (Hardcover - Feb 15, 2008)
Component-Based Software Testing with UML by Hans-Gerhard
Software Testing and Analysis: Process, Principles and Techniques by Mauro Pezze and Michal Young
Software Testing In The Real World: Improving The
Process (ACM Press) by Edward Kit
Software Testing: Testing Across the Entire Software Development Life Cycle by Gerald D. Everett and Raymond, Jr. McLeod
Who TestsSoftware?
Who Tests the Software?
DeveloperDeveloper Independent TesterIndependent Tester
Understands the system Understands the system but, will test "gently" and, but, will test "gently" and, is is driven by "delivery"driven by "delivery"
MustMust learn about the system,learn about the system,but, will attempt to break itbut, will attempt to break itand, is and, is driven by qualitydriven by quality
1. Misplaced Faith: Those Who Write The Software generally don’t believe they will find any faults and generally won’t find as many
2. Must Fix Faults: Testers Who have to fix any faults they find do not tend to find as many
3. Mixed Goals: Those Who Write The Software want the code to be fault free but effective testers must want to find faults
3 Reasons The Software Developer Should Not (be
the only one to )Test His/Her Own Software
DeveloperDeveloper
4 Major TestingPhases
Unit/ComponentUnit/ComponentTestingTesting
IntegrationIntegrationTestingTesting
Final/Beta/ValidationFinal/Beta/ValidationTestingTesting
System/AlphaSystem/AlphaTestingTesting
4 Major Testing Phases
Unit/ComponentUnit/ComponentTestingTesting
IntegrationIntegrationTestingTesting
Final/Beta/ValidationFinal/Beta/ValidationTestingTesting
System/AlphaSystem/AlphaTestingTesting
1. Unit/Component Testing: Testing of Code generally written by a single programmer
4 Major Testing Phases
2. Integration Testing: Testing of Several Units, Combined To Form A Unit or SubSystem By Testing Team
4 Major Testing Phases
4. Final/Beta Testing: Testing of The Finished Program, By End Users or Their Representatives
4 Major Testing Phases
1. Unit/Component Testing: Testing of Code generally written by a single programmer
2. Integration Testing: Testing of Several Units, Combined To Form A Unit or SubSystem By Testing Team
3. System/Alpha Testing: Testing of The Finished Program, By Program Testers
4. Final/Beta Testing: Testing of The Finished Program, By End Users or Their Representatives
4 Major Testing Phases (Summary)
Testing Strategy
We begin by ‘testing-in-the-small’ and move toward ‘testing-in-the-large’
For conventional software
1. The module (component) is our initial focus
2. Integration of modules follows
For OO software
The testing focus is an OO class that encompasses attributes and operations and implies communication and collaboration
Successful Software Testers Do 7 Things: 1- 4
1. State testing objectives explicitly.
2. Understand the users of the software and develop a profile for each user category.
3. Develop a testing plan that emphasizes “rapid cycle testing.”
4. Build “robust” software that is designed to test itself
Successful Software Testers Do 7 Things: 5 - 7
5. Use effective formal technical reviews as a filter
prior to testing
6. Conduct formal technical reviews to assess the
test strategy and test cases themselves.
7. Develop a continuous improvement approach for
the testing process.
Unit Testing
modulemoduletoto bebe
testedtested
test casestest cases
resultsresults
softwaresoftwareengineerengineer
Unit/ComponentTesting
IntegrationTesting
Final/Beta/ValidationTesting
System/AlphaTesting
About Unit Testing’s Test Cases
interface interface locallocal datadata structuresstructuresboundaryboundary conditionsconditionsindependentindependent pathspathserrorerror handlinghandling pathspaths
modulemoduleto beto be
testedtested
test casestest casesUnit/Component
TestingIntegration
Testing
Final/Beta/ValidationTesting
System/AlphaTesting
Unit Test Environment
ModuleModule
stubstub stubstub
Driver/HarnessDriver/Harness
RESULTSRESULTS
interfaceinterfacelocallocal datadata structuresstructuresboundaryboundary conditionsconditionsindependentindependent pathspaths
errorerror handlinghandling pathspaths
testtest casescases
Test Harness, also called a Test Driver,
is the supporting code and data used to
provide an environment for testing part
of a system in isolation.
Test Harness
A Stub is a dummy procedure, module,
or unit, that stands in for an unfinished
portion of the program.
Stub
Unit/ComponentTesting
IntegrationTesting
Final/Beta/ValidationTesting
System/AlphaTesting
The Component Testing ProcessAlso Called Unit Testing
Component Testing
Component/Unit Testing is the testing of individual
program components
Component Testing is usually the responsibility of
the component developer (except sometimes for
critical systems)
Component Tests are derived from the developer’s
experience
The Integration Testing Process
Integration Testing
Integration Testing is the testing of groups of components integrated to create a system or sub-system
Integration Testing is the responsibility of an independent testing team
Integration Testing is based on the system specifications Unit/Component
TestingIntegration
Testing
Final/Beta/ValidationTesting
System/AlphaTesting
Integration Testing Strategies
Options:Options:•• the “big bang” approachthe “big bang” approach•• an incremental construction strategyan incremental construction strategy
Unit/ComponentTesting
IntegrationTesting
Final/Beta/ValidationTesting
System/AlphaTesting
Top-Down Design Testing &
Implementation
Top-Down Integration & Testing
Create stubs for eachCreate stubs for eachAnd every module it callsAnd every module it calls
AA
Stub BStub B Stub CStub C Stub DStub D
Designing & Testing Stubs
Create the Stub – Pass Something Known To It!
int Foo (int A, int & B){ }
void Goo (void){int Temp = 20 X = 5, Y = 6;. . . Temp = Foo (X,Y);. . .}
Create the Stub – Verify All In-Coming Communication
int Foo (int A, int & B){ printf (“Stub - Foo A = %ld B = %ld\n”, A, B);
}
void Goo (void){int Temp = 20 X = 5, Y = 6;. . . Temp = Foo (X,Y);. . .}
Create the Stub – Alter All Reference Variables
int Foo (int A, int & B){ printf (“Stub -> Foo A = %ld B = %ld\n”, A, B);
A++; B++;
printf (“Stub <- Foo A = %ld B = %ld\n”, A, B);}
void Goo (void){int Temp = 20 X = 5, Y = 6;. . . Temp = Foo (X,Y);. . .}
Create the Stub – Assign A Known Return
int Foo (int A, int & B){ printf (“Stub -> Foo A = %ld B = %ld\n”, A, B);
A++; B++; printf (“Stub <- Foo A = %ld B = %ld\n”, A, B); return (1);}
void Goo (void){int Temp = 20 X = 5, Y = 6;. . . Temp = Foo (X,Y);. . .}
Stubs are all about testing communication.
We do not yet know that this works effectively
Create the Stub – Verify Return Info
int Foo (int A, int & B){ printf (“Stub -> Foo A = %ld B = %ld\n”, A, B);
A++; B++; printf (“Stub <- Foo A = %ld B = %ld\n”, A, B); return (1);}
void Goo (void){int Temp = 20 X = 5, Y = 6; printf (“X =% ld Y = %ld Temp = %ld\n”, X, Y, Temp); Temp = Foo (X,Y); printf (“X = %ld Y = %ld Tem p= %ld\n”, X, Y, Temp);}
It is much better yet to use one of your test cases for the stub.
Create the Stub – Verify Return Info
int Add (int Addend1, Addend2){ printf (“Stub -> Foo Addend1 = %ld ”, Addend1 printf (“Addend2 = %ld\n”, Addend2);
Addend1 = 1;Addend1 = 2;
// Let us assume that the module is complex – will fill in pieces later - communication printf (“Stub <- Foo A = %ld B = %ld\n”, A, B); return (11);}
void Goo (void){int Sum = 20 X = 5, Y = 6; printf (“X =% ld Y = %ld Sum = %ld\n”, X, Y, Sum ); Sum = Add (X,Y); printf (“X = %ld Y = %ld Sum %ld\n”, X, Y, Sum );}
Remember, with the exception of arrays, modules should be able to change only
that data the module is designed to change. As Needed Basis!
Top-Down Integration & Testing
Create stubs for eachCreate stubs for eachAnd every module it callsAnd every module it calls
AA
Stub BStub B Stub CStub C Stub DStub D
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
Stub EStub E Stub FStub F Stub GStub G
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE Stub FStub F Stub GStub G
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE Stub FStub F Stub GStub G
Stub HStub H
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE Stub FStub F Stub GStub G
HH
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE Stub FStub F Stub GStub G
HH
Stub IStub I Stub JStub J
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE Stub FStub F Stub GStub G
HH
II Stub JStub J
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE Stub FStub F Stub GStub G
HH
II JJ
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF Stub GStub G
HH
II JJ
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF GG
HH
II JJ
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF GG
HH
II JJ
Stub KStub K Stub LStub L Stub MStub M
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF GG
HH
II JJ
KK Stub LStub L Stub MStub M
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF GG
HH
II JJ
KK LL Stub MStub M
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF GG
HH
II JJ
KK LL Stub MStub M
Stub NStub N
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF GG
HH
II JJ
KK LL Stub MStub M
NN
Top-Down Integration & Testing
AA
BB Stub CStub C Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF GG
HH
II JJ
KK LL MM
NN
Top-Down Integration & Testing
AA
BB CC Stub DStub D
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF GG
HH
II JJ
KK LL MM
NN
Top-Down Integration & Testing
AA
BB CC DD
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF GG
HH
II JJ
KK LL MM
NN
Top-Down Integration & Testing
AA
BB CC DD
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF GG
HH
II JJ
KK LL MM
NN
Stub NStub N Stub OStub O
Top-Down Integration & Testing
AA
BB CC DD
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF GG
HH
II JJ
KK LL MM
NN
NN Stub OStub O
Top-Down Integration & Testing
AA
BB CC DD
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF GG
HH
II JJ
KK LL MM
NN
NN OO
Top-Down Integration & Testing
AA
BB CC DD
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF GG
HH
II JJ
KK LL MM
NN
NN OO
Stub PStub P
Top-Down Integration & Testing
AA
BB CC DD
stubs are replaced one at a stubs are replaced one at a time, "depth first" time, "depth first"
EE FF GG
HH
II JJ
KK LL MM
NN
NN OO
PP
Top-Down can be done within units and
sub-systems
Bottom-Up Integration & Testing
Lowest level – bottom modules Lowest level – bottom modules tested first – Maybe start withtested first – Maybe start withModule AModule A
AA
Bottom-Up Integration & Testing
Lowest level – bottom modules Lowest level – bottom modules tested firsttested firstAA
Harness - BHarness - B
Create just enough of a Create just enough of a harness to test Bharness to test B
Bottom-Up Integration & Testing
AA
Harness - BHarness - B
CCLowest level – bottom modules Lowest level – bottom modules tested first – Maybe continue tested first – Maybe continue with Module Cwith Module C
Bottom-Up Integration & Testing
AA
Harness - BHarness - B
Modify the harness to test EModify the harness to test E
CCLowest level – bottom modules Lowest level – bottom modules tested first – Maybe continue tested first – Maybe continue with Module Ewith Module E
Bottom-Up Integration & Testing
AA
Harness - BHarness - B
CC DD
Harness - EHarness - E
FF GG HH
Harness - IHarness - I
Bottom-Up Integration & Testing
AA
Harness - BHarness - B
CC DD
Harness - EHarness - E
FF GG HH
Harness - IHarness - I
JJ
Bottom-Up Integration & Testing
AA
BB
CC DD
Harness - EHarness - E
FF GG HH
Harness - IHarness - I
JJ
Bottom-Up Integration & Testing
AA
BB
CC DD
Harness - EHarness - E
FF GG HH
Harness - IHarness - I
JJ
Harness - KHarness - K
Bottom-Up Integration & Testing
AA
BB
CC DD
EE
FF GG HH
Harness - IHarness - I
JJ
Harness - KHarness - K
Bottom-Up Integration & Testing
AA
BB
CC DD
EE
FF GG HH
Harness - IHarness - I
JJ
Harness - KHarness - K
Bottom-Up Integration & Testing
AA
BB
CC DD
EE
FF GG HH
II
JJ
Harness - KHarness - K Harness - LHarness - L
Bottom-Up Integration & Testing
DD
CC
EE FF
EE
HH II JJ
KK
LL
MM Harness - NHarness - N
Harness - OHarness - O
Bottom-Up Integration & Testing
DD
CC
EE FF
EE
HH II JJ
KK
LL
MM NN
OO
Bottom-Up can be done within units and
sub-systems
Sandwich Integration & Testing
DD
CC
EE FF
EE
HH II JJ
KK
LL
MM NN
OO
Might Work On Lower Portions Some
More
Sandwich Integration & Testing
DD
CC
EE FF
EE
HH II JJ
KK
LL
MM NN
OO
Gradually Fit It All Together
No Specific Order!
Object-Oriented TestingOO Testing begins by evaluating the correctness
and consistency of the OO models
Testing Strategy Changes
In OO, the concept of the ‘unit’ testing broadens due to encapsulation
In OO, integration testing focuses on classes and their execution across a ‘thread’ or in the context of a usage scenario
In OO, validation uses conventional black box methods
Test case design draws on conventional methods, but also encompasses special features
3 Steps Of Smoke Testing
Smoke Testing is a common approach for creating “daily
builds” for product software
Smoke Testing Steps:
1. Software components, that have been translated into
code, are integrated into a “build.”
A build includes all data files, libraries, reusable
modules, and engineered components that are
required to implement one or more product
functions.
3 Steps Of Smoke Testing Smoke Testing Steps (cont):
2. A series of tests is designed to expose errors that will
keep the build from properly performing its function.
The intent should be to uncover “show stopper”
errors that have the highest likelihood of throwing
the software project behind schedule.
3. The build is integrated with other builds and the entire
product (in its current form) is smoke tested daily.
The integration approach may be top down or
bottom up.
“show stopper”Smoke TestingFind The Fires!
Unit/ComponentTesting
IntegrationTesting
Final/Beta/ValidationTesting
System/AlphaTesting
7 Testing Objectives
1. Validation Testing
Focus is on software requirements
2. System Testing
Focus is on system integration
3. Alpha/Beta Testing
Focus is on customer usage
4. Recovery Testing
forces the software to fail in a variety of ways and verifies that recovery is properly performed
7 Testing Objectives
5. Security Testing
verifies that protection mechanisms built into a system will, in fact, protect it from improper penetration
6. Stress Testing
executes a system in a manner that demands resources in abnormal quantity, frequency, or volume
7. Performance Testing
test the run-time performance of software within the context of an integrated system
Unit/ComponentTesting
IntegrationTesting
Final/Beta/ValidationTesting
System/AlphaTesting
The Debugging Process
test casestest cases
resultsresults
DebuggingDebugging
suspectedsuspectedcausescauses
identifiedidentifiedcausescauses
correctionscorrections
regressionregressionteststests
new testnew testcasescases
Debugging Effort
time requiredtime requiredto diagnose theto diagnose thesymptom andsymptom anddetermine thedetermine thecausecause
time requiredtime requiredto correct the errorto correct the errorand conductand conductregression testsregression tests
Symptoms & Causes Of Program Bugs
symptomsymptomcausecause
symptom and cause may be symptom and cause may be geographically separatedgeographically separated
symptomsymptom maymay disappeardisappear whenwhen anotheranother problemproblem isis fixedfixed
causecause maymay bebe duedue toto a a combinationcombination ofof non-errorsnon-errors
causecause maymay bebe duedue toto a systema system oror compilercompiler errorerror
causecause maymay bebe duedue toto assumptionsassumptions thatthat everyoneeveryone believesbelieves
symptomsymptom maymay bebe intermittentintermittent
Consequences of Bugs
damagedamage
mildmild annoyingannoying
disturbingdisturbingseriousserious
extremeextremecatastrophiccatastrophic
infectiousinfectious
BugBug TypeType
Bug Categories: function-related bugs, system-related bugs, data bugs, coding bugs, design bugs, documentation bugs, standards violations, etc.
3 Major Debugging Strategies
1.1. Brute Force – Brute Force – Most commonly used and least effective – use memory dumps, run traces, and/or many print statements.
2.2. Backtracking Backtracking – can be used In small programs – begin where error is found and manually backtrack examining logic and output statements until problem found.
3.3. Cause Elimination – Cause Elimination – use induction and deduction to produce lists of potential causes is identified. Test cases are constructed to systematically prove or disprove each potential hypothesis.
Debugging: Final Thoughts
1.1. Don't run off half-cocked, Don't run off half-cocked, think think about the about the symptom you're seeing.symptom you're seeing.
2.2. Use tools (e.g., dynamic debugger) to gain Use tools (e.g., dynamic debugger) to gain
more insight.more insight.
3.3. If at an impasse, get help from someone else.If at an impasse, get help from someone else.
4.4. Be absolutely sure to conduct regression Be absolutely sure to conduct regression
tests when you do “fix” the bug.tests when you do “fix” the bug.
Exhaustive Testing is executing a program with all possible combinations of inputs or values for program variables.
Only Exhaustive Testing can show a program is free from defects. However, exhaustive testing is impossible
Testing Priorities
Exhaustive Testing
loop < 20 Xloop < 20 X
1414There are 10 possible paths! If we execute one test There are 10 possible paths! If we execute one test per millisecond, it would take 3,170 years to test this per millisecond, it would take 3,170 years to test this
program!!program!!
Selective Testing
loop < 20 Xloop < 20 X
Selected pathSelected path
There are 10 possible paths!There are 10 possible paths!
Defect Testing
The Goal of Defect Testing is to discover defects in programs
A Successful Defect Test is a test which causes a program to behave in an anomalous way
Tests Show the Presence not the Absence of Defects
Only Exhaustive Testing can show a program is free from defects. However, exhaustive testing is impossible
Tests Should Exercise a system's capabilities rather than its components
Testing Old Capabilities is more important than testing new capabilities
Testing Typical Situations is more important than boundary value cases
Testing Priorities
Test Data are Inputs which have been devised to test the system
Test Cases are Inputs to test the system and the predicted outputs from these inputs if the system operates according to its specification
Test Cases Have 3 Parts Goal
Input & System State
Expected behavior
Test Data & Test Cases
Black-Box Testing Black-Box Testing is an approach to testing where the
program is considered as a ‘black-box’; it is also known as function testing.
In Black-Box Testing the program test cases are based on the system specification
In Black-Box Testing is a software testing technique whereby the internal workings of the item being tested are not known by the tester
In Black-Box Testing on a software design, the tester only knows the inputs and what the expected outcomes should be and not how the program arrives at those outputs.
In Black-Box Testing the tester does not ever examine the programming code and does not need any further knowledge of the program other than its specifications.
Black-Box Testing – Concerns?
How is functional validity tested?
How is system behavior and performance tested?
What classes of input will make good test cases?
Is the system particularly sensitive to certain input values?
How are the boundaries of a data class isolated?
What data rates and data volume can the system tolerate?
What effect will specific combinations of data have on system operation?
Pros & Cons Of Black-Box Testing
The advantages of Black-Box testing include: 1. The test is unbiased because the designer and the tester are
independent of each other.
2. The tester does not need knowledge of any specific programming languages.
3. The test is done from the point of view of the user, not the designer.
4. Test cases can be designed as soon as the specifications are complete.
The disadvantages of Black-Box testing include: 1. The test can be redundant if the software designer has already
run a test case.
2. The test cases are difficult to design.
3. Testing every possible input stream is unrealistic because it would take a inordinate amount of time; therefore, many program paths will go untested.
Black-Box Testing
Ie
Input test data
OeOutput test results
System
Inputs causinganomalousbehaviour
Outputs which revealthe presence ofdefects
Equivalence Partitioning
Input Data & Output Results often fall into different classes where all members of a class are related
Each of these Classes is an Equivalence Partition where the program behaves in an equivalent way for each class member
Test Cases should be chosen from each partition
Equivalence Partitioning
useruserqueriesqueries mousemouse
pickspicks
outputoutputformatsformats
promptsprompts
FKFKinputinput
datadata
Sample Equivalence Classes
user supplied commandsuser supplied commands
responses to system promptsresponses to system prompts
file namesfile namescomputational datacomputational data physical parameters physical parameters bounding valuesbounding values initiation valuesinitiation valuesoutput data formattingoutput data formattingresponses to error messagesresponses to error messagesgraphical data (e.g., mouse picks)graphical data (e.g., mouse picks)
data outside bounds of the program data outside bounds of the program physically impossible dataphysically impossible dataproper value supplied in wrong placeproper value supplied in wrong place
Valid dataValid data
Invalid dataInvalid data
Boundary Value Analysis
useruserqueriesqueries
mousemousepickspicks
outputoutputformatsformats
promptsprompts
FKFKinputinput
datadata
outputoutputdomaindomaininput domaininput domain
Partition System Inputs and Outputs into ‘Equivalence Sets’
Suppose the input is a 5-digit integer between 10,000 and 99,999
Equivalence Partitions are
> 10,000 [Too Big]
10,000 - 99,999 [Expected]
< 10, 000 [Too Small]
Equivalence Partitioning
Write Good Test Data For The Equivalence Partitions
Most Authorities Agree That This Equivalence Partitioning Should Have at least 7
Equivalence Partitions 5-digit integer between 10,000 and 99,999
> 10,000 [Too Big]
10,000 - 99,999 [Expected]
< 10, 000 [Too Small] Choose One Expected General Case
50,000 Choose One Clearly Too Big
150,000
Choose One Clearly Too Small
5,000
Test The Boundary Conditions
9,999 & 10,000 & 99,999 & 100,000
Testing Equivalence Partitioning
There are a number of ways of writing a specification. Sometimes, the easiest way to communicate with
developers is with conditional logic. This would not the best way to communicate with stakeholders.
procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ;
Pre-condition-- the array has at least one elementT’FIRST <= T’LAST
Post-condition-- the element is found and is referenced by L( Found and T (L) = Key)
or -- the element is not in the array( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
Binary Search Routine Specification
procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ;
Pre-condition-- the array has at least one elementT’FIRST <= T’LAST
Post-condition-- the element is found and is referenced by L( Found and T (L) = Key)
or -- the element is not in the array( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
What Are The Search Routine Equivalence Partitions?
How Many Search Routine Equivalence Partitions?
1. Inputs which Conform to the Pre-Conditions
2. Inputs where a Pre-Condition Does Not Hold
3. Inputs where the Key Element is a Member of the Array
4. Inputs where the Key Element is Not a Member of the Array
procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ;
Pre-condition-- the array has at least one elementT’FIRST <= T’LAST
Post-condition-- the element is found and is referenced by L( Found and T (L) = Key)
or -- the element is not in the array( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
4 Search Routine - Input Partitions
Testing Binary Search Routine SpecificationNot My Choice Of Data Representation –
Designer’s Choice!
1. Inputs where a Pre-Condition Does Hold
This Is Often The Case For The Software Testing Team
Max
Last
First0
1
2
3
4
5
6
7
6
2
12
13
34
45
66
7
T
Testing Binary Search Routine Specification
2. Inputs where a Pre-Condition Does Not Hold
Any Other Condition 2Scenario To Test?
Say Yes!
Max
Last
First0
1
2
3
4
5
6
7
0
4
8
9
4
5
6
7
T
Testing Binary Search Routine Specification
Empty!Empty!
2.2. Inputs where a Pre-Condition Does Not HoldInputs where a Pre-Condition Does Not Hold
Max
Last
First0
1
2
3
4
5
6
7
-1
-1
7
T
Testing Binary Search Routine Specification
3.3. Inputs where the Key Element is a Member of the ArrayInputs where the Key Element is a Member of the Array
Choose Key In Middle 34
Choose Key At Boundary Points 66 & 11
Check Return Found = True Check T[L] = Key
Max
Last
First0
1
2
3
4
5
6
7
6
2
11
13
34
45
66
7
Key
34
Found
T
L
4
T
Testing Binary Search Routine Specification4. Inputs where the Key Element is Not a Member of the Array
Choose Non-Existent Key In Middle 15
Choose Non-Existent Key Inside Boundary Points 12 & 65
Check Return Found = False
Key
15
Found
F
L
--
Max
Last
First0
1
2
3
4
5
6
7
6
2
11
13
34
45
66
7
T
procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ;
Pre-condition-- the array has at least one elementT’FIRST <= T’LAST
Post-condition-- the element is found and is referenced by L( Found and T (L) = Key)
or -- the element is not in the array( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
A Good Software Testing Team Often Identifies & Corrects Weak Specifications
This Specification Has A Real Obvious Omission. Create A Test Case To This Specification Has A Real Obvious Omission. Create A Test Case To Illustrate The Weakness Of This Specification.Illustrate The Weakness Of This Specification.
Our Existing Examples Also Need Further Refinement!Our Existing Examples Also Need Further Refinement!
Testing Binary Search Routine Specification
4. Inputs where the Key Element is Not a Member of the Array
Choose Non-Existent Key In MiddleChoose Non-Existent Key In Middle 1515
Choose Non-Existent Key Inside Choose Non-Existent Key Inside Boundary PointsBoundary Points 12 & 65 12 & 65
Choose Non-Existent Key Outside Choose Non-Existent Key Outside Boundary PointsBoundary Points67 & 1067 & 10
Check Return Found = FalseCheck Return Found = False
Key
67
Found
F
L
--
Max
Last
First0
1
2
3
4
5
6
7
6
2
11
13
34
45
66
7
T
Testing Binary Search Routine SpecificationParting Thoughts About Arrays!
Successful Test Case of 1 Element!
Unsuccessful Test Case of 1 Element!
Test With A Full Array To Verify That There Are No Subscript Errors In Which T[-1] or T[8] are erroneously examined!
Nothing in the specification addressed duplicates; does the algorithm break with a duplicate?
Nothing in the specification addressed duplicates; which duplicate should be returned?
Experience Helps!
Key
12
Found
T
L
?
Max
Last
First0
1
2
3
4
5
6
7
6
2
12
13
34
45
66
7
T
71
12
12
White-Box Testing
... our goal is to ensure that all statements and ... our goal is to ensure that all statements and conditions have been executed at least once ... conditions have been executed at least once ...
White-box Testing, sometimes called “structural testing”
or “clear box testing” or “open box testing”, is a software testing technique whereby explicit knowledge of the internal workings of the item being tested are used to select the test data.
The Objective of Structural/White-Box Testing is to exercise all program statements (not all path combinations)
White-Box Testing derives test cases according to program structure. Knowledge of the program is used to identify these additional structural test cases
Structural Testing – White-Box Testing
White-Box testing uses specific knowledge of programming code to examine outputs. The test is accurate only if the tester knows what the program is supposed to do.
The White-Box tester can then see if the program diverges from its intended goal.
White box testing does not account for errors caused by omission, and all visible code must also be readable.
Structural Testing – White-Box Testing
Why Do White-Box Testing
LogicLogic errorserrors andand incorrectincorrect assumptions assumptions are inversely proportional to a path'sare inversely proportional to a path's execution probabilityexecution probability
TypographicalTypographical errorserrors areare random; it's random; it's likely that untested paths will contain likely that untested paths will contain some some
class BinSearch {
// This is an encapsulation of a binary search function that takes an array of// ordered objects and a key and returns an object with 2 attributes namely// index - the value of the array index// found - a boolean indicating whether or not the key is in the array// An object is returned because it is not possible in Java to pass basic types by// reference to a function and so return two values// the key is -1 if the element is not found
public static void search ( int key, int [] elemArray, Result r ){
int bottom = 0 ;int top = elemArray.length - 1 ;int mid ;r.found = false ; r.index = -1 ;while ( bottom <= top ){
mid = (top + bottom) / 2 ;if (elemArray [mid] == key){
r.index = mid ;r.found = true ;return ;
} // if partelse{
if (elemArray [mid] < key)bottom = mid + 1 ;
elsetop = mid - 1 ;
}} //while loop
} // search} //BinSearch
Binary search (Java)Consider White-Box
Paths
Need To Exercise Each
Line At Least Once!
Pre-conditions satisfied, key element in arrayPre-conditions satisfied, key element in array
Pre-conditions satisfied, key element not in Pre-conditions satisfied, key element not in arrayarray
Pre-conditions unsatisfied, key element in Pre-conditions unsatisfied, key element in arrayarray
Pre-conditions unsatisfied, key element not in Pre-conditions unsatisfied, key element not in arrayarray
Input array has a single valueInput array has a single value
Input array has an even number of valuesInput array has an even number of values
Input array has an odd number of valuesInput array has an odd number of values
Binary Search - Equivalent PartitionsBased On White-Box Examination Of The Code
Binary Search Equivalent Partitions
Mid-point
Elements < Mid Elements > Mid
Equivalence class boundaries
procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ;
Pre-condition-- the array has at least one elementT’FIRST <= T’LAST
Post-condition-- the element is found and is referenced by L( Found and T (L) = Key)
or -- the element is not in the array( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
Using the specification above as a starting point, create a good specification for
inserting a new item into the sorted array
Let us assume that our lowest element will always be in T[0] – No Duplicates Allowed
bool UniqueArrayInplace (in out long T[ ], long NewInfo, in out long Max, in out long
ActNo)
Write The Specification
Successful
T
NewInfo
50
Max
ActNo
Sorted0
1
2
3
4
5
6
7
7
T
12
13
34
45
66
7
T
2
1
UniqueT
Bool function ArrayInplace (in out T: ELEM_ARRAY, NewInfo ELEM, long Max, in
out long ActNo)
Pre-Condition-- Not Full-Room For 1 More ActNo <= Max
Or
-- if Unique & Sorted – if Binary Search Successfulor -- if not sorted & Not Sorted – if Sequential Search Successful
Post Condition
-- if sorted - Add NewInfo To Correctly Ordered Position
Or
-- if not sorted - Append NewInfo To Next Available Position
Successful
T
NewInfo
50
Max
ActNo
Sorted0
1
2
3
4
5
6
7
7
T
12
13
34
45
66
7
T
2
1
UniqueT
Suppose This Is The Specification
Partition System Inputs and Outputs into Black-Box Inspection ‘Equivalence Sets’!
Write Them!Bool function ArrayInplace (in out T: ELEM_ARRAY, NewInfo ELEM, long Max, Bool function ArrayInplace (in out T: ELEM_ARRAY, NewInfo ELEM, long Max,
in out long ActNo)in out long ActNo)
Pre-ConditionPre-Condition-- Not Full-Room For 1 More ActNo <= Max-- Not Full-Room For 1 More ActNo <= Max
OrOr
-- if Unique & Sorted – if Binary Search Successful-- if Unique & Sorted – if Binary Search Successfuloror -- if not sorted & Not Sorted – if Sequential Search Successful -- if not sorted & Not Sorted – if Sequential Search Successful
Post ConditionPost Condition
-- if sorted - Add NewInfo To Correctly Ordered Position-- if sorted - Add NewInfo To Correctly Ordered Position
OrOr
-- if not sorted - Append NewInfo To Next Available Position-- if not sorted - Append NewInfo To Next Available Position
Suppose This Is The Specification
We could have you write the ArrayInplace function in C Bool function BinSearch (T: ELEM_ARRAY, SoughtInfo ELEM, long ActNo, long Position
Bool function SequentialSearch (T: ELEM_ARRAY, SoughtInfo ELEM, long ActNo, long Position)
Bool function ArrayInplace (in out T: ELEM_ARRAY, NewInfo ELEM, Bool function ArrayInplace (in out T: ELEM_ARRAY, NewInfo ELEM, long Max, in out long ActNo)long Max, in out long ActNo)
Pre-ConditionPre-Condition-- Not Full-Room For 1 More ActNo <= Max-- Not Full-Room For 1 More ActNo <= Max
OrOr
-- if Unique & Sorted – if Binary Search Successful-- if Unique & Sorted – if Binary Search Successfuloror -- if not sorted & Not Sorted – if Sequential Search Successful -- if not sorted & Not Sorted – if Sequential Search Successful
Post ConditionPost Condition
-- if sorted - Add NewInfo To Correctly Ordered Position-- if sorted - Add NewInfo To Correctly Ordered Position
OrOr
-- if not sorted - Append NewInfo To Next Available Position-- if not sorted - Append NewInfo To Next Available Position
Examine your C solution!
Write the White-Box Inspection ‘Equivalence Sets’!
Bool function ArrayInplace (in out T: ELEM_ARRAY, NewInfo ELEM, long Max, in out long ActNo)
Pre-Condition-- Not Full-Room For 1 More ActNo <= Max
Or
-- if Unique & Sorted – if Binary Search Successfulor -- if not sorted & Not Sorted – if Sequential Search Successful
Post Condition
-- if sorted - Add NewInfo To Correctly Ordered Position
Or
-- if not sorted - Append NewInfo To Next Available Position
Bool function ArrayInplace (in out T: ELEM_ARRAY, NewInfo ELEM, long Max, in
out long ActNo)
Pre-Condition-- Not Full-Room For 1 More ActNo <= Max
Or
-- if Unique & Sorted – if Binary Search Successfulor -- if not sorted & Not Sorted – if Sequential Search Successful
Post Condition
-- if sorted - Add NewInfo To Correctly Ordered Position
Or
-- if not sorted - Append NewInfo To Next Available Position
Successful
T
NewInfo
50
Max
ActNo
Sorted0
1
2
3
4
5
6
7
7
T
12
13
34
45
66
7
T
2
1
UniqueT
Path TestingBinary-Search Example
The Objective of Path Testing is to ensure that the set of test cases is such that each path/branch through the program is executed at least once
1
2
3
4
65
7
while bottom <= top
if (elemArray [mid] == key
(if (elemArray [mid]< key8
9
bottom > top
The Starting Point for Path Testing is a Program Flow Graph that shows Nodes representing Program Decisions and Arcs Representing the Flow of Control
Statements With Conditions are therefore Nodes in the flow graph
Binary Search Cyclomatic Complexity = _______________
Program Flow Graphs
Program Flow Graphs describes the program control flow. Each branch is shown as a separate path and loops are shown by arrows looping back to the loop condition node
1
2
3
4
65
7
while bottom <= top
if (elemArray [mid] == key
(if (elemArray [mid]< key8
9
bottom > top
The Program Flow Graph is used as a basis for computing
the Cyclomatic Complexity Cyclomatic Complexity = # Edges - # Nodes + 2
11 - 9 + 2 = 4
Sketch The Program Flow Graph
If (. . .) Then{ X = . . . . . .}Else{ Y + . . . . . .}
Start 1
End 2
3 – 3 + 2 = 2
Cyclomatic Complexity
# Edges - # Nodes + 2
If - 3
a
TF
bc
List The Paths For The Program Flow Graph
If (. . .) Then{ X = . . . . . .}Else{ Y + . . . . . .}
3 – 3 + 2 = 2
# Edges - # Nodes + 2
Testing Paths
Cyclomatic Complexity
ab ac
Start 1
End 2
If - 3
a
TF
bc
Sketch the Program Flow GraphsIf (. . .) Then{ Y = . . . Z = . . . While (. . .) { If (. . .) Then . . . Else . . . }}Else . . .
Start 1
End 2
While - 4
If - 5
# Edges - # Nodes + 2
Cyclomatic Complexity
If - 3
a
TF
c
b
T
F
d
eT F
f g
7 - 5 + 2 = 4
If (. . .) Then{ Y = . . . Z = . . . While (. . .) { If (. . .) Then . . . Else . . . }}Else . . .
Start 1
End 2
While - 4
If - 5
# Edges - # Nodes + 2
Cyclomatic Complexity
If - 3
a
TF
c
b
T
F
d
eT F
f g
List The Paths For The Program Flow Graph
7 - 5 + 2 = 4
Testing Paths
ac abe
abdfe abdge
Sketch The Program Flow Graph
Number the nodes 1,2,3,…Label the edges a,b,c,d,…
Compute Cyclomatic ComplexityList all of the paths
Sketch The Program Flow GraphIf (. . .) Then{ X = . . . If (. . .) Then { While (. . .) { If (. . .) Then . . . } } Else . . .}Else{ For (. . . ) { If (. . .) Then . . . Else . . . }
}
Number the nodes 1,2,3,…Label the edges a,b,c,d,…
Compute Cyclomatic ComplexityList all of the paths
Graph Matrices
A graph matrix is a square matrix whose size (i.e., number of rows and columns) is equal to the number of nodes on a flow graph
Each row and column corresponds to an identified node, and matrix entries correspond to connections (an edge) between nodes.
By adding a link weight to each matrix entry, the graph matrix can become a powerful tool for evaluating program control structure during testing
Cyclomatic Complexity - 1
Cyclomatic Complexity is the most widely used member of a class of static software metrics.
Cyclomatic Complexity may be considered a broad measure of soundness and confidence for a program.
Cyclomatic Complexity was introduced by Thomas McCabe in 1976, it measures the number of linearly-independent paths through a program module.
Cyclomatic Complexity provides a single ordinal number that can be compared to the complexity of other programs
http://www.sei.cmu.edu/str/descriptions/cyclomatic_body.html
Cyclomatic Complexity – 2“McCabe’s Complexity” - “Program Complexity”
Cyclomatic Complexity is often referred to simply as “Program Complexity”, or as “McCabe's Complexity”.
Cyclomatic Complexity is often used in concert with other software metrics. As one of the more widely-accepted software metrics, it is intended to be independent of language and language format
Cyclomatic Complexity has also been extended to encompass the design and structural complexity of a system
Cyclomatic Complexity – 3Assists In Risk Evaluation!
Cyclomatic Complexity Risk Evaluation
1-10 a simple program, without much risk
11-20 more complex, moderate risk
21-50 complex, high risk program
greater than 50 untestable program (very high risk)
$$
Someone Has To Estimate The Cost For Each & Every Project
http://www.mccabe.com/iq_research_metrics.htm
Other Complexity Instruments
Complexity Measurement Primary Measure of
Halstead Complexity Measures
Algorithmic complexity, measured by counting operators and operands
Henry and Kafura metrics
Coupling between modules (parameters, global variables, calls)
Bowles Metrics Module and system complexity; coupling via parameters and global variables
Troy and Zweben Metrics
Modularity or coupling; complexity of structure (maximum depth of structure chart); calls-to and called-by
Ligier Metrics Modularity of the structure chart
List All Of The Paths For The Binary Search!
Through 1-2-8-9
Through1-2-3-8-9
Through1-2-3-4-6-7
Through1-2-3-4-5-7
What Was TheCyclomatic ComplexityFor The Binary Search?
4
How Many PathsFor The Binary Search?
4?
List The Paths For The Program Flow Graph
Cyclomatic Complexity =
7 – 5 + 2 = 4
# Edges - # Nodes + 2
A D
A B C
A B F G
A B F E B C
The Number of Tests to test all control
statements equals the Cyclomatic Complexity
Does The Cyclomatic Complexity equals number of Conditions in a Program? – [You Do!]
Testing Each Path is certainly useful if used with care, but it does not imply adequacy of testing.
Although all paths are executed, all combinations of paths are not executed
Path TestingCyclomatic Complexity
In SummaryIndependent Paths
1, 2, 3, 8, 9
1, 2, 3, 4, 6, 7, 2
1, 2, 3, 4, 5, 7, 2
1, 2, 3, 4, 6, 7, 2, 8, 9
Test cases should be derived so that all of these paths are executed
A Dynamic Program Analyser may be used to check that paths have been executed
1
2
3
4
65
7
while bottom <= top
if (elemArray [mid] == key
(if (elemArray [mid]< key8
9
bottom > top
Available In Some Case Tool Sets!
Loop Testing
Nested Nested LoopsLoops
ConcatenatedConcatenated Loops Loops Unstructured Unstructured
LoopsLoops
Simple Simple looploop
Loop Testing: Simple Loops
MinimumMinimum conditions—Simpleconditions—Simple LoopsLoops
1.1. skipskip thethe looploop entirelyentirely2. only one pass through the loop2. only one pass through the loop3. two passes through the3. two passes through the looploop4. m passes through the loop m < n4. m passes through the loop m < n5. (n-1), n, and (n+1) passes through5. (n-1), n, and (n+1) passes through thethe looploop
wherewhere n is the maximum number n is the maximum number of allowableof allowable passespasses
Loop Testing: Nested Loops
Start at the innermost loop. Set all outer loops to their minimum iteration Start at the innermost loop. Set all outer loops to their minimum iteration parameter values. Test the min+1, typical, max-1 and max for the parameter values. Test the min+1, typical, max-1 and max for the innermost loop, while holding the outer loops at their minimum values. innermost loop, while holding the outer loops at their minimum values. Move out one loop and set it up as in step 2, holding all other loops at Move out one loop and set it up as in step 2, holding all other loops at typical values. Continue this step until the outermost loop has been typical values. Continue this step until the outermost loop has been tested.tested.
If the loops are independent of one anotherIf the loops are independent of one another thenthen treattreat eacheach asas a simple loopa simple loop else* treat as nested loopselse* treat as nested loops
endifendif**
forfor example, the final loop counter value of loop 1 is example, the final loop counter value of loop 1 is
used to initialize loop 2.used to initialize loop 2.
Nested LoopsNested Loops
Concatenated LoopsConcatenated Loops
Software EngineeringCSCI 3342
Dr. Thomas E. HicksComputer Science Department
Trinity University
Textbook: Software EngineeringBy Roger Pressman
Textbook: Software EngineeringBy Ian Sommerville
Special Thanks To WCB/McGraw-Hill & Addison Wesley For Providing Graphics Of Some Of Text Book Figures For Use In This
Presentation.