0 All Slides
-
Upload
hoang-hieu -
Category
Documents
-
view
108 -
download
10
Transcript of 0 All Slides
-
22/08/2011
1
Software testingNguyen Thanh Binh
IT Faculty, Danang University of TechnologyEmail: [email protected]
Website: www.dut.edu.vn/itf/ntb
2
Prerequisites No testing experience needed Some familiarity with the development
phases of a project would be helpful. Some mathematics would help
3
References1. Paul Jorgensen, Software Testing-A Craftsman's Approach, CRC
Press, 1995.2. Spyos Xanthakis, Pascal Rgnier, Constantin Karapoulios, Le test des
logiciels, Hermes Science, 2000.3. Hung Q. Nguyen and al., Testing application on the Web, John Wiley &
Sons, 2004.4. Ilene Burnstein, Practical Software Testing, Springer, 2003.5. Glenford J. Myers, The art of software testing, Wiley, 2004.6. Cem Kaner, Jack Falk, Hung Q. Nguyen, Testing Computer Software,
2nd Edition, John Wiley & Sons, 1999.7. Boris Beizer, Software Testing Techniques, International Thomson
Computer Press, Second Edition, 1990.8. Neil Bitzenhofer, Software Testing and Verification, Course, MSSE, 2008.9. Paul Ammann and Jeff Offutt, Introduction to Software Testing,
Cambridge University Press, Cambridge, UK, ISBN 0-52188-038-1, 2008.
10. Mauro Pezz, Michal Young, Software Testing and Analysis: Process, Principles, and Techniques, John Wiley & Sons.
-
22/08/2011
2
4
Course description Cover both theoretical and practical aspects of
testing software The student will participate in the entire range of test
activities: Analyze a requirements document for test conditions Write a test plan Design, create and execute test cases using various
testing approaches Record defects Write a test report
5
Contents Session 1: Introductory lecture
Introductions and expectations Course overview Contents
6
Contents Session 2: Introduction to Software Testing
Definitions, Principles, Axioms Stages of testing Perspectives on Software Testing
-
22/08/2011
3
7
Contents Session 3: Requirements analysis
Software Development Life Cycle (SDLC) Software Development stage
Requirements Testing and requirements
Learn to think like a tester Some examples
8
Contents Session 4
Exercise 1: Examining requirements
9
Contents Session 5: Structural Testing
White box testing / Structural testing Graph Theory Control flow criteria Data flow criteria Graph Coverage for Source Code Testing State Behavior
-
22/08/2011
4
10
Contents Session 6: Static Testing
Reviews and the test process Types of review Static analysis
11
Contents Session 7: Functional Testing
Introduction to functional testing Functional testing techniques
Boundary Value testing Equivalence Class testing Special Value testing Decision Tables
12
Contents Session 8: Test Documentation
Test Plan The need for test plans The structure of test plans A Test Plan Template A Test Plan example Testing on a large project
Test Cases Test Case Design Test Case Examples
-
22/08/2011
5
13
Contents Session 9
Exercise 2: Writing a test plan and test cases
14
Contents Session 10: Levels of Testing
Levels of Testing Integration Testing System Testing
Additional System Test Categories
15
Contents Session 11: Defect Reports/Test Reports
Handling Defects Bug Tracking System Test Reports Examples
-
22/08/2011
6
16
Contents Session 12: Test Automation and Tools
Test Automation Test tools
17
Why test? List of 107 software failures that should have
been caught by testinghttp://www.cs.tau.ac.il/~nachumd/verify/horror.html
One vital consideration from Myers bookThe Art of Software Testing
Mars Climate Orbiter Mars Polar Lander
18
Software Errors1. The Mars Climate Orbiter crashed in September 1999 because of a
"silly mistake": wrong units in a program. 2. The 1988 shooting down of the Airbus 320 by the USS Vincennes
was attributed to the cryptic and misleading output displayed by the tracking software.
3. Death resulted from inadequate testing of the London Ambulance Service software.
4. Several 1985-7 deaths of cancer patients were due to overdoses of radiation resulting from a race condition between concurrent tasks in the Therac-25 software.
5. Errors in medical software have caused deaths. Details in B.W. Boehm, "Software and its Impact: A Quantitative Assessment," Datamation, 19(5), 48-59(1973).
6. An Airbus A320 crashes at an air show.7. A China Airlines Airbus Industrie A300 crashes on April 26, 1994
killing 264. Recommendations include software modifications.
-
22/08/2011
7
19
Software Errors The Explosion of the Ariane 5
On June 4, 1996 an unmanned Ariane 5 rocket launched by the European Space Agency exploded just forty seconds after its lift-off from Kourou, French Guiana. The rocket was on its first voyage, after a decade of development costing $7 billion. The destroyed rocket and its cargo were valued at $500 million. A board of inquiry investigated the causes of the explosion and in two weeks issued a report. It turned out that the cause of the failure was a software error in the inertial reference system. Specifically a 64 bit floating point number relating to the horizontal velocity of the rocket with respect to the platform was converted to a 16 bit signed integer. The number was larger than 32,767, the largest integer storeable in a 16 bit signed integer, and thus the conversion failed.
20
Software Errors The Patriot Missile Failure
On February 25, 1991, during the Gulf War, an American Patriot Missile battery in Dharan, Saudi Arabia, failed to track and intercept an incoming Iraqi Scud missile. The Scud struck an American Army barracks, killing 28 soldiers and injuring around 100 other people. A report of the General Accounting office, GAO/IMTEC-92-26, entitled Patriot Missile Defense: Software Problem Led to System Failure at Dhahran, Saudi Arabia reported on the cause of the failure. It turns out that the cause was an inaccurate calculation of the time since boot due to computer arithmetic errors. Specifically, the time in tenths of second as measured by the system's internal clock was multiplied by 1/10 to produce the time in seconds. This calculation was performed using a 24 bit fixed point register. In particular, the value 1/10, which has a non-terminating binary expansion, was chopped at 24 bits after the radix point. The small chopping error, when multiplied by the large number giving the time in tenths of a second, led to a significant error. Indeed, the Patriot battery had been up around 100 hours, and an easy calculation shows that the resulting time error due to the magnified chopping error was about 0.34 seconds. A Scud travels at about 1,676 meters per second, and so travels more than half a kilometer in this time. This was far enough that the incoming Scud was outside the "range gate" that the Patriot tracked. Ironically, the fact that the bad time calculation had been improved in some parts of the code, but not all, contributed to the problem, since it meant that the inaccuracies did not cancel.
21
Software Errors Y2K
Spent some billions dollars
-
22/08/2011
8
22
Chapter 1 of the textbook A Perspective on Testing Basic Definitions
Error a mistake in design, coding, requirements, even testing
Fault the representation of the error Failure what happens when the fault executes
23
A Perspective on Testing More Definitions
Testing the process of finding errors and of validating the program/system
Test Case a test case has Inputs Steps Outputs Expected results
Process Test plan, Write test cases Run the test cases Evaluate results
24
A Perspective on Testing Test Cases
Test Cases will be discussed in detail in Session 7 and throughout the course.
Testing entails establishing the environment, providing the inputs (running the test case), observing outputs, and comparing to expected outputs.
Test Cases are developed, reviewed, used, managed, and saved and hopefully reused!
-
22/08/2011
9
25
A Perspective on Testing Identifying Test Cases
Functional Testing The program is a function that maps input values to
output values The only information used is the software specification In our Venn diagram, the Functional Test Cases are a
subset of S Further elaborated on in Part II Math background: Chapter 3 We will discuss in Session 6
26
A Perspective on Testing Identifying Test Cases
Structural Testing Uses the information inside the black box the
actual implementation In our Venn diagram, the Structural Test Cases are a
subset of P. Further elaborated on in Part III of the text Math background: Chapter 4 We will discuss this in Session 4 Main method: Test coverage metrics
27
A Perspective on Testing Identifying Test Cases
Comparing the two (Functional vs Structural) We will discuss this in Sessions 4 and 6If all specified behaviors have not been implemented,
structural test cases will never be able to recognize this.
Conversely, if the program implements behaviors that have not been specified, this will never be revealed by functional test cases.
-
22/08/2011
10
28
A Perspective on Testing Levels of Testing
Again, this will be covered in detail in Session 2.
RequirementsSpecification
PreliminaryDesign
DetailedDesign
Coding
Unit Testing
IntegrationTesting
SystemTesting
29
Testing a ProgramA program that we want to test reads in 3 integer values these 3
values are interpreted as the lengths of the sides of a triangle.The program prints a message that states whether the triangle is
Equilateral (all 3 sides equal)Isosceles (exactly 2 of the 3 sides are equal)Scalene (all 3 sides are of a different length)
On a sheet of paper, write specific sets of test data that you feel would adequately test this program.
Introduction to software testing
-
22/08/2011
11
31
Contents Definitions, Principles, Axioms Stages of testing Perspectives on Software Testing
32
Definitions, Principles, Axioms
33
Definitions Testing Verification Validation Error/Defect Black box testing White box testing
-
22/08/2011
12
34
Definitions of Testing (1)
The process of executing a program (or part of a program) with the intention of finding errors (Myers, Humphrey)
The purpose of testing is to find errors. Testing is the process of trying to discover every conceivable fault or weakness in a work product (Myers, Kit)
The process of searching for errors (Kaner)
35
Definitions of Testing (2) Testing the process of finding errors and of
validating the program/system (Jorgensen) The purpose of software testing is to find
errors and to get them fixed (Bitzenhofer) The purpose of software testing is to reduce
risk (E&M)
36
What Testing is Not Testing is not just another phase of the
project Testing is not just finding defects Testing is not debugging Testing is not a final exam
-
22/08/2011
13
37
Verification and Validation: Myers Verification: An attempt to find errors by
executing a program in a test or simulated environment
Validation: An attempt to find errors by executing a program in a real environment
38
Verification and Validation: IEEE
Verification: The process of evaluating a system or component to determine whether the productssatisfy the conditions imposed
Validation: The process of evaluating a system or componentto determine whether it satisfies specified requirements.
39
Verification and Validation: Kaner Verification: Checking a program against the
most closely related design documents or specifications (Design Verification Testing)
Validation: Checking the program against the published user or system requirements (System Validation Testing)
-
22/08/2011
14
40
Software Fault Terminology Error a mistake in design, coding,
requirements, even testing
Fault the representation of the error
Failure what happens when the fault executes
41
Software Fault Terminology: Examples Error
A wrong loop index Programmer-to-device pointer error
Fault the representation of the error A loop executes one too many times A programmed parameter gets written to the wrong address
Failure what happens when the fault executes A table overflows; memory gets overwritten The physician only thinks he/she changed that heart parameter
42
Black and White Box Black box testing
Designed without knowledge of the programs internal structure and design
Based on functional requirements Also called Functional Testing
White box testing Examines the internal design of the program Requires detailed knowledge of its structure Also called Structural Testing
-
22/08/2011
15
43
Six Essentials of Software Testing (1)1.The quality of the test process determines the
success of the test effort.2.Prevent defect migration by using early life-
cycle testing techniques.3.The time for software testing tools is now.
Adapted from Software Testing in the Real World, Edward Kit; Addison-Wesley, 1995
44
Six Essentials of Software Testing (2)4.A real person must take responsibility for
improving the testing process.5.Testing is a professional discipline requiring
trained, skilled people.6.Cultivate a positive team attitude of creative
destruction.Adapted from Software Testing in the Real World, Edward Kit; Addison-
Wesley, 1995
45
Some Principles and Axioms of Testing (1) Program testing can be used to show the
presence of bugs, but never their absence. (Dijkstra, 1969)
One of the most difficult problems in testing is knowing when to stop.
It is impossible to test your own program.
-
22/08/2011
16
46
Some Principles and Axioms of Testing (2) As the number of detected defects in a piece
of software increases, the probability of the existence of more undetected defects also increases.
Testing is an extremely creative and intellectually challenging task (Myers again)
Exhaustive testing is impossible.
47
Stages of testing
48
The Test Process Requirements phase Planning phase Design and development phase Execution phase Reporting phase
-
22/08/2011
17
49
Software Test Execution Stages Unit Test Design Verification Test (DVT)
Interface, Integration, Exit/Formal System Validation Test (SVT)
Main, Regression, Acceptance Customer Acceptance Test
50
Unit Test Performed by the developers White-box testing Should be done with each release to testing Results should be communicated to the Test
Group
51
Typical Entry Criteria (Unit Test) Module compiles successfully (no errors,
approved warnings) Unit Test Plan complete Unit Test Cases complete Tools are available
-
22/08/2011
18
52
Typical Unit Test Activities Source level debugging if it wont compile Design and review the Unit Test cases Track test defects (maybe) Measure Unit Test coverage Make sure it builds
53
Typical Exit Criteria (Unit Test) All tests attempted 99% of the tests executed successfully 85% statement coverage Test cases have been stored under
Configuration Management
54
Software Test Stages Unit Test Design Verification Test
Interface, Integration, Exit/Formal Test Phase System Validation Test
Main, Regression, Acceptance Customer Acceptance Test
-
22/08/2011
19
55
Design Verification Test (DVT) Performed by the test organization Black box testing Verification of engineering requirements Verification of the software design
56
Three Components of DVT Interface Testing Integration Testing DVT Exit/Formal Test Phase
57
Typical Entry Criteria (DVT) Unit Test complete (for all code being tested) Engineering requirements and program
design are complete The code is under Configuration
Management A current build is available that contains all
necessary integrated code Interface and Integration test cases have
been written
-
22/08/2011
20
58
Typical DVT activities Design, write and review the Test Plan Write the DVT Test Cases Execute Interface/Integration test cases Track defects to the code and to the tests
59
Interface Testing How does a user interact with the program? What input variables are required? What outputs are produced? Extremely important for e-business
applications
60
Integration Testing Integration between software components Integration with the front-end GUIs Integration between a server and its clients Integration between software and hardware Integration between Web services
-
22/08/2011
21
61
DVT Exit or Formal Test Phase This is a requirement for System Test Probably involves running a suite of
Regression Tests Defect requirements of some sort must be
present The whole point: Are we ready for System
Test?
62
Typical Exit Criteria (DVT) All test cases executed 98% of the test cases executed successfully High severity defects have been fixed and
verified
63
Software Test Stages Unit Test Design Verification Test
Interface, Integration, Exit/Formal System Validation Test
Main, Regression, Acceptance Customer Acceptance Test
-
22/08/2011
22
64
System Validation Test (SVT) Performed by the test organization Validation of customer requirements Three components
Main Test Regression Test SVT Acceptance Test
65
Typical Entry Criteria (SVT) DVT Exit completes successfully Test Plan has been updated and frozen All test cases have been finalized No high severity problems are open Code is frozen The install works
66
SVT Main Test Verify the functional requirements of the
product from the Customers point of view Includes usability, reliability, performance,
and installation Also has its own Entry and Exit criteria Use Cases form a big part of this testing Marketing issues are addressed here too
-
22/08/2011
23
67
SVT Regression Test Verify that all defects identified during SVT
have been addressed; i.e. all tests run OK
68
System Acceptance Test Verify final manufacturing build Verify all installation scenarios Verify delivery of all media
69
Typical Exit Criteria (SVT) All test cases attempted All test cases executed successfully, OR
every defect that has been submitted has been addressed
No high severity defects are open Test Cases are under Configuration
Management Final Test Report has been written
-
22/08/2011
24
70
Software Test Stages Unit Test Design Verification Test
Interface, Integration, Exit System Validation Test
Main, Regression, Exit Customer Acceptance Test
71
Customer Acceptance Testing Usually performed at the customer site Must be well-written and simple, but
meaningful May be dictated by the customer
72
Perspectives on Software Testing
-
22/08/2011
25
73
The Objectives and Limits of Testing (1) Limits
You cannot test a program completely Even if you do find the last bug, youll never know
it It takes more time than you have to test less than
youd like You will run out of time before you run out of test
cases
74
The Objectives and Limits of Testing (2) Limits
You cannot test every path You cannot test every valid input You cannot test every invalid input You cannot prove a program correct (because it
isnt!)
75
The Objectives and Limits of Testing (3) Objectives
The purpose of testing is to find problems The purpose of finding problems is to get them
corrected The purpose of testing is to reduce identifiable
risks
-
22/08/2011
26
76
Tests and Test Cases (1) Test a collection of Test Cases Test Case a test case has inputs, steps,
and outputs, as well as expected results Process
write the Test Plan write test cases run the test cases evaluate results
77
Tests and Test Cases (2) We will discuss Test Cases in detail in Session 7
and throughout the course. Testing entails establishing the environment,
providing the inputs (running the test case), observing outputs, and comparing to expected outputs. (a paraphrase)
Test Cases are developed, reviewed, used, managed, and saved and hopefully reused!
78
What is Quality? Quality from the Producers viewpoint: Does the
product meet the requirements? Quality from the Customers viewpoint: Fitness for
use (reliability, usability, portability, etc.) Quality (one strongly held view): The measure of a
products quality is the satisfaction of the customers, not whether it meets a specification
-
22/08/2011
27
79
Quality Assurance vs Quality Control Quality Assurance is the activity that
establishes and evaluates the processes that produce products QA improves the product by improving the
process Quality Control is the activity that evaluates
the product produced QC improves the product by finding defects
80
A Generic Testing Approach (SPRAE) Specification
A written statement of expected software behavior Premeditation
Written test plans, test scripts, test data, test schedules Repeatability
Same results every time a result of a mature process Accountability
Test logs; analysis and interpretation of results Economy
The cost effectiveness of testing
81
The Testers Toolkit Static testing Structural testing methods Functional testing methods Non-functional techniques
-
22/08/2011
28
Requirements analysis
83
Contents Software Development Life Cycle (SDLC) Software Development stage
Requirements Testing and requirements
Learn to think like a tester Some examples Writing test requirements
84
Software Development Life Cycle (SDLC)
-
22/08/2011
29
85
Software Development Life Cycle (SDLC)
SDLC is a disciplined and systematic approach that divides the software development process into various phases, such as requirement, design, and coding
The phase-wise software development process helps you track schedule, cost, and quality of the software projects
86
Software Development Life Cycle (SDLC)
There are six phases of SDLC Feasibility analysis phase includes the analysis of project
requirement. Requirement analysis and specification phase includes gathering,
analyzing, validating, and specifying requirements. Design phase includes translation of requirements specified in the
SRS into a logical structure that can be implemented in a programming language.
Coding phase includes implementation of design specified in the design document into executable programming language code.
Testing phase includes detection of errors in the software. Maintenance phase includes implementation of changes and new
requirements in the software at the customer location.
87
SDLC models Are a tailored form of SDLC phases. Provide a basis for categorizing and
controlling the various activities required to develop and maintain the software system
Typical types of SDLC models are Linear model Iterative model
-
22/08/2011
30
88
Linear Models Linear models are suitable for projects where
all the requirements are identified and well understood before the design of the software begins.
The two types of linear models are: Waterfall model Prototyping model
89
Waterfall model Describes the software development process in a
linear sequential flow. Defines the software development process in seven
phases: Conception Initiation Analysis Design Construction Integration and testing Implementation and maintenance
90
Waterfall model
-
22/08/2011
31
91
Prototyping model Is a sample implementation of the system
that shows limited and main functional capabilities of the proposed system.
Helps the customer determine how the features will function in the final software.
92
Prototyping model
93
Iterative model Iterative model is used when the
requirements for the software are likely to evolve throughout the development process.
The various types of iterative models are Spiral model Component-based development model Unified process
-
22/08/2011
32
94
Spiral model Spiral model
Includes the iterative nature of the prototyping model and the linear nature of the waterfall model.
Is ideal for developing software that are released in various versions.
The six phases of spiral model are: Customer communication Planning Risk analysis Engineering Construction and release Customer evaluation
95
Spiral model
96
Component-Based Development Model In a component-based development model:
Components are reused and combined with other components in the same or other computers in a distributed network to form an application.
All related modules that form a component are tested to ensure that they work together
-
22/08/2011
33
97
Advantages: Linear Models Requirements are in the hands of the testers
early The project is better designed, with better
focus Late changes to requirements or design are
limited Test tasks of scheduling, planning, etc., are
made easier
98
Advantages: Iterative Models A shippable product is available early Progress (Where are we?) is much easier to
track Many opportunities to reappraise
requirements or rethink the design Easy to drop features
99
Disadvantages: Linear Models Big decisions must be made early It is unknown what will be difficult to
implement Changing a specification affects everyone Tremendous pressure on testers to prove
that a product is NOT ready for release
-
22/08/2011
34
100
Disadvantages: Iterative Models Product design may become a bottleneck Products often ship with half the features missing Big temptation for developers to keep writing new
code rather than fix whats broken Lack of early planning may cause problems when
later features are added How does Marketing market the product? Constant re-running of the same tests Potentially longer product cycles
101
Effects on Testing: Linear Models Requirements must be well-reviewed early Test plan can be written early When System Testing does start, you are on
the critical path
102
Effects on Testing: Iterative Models Testing starts as soon as the first level of
functionality is reached Test plan is written as you go Requests for design changes can be more
easily made
-
22/08/2011
35
103
Software Development stageRequirements
104
Software Development Stages Project/Product Planning Requirements and Design Coding and Documentation Testing and fixing bugs Maintenance and Enhancement
105
Why Projects are CancelledProject Impaired Factors % of Responses
1. Incomplete Requirements 13.1%2. Lack of User Involvement 12.4%3. Lack of Resources 10.6%4. Unrealistic Expectations 9.9%5. Lack of Executive Support 9.3%6. Changing Requirements & Specifications 8.7%7. Lack of Planning 8.1%8. Didn't Need It Any Longer 7.5%9. Lack of IT Management 6.2%10. Technology Illiteracy 4.3%Other 9.9%
-
22/08/2011
36
106
Why Projects are ChallengedProject Challenged Factors % of Responses
1. Lack of User Input 12.8%2. Incomplete Requirements & Specifications 12.3%3. Changing Requirements & Specifications 11.8%4. Lack of Executive Support 7.5%5. Technology Incompetence 7.0%6. Lack of Resources 6.4%7. Unrealistic Expectations 5.9%8. Unclear Objectives 5.3%9. Unrealistic Time Frames 4.3%10. New Technology 3.7%Other 23.0%
107
Why Projects SucceedProject Success Factors % of Responses
1. User Involvement 15.9%2. Executive Management Support 13.9%3. Clear Statement of Requirements 13.0%4. Proper Planning 9.6%5. Realistic Expectations 8.2%6. Smaller Project Milestones 7.7%7. Competent Staff 7.2%8. Ownership 5.3%9. Clear Vision & Objectives 2.9%10. Hard-Working, Focused Staff 2.4%Other 13.9%
108
Software Requirements According to software engineering theory,
software requirements should contain a finite list of behaviors and features, and each requirement should be written to be verifiable.
Given a finite list of requirements and a set of completion criteria, requirements-based testing becomes a feasible process.
-
22/08/2011
37
109
Typical Requirements (FURPS) Functionality does it do what I need? Usability is it easy to do what I need? Reliability can it do what I need when I
need it? Performance does it do what I need in a
reasonable amount of time? Supportability what if is DOESNT do what I
need?
110
Functionality Requirements Correctness
Extent to which a program satisfies its specifications and meets customer expectations
Features/Capabilities Generality Security
Extent to which access to software or data by unauthorized persons can be controlled
111
Usability Requirements Ease of use
Measures the effort required to learn, operate, prepare input for, and interpret the output of, a program
Intuitive Human factors Consistency Documentation
-
22/08/2011
38
112
Reliability Requirements Frequency and severity of failure - MTBF Recoverability Predictability Accuracy
113
Performance Requirements Speed Efficiency Resource consumption Throughput Response time
114
Supportability Requirements Testability Adaptability Maintainability
Effort required to locate and fix errors Serviceability
Effort required to modify/update a program Installability Portability
-
22/08/2011
39
115
Contents
Testing and requirementsLearn to think like a tester
116
Testing Requirements Usually done by review Are they complete? Are they reasonable? Can they be tested?
117
Requirements Testing Testing of the requirements Determine all testable requirements Determine all non-testable requirements Write and execute the tests
-
22/08/2011
40
118
Reviewing Software Requirements Learn to recognize what has not been
specified (limits, consequences) Learn to recognize what is vague or
ambiguous (sometimes, faster) As a tester:
Make sure testers are included in the reviews! Push back to get requirements clarified Always be thinking And just how am I going to
test this?
119
Reviewing the Design Helps determine how to test Shows what to look for Do you need to read the code?
120
Some examples
Some requirements statements Some examples of Test Cases
-
22/08/2011
41
121
Requirements Example 1 After a high temperature is detected, an
alarm must be raised quickly
122
Testable Requirements: Ex. 1 If the threshold temperature is reached, an
alarm is triggered. As long as the temperature remains below
the threshold temperature, no alarm is triggered.
If the temperature dips below the threshold (having gone above it), the alarm does not cease.
123
Requirements Example 2 Novice users should be able to learn the
interface with little training
When thinking about a test for this, ask yourself: What would be the criterion for saying the test passed? failed?
-
22/08/2011
42
124
Requirements Example 3There was a hidden requirement in this
feature change:The menu option 6. Utilities | 1. Security will
now display a new submenu rather than a Security dialog box. The submenu will have 3 items - 1. CSM, 2. CIS, and 3. Options.
125
Requirements Example 4How many testable requirements are here?The Access Profile choices for 2. CIS and 3.
Options will be grayed out unless those options have been purchased and installed.
126
Testable Requirements for Ex. 4
The Access Profile choice for CIS is available if and only if the CIS feature has been installed.
The Access Profile choice for Options is available if and only if the Options feature has been installed.
-
22/08/2011
43
127
Some Test Cases for Example 41. If the CIS option is installed, Access Profile
choices will be available for CIS2. If the Options option is installed, Access
Profile choices will be available for Options3. If the CIS option is not installed, Access
Profile choices for CIS will not be available 4. If the Options option is not installed,
Access Profile choices for Options will not be available
128
Requirements Example 5
An upgraded security system was being installed with enhanced password protection. Here is one requirement:
Existing passwords which do not conform to the new restrictions will be reset to the User ID, and a list of those users whose passwords were reset will be generated.
129
Requirements Example 6Untestable requirementNumber of invalid attempts before a user is
suspended: This count is reset when a successful logon is completed for the user. The default is that the User will never be suspended. The valid range is from 0 (never) to 10 attempts.
-
22/08/2011
44
130
Testable Requirements for Example 6Requirement: The number of invalid attempts before suspension
may be set to 2Test Case 1: After 2 bad attempts to log on, the Users USERID is
suspended.Test Case 2: A bad attempt, followed by a good attempt, can then
be followed by 2 bad attempts again before suspension.
The same holds for 1, 3-10
Requirement: The number of invalid attempts before suspension cannot be set to 11
Test Case 3: Attempt to set invalid attempts to 11, or verify it cant be done.
131
Requirements Example 7Vagueness in different disguises:The Advanced Color Module will improve the
print quality of the module.The Advanced Color Module firmware will
increase the number of color shades per panel from 4 - 9 shades to 25 - 30 shades.
The Advanced Color Module will reduce the tiling effects associated with previous modules.
132
Requirements Example 8A lot of test cases from just one sentence:
All new and modified screens will provide online help.
-
22/08/2011
45
133
Requirements Example 9 The user interface must be easy to use. In case of failure, the system must be easy to
recover and must suffer minimum loss of important data.
134
Lessons Learned Watch for words like always, never, every, etc.
They are difficult to verify. Words like must, should, shall, will, has to,
etc. signal potential requirements. Watch for hidden or unstated requirements. Watch for vague requirements: high, quickly,
better, faster, should be able to. For negative testing, ask What happens if the test
case fails? Ask yourself the question: How will I know if the
requirement is not met?
Structural testing
-
22/08/2011
46
136
Contents White box testing / Structural testing Graph Theory Control flow criteria Data flow criteria Graph Coverage for Source Code Testing State Behavior
137
White box testing / Structural testing
138
What is white-box software testing? What is white-box software testing?
Basic idea is to test a program based on the structure of a program
Pseudonyms: Glass Box, Clear Box, Crystal Box, Structural Testing
To distinguish from functional (requirements-based, black-box testing)
What do you need for white-box testing? A white-box testing model and test criteria A white-box test design and generation method Program source code
-
22/08/2011
47
139
White-Box Testing Objectives The major objective of white-box testing is to focus
on internal program structure, and discover all internal program errors Errors made by developers
The major testing focuses Program structures
Program statements and branches Various kinds of program paths
Program internal logic and data structures Program internal behaviors and states
140
Traditional White-Based Software Testing Methods Test Model: control flow graph
Test case design Various white-box testing methods generate test cases based on
a given control flow graph for a program
The goal is to Guarantee that all independent paths within a module have been
exercised at least once Exercise all logical decisions one their true and false sides Execute all loops at their boundaries and within their operational
bounds Exercise internal data structures to assure their validity Exercise all data define and use paths
141
No guarantees Executing all control flow elements does not
guarantee finding all faults Execution of a faulty statement may not always result in a
failure The state may not be corrupted when the statement is
executed with some data values Corrupt state may not propagate through execution to
eventually lead to failure What is the value of structural coverage?
Increases confidence in thoroughness of testing Removes some obvious inadequacies
-
22/08/2011
48
142
Graph theory
143
Definition of a Graph A set N of nodes, N is not empty
A set N0 of initial nodes, N0 is not empty
A set Nf of final nodes, Nf is not empty
A set E of edges, each edge from one node to another ( ni , nj ), i is predecessor, j is successor
144
Three Example Graphs0
21
3
N0 = { 0 }Nf = { 3 }
0
21
3
N0 = { }Nf = { 3 }
9
0
43
7
1
5
8
2
6
N0 = { 0, 1, 2 }Nf = { 7, 8, 9 }
Not aNot avalidvalidgraphgraph
-
22/08/2011
49
145
Paths in Graphs Path : A sequence of nodes [n1, n2, , nM]
Each pair of nodes is an edge Length : The number of edges
A single node is a path of length 0 Subpath : A subsequence of nodes in p is a subpath of p Reach (n) : Subgraph that can be reached from n
97 8
0 1 2
43 5 6
Paths
[ 0, 3, 7 ][ 1, 4, 8, 5, 1 ][ 2, 6, 9 ]
Reach (0) = { 0, 3, 4, 7, 8, 5, 1, 9 }Reach ({0, 2}) = GReach([2,6]) = {2, 6, 9}
146
Test Paths and SESEs Test Path : A path that starts at an initial node and ends at a final
node Test paths represent execution of test cases
Some test paths can be executed by many tests Some test paths cannot be executed by any tests
SESE graphs : All test paths start at a single node and end at another node Single-entry, single-exit N0 and Nf have exactly one node
0
2
1
63
5
4Double-diamond graph
Four test paths[ 0, 1, 3, 4, 6 ][ 0, 1, 3, 5, 6 ][ 0, 2, 3, 4, 6 ][ 0, 2, 3, 5, 6 ]
147
Tests and Test Paths path (t) : The test path executed by test t path (T) : The set of test paths executed by the set
of tests T
Each test executes one and only one test path A location in a graph (node or edge) can be reached
from another location if there is a sequence of edges from the first location to the second Syntactic reach : A subpath exists in the graph Semantic reach : A test exists that can execute that subpath
-
22/08/2011
50
148
Testing and Covering Graphs We use graphs in testing as follows :
Developing a model of the software as a graph Requiring tests to visit or tour specific sets of nodes, edges or subpaths
Test Requirements (TR) : Describe properties of test paths Test Criterion : Rules that define test requirements Satisfaction : Given a set TR of test requirements for a
criterion C, a set of tests T satisfies C on a graph if and only iffor every test requirement in TR, there is a test path in path(T)that meets the test requirement tr
Control flow (Structural) Coverage Criteria : Defined on a graph just in terms of nodes and edges
Data Flow Coverage Criteria : Requires a graph to be annotated with references to variables
149
Control flow test criteria
150
Node and Edge Coverage The first (and simplest) two criteria require that each node
and edge in a graph be executed
NodeNode CoverageCoverage (NC)(NC) :: TestTest setset TT satisfiessatisfies nodenode coveragecoverage onongraphgraph GG ifif onlyonly ifif forfor everyevery syntacticallysyntactically reachablereachable nodenode nninin NN,, therethere isis somesome pathpath pp inin path(T)path(T) suchsuch thatthat pp visitsvisits nn..
NodeNode CoverageCoverage (NC)(NC) :: TRTR containscontains eacheach reachablereachable nodenode ininGG..
We can abbreviate it in terms of the set of test requirements
-
22/08/2011
51
151
Node and Edge Coverage Edge coverage is slightly stronger than node coverage
EdgeEdge CoverageCoverage (EC)(EC) :: TRTR containscontains eacheach reachablereachable pathpath ofoflengthlength upup toto 11,, inclusive,inclusive, inin GG..
The length up to 1 allows for graphs with one node and no edges
NC and EC are only different when there is an edge and another subpath between a pair of nodes (as in an if-else statement)
Node Coverage : TR = { 0, 1, 2 }Test Path = [ 0, 1, 2 ]
Edge Coverage : TR = { (0,1), (0, 2), (1, 2) }Test Paths = [ 0, 1, 2 ]
[ 0, 2 ]
1
2
0
152
Covering Multiple Edges Edge-pair coverage requires pairs of edges, or subpaths of
length 2EdgeEdge--PairPair CoverageCoverage (EPC)(EPC) :: TRTR containscontains eacheach reachablereachablepathpath ofof lengthlength upup toto 22,, inclusive,inclusive, inin GG..
The length up to 2 is used to include graphs that have less than 2 edges
CompleteComplete PathPath CoverageCoverage (CPC)(CPC) :: TRTR containscontains allall pathspaths ininGG..
SpecifiedSpecified PathPath CoverageCoverage (SPC)(SPC) :: TRTR containscontains aa setset SS ofoftesttest paths,paths, wherewhere SS isis suppliedsupplied asas aa parameterparameter..
The logical extension is to require all paths
Unfortunately, this is impossible if the graph has a loop, so a weak compromise is to make the tester decide which paths:
153
Structural Coverage ExampleNode Coverage
TR = { 0, 1, 2, 3, 4, 5, 6 }Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ]
6
0
2
1
3 4
Edge CoverageTR = { (0,1), (0,2), (1,2), (2,3), (2,4), (3,6), (4,5), (4,6), (5,4) }Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 6 ]
Edge-Pair CoverageTR = { [0,1,2], [0,2,3], [0,2,4], [1,2,3], [1,2,4], [2,3,6],
[2,4,5], [2,4,6], [4,5,4], [5,4,5], [5,4,6] }Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 2, 3, 6 ]
[ 0, 2, 4, 5, 4, 5, 4, 6 ]
Complete Path CoverageTest Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 5, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 5, 4, 5, 4, 6 ]
5
-
22/08/2011
52
154
Loops in Graphs If a graph contains a loop, it has an infinite number of
paths
Thus, CPC is not feasible
SPC is not satisfactory because the results are subjective and vary with the tester
Attempts to deal with loops: 1970s : Execute cycles once ([4, 5, 4] in previous example, informal) 1980s : Execute each loop, exactly once (formalized) 1990s : Execute loops 0 times, once, more than once (informal description) 2000s : Prime paths
155
Simple Paths and Prime Paths Simple Path : A path from node ni to nj is simple if no node
appears more than once, except possibly the first and last nodes are the same No internal loops Includes all other subpaths A loop is a simple path
Prime Path : A simple path that does not appear as a proper subpath of any other simple path
Simple Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ],[ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ],[ 2, 3, 0, 1 ], [ 0, 1, 3 ], [ 0, 2, 3 ], [ 1, 3, 0 ], [ 2, 3, 0 ],[ 3, 0, 1 ], [3, 0, 2 ], [ 0, 1], [ 0, 2 ], [ 1, 3 ], [ 2, 3 ], [ 3, 0 ], [0], [1], [2], [3]
Prime Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ],[ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ],[ 2, 3, 0, 1 ]
1 2
0
3
156
Prime Path Coverage A simple, elegant and finite criterion that requires loops to be
executed as well as skipped
PrimePrime PathPath CoverageCoverage (PPC)(PPC) :: TRTR containscontains eacheach primeprime pathpath inin GG..
Will tour all paths of length 0, 1, That is, it subsumes node, edge, and edge-pair coverage
-
22/08/2011
53
157
Round Trips Round-Trip Path : A prime path that starts and ends at the
same node
SimpleSimple RoundRound TripTrip CoverageCoverage (SRTC)(SRTC) :: TRTR containscontains atat leastleastoneone roundround--triptrip pathpath forfor eacheach reachablereachable nodenode inin GG thatthat beginsbeginsandand endsends aa roundround--triptrip pathpath..
CompleteComplete RoundRound TripTrip CoverageCoverage (CRTC)(CRTC) :: TRTR containscontains allallroundround--triptrip pathspaths forfor eacheach reachablereachable nodenode inin GG..
These criteria omit nodes and edges that are not in round trips That is, they do not subsume edge-pair, edge, or node coverage
158
Prime Path Example The previous example has 38 simple paths Only nine prime paths
Prime Paths[ 0, 1, 2, 3, 6 ][ 0, 1, 2, 4, 5 ][ 0, 1, 2, 4, 6 ][ 0, 2, 3, 6 ][ 0, 2, 4, 5][ 0, 2, 4, 6 ]
[ 5, 4, 6 ][ 4, 5, 4 ][ 5, 4, 5 ]
Execute loop once
Execute loop more than once
5
0
2
1
3 4
6
Execute loop 0 times
159
Simple & Prime Path Example
5
0
2
1
3 4
6
Len 0[0][1][2][3][4][5][6] !
! means path terminates
Len 1[0, 1][0, 2][1, 2][2, 3][2, 4][3, 6] ![4, 6] ![4, 5][5, 4]
Len 2[0, 1, 2][0, 2, 3][0, 2, 4][1, 2, 3][1, 2, 4][2, 3, 6] ![2, 4, 6] ![2, 4, 5] ![4, 5, 4] *[5, 4, 6] ![5, 4, 5] * * means path
cycles
Len 3[0, 1, 2, 3][0, 1, 2, 4][0, 2, 3, 6] ![0, 2, 4, 6] ![0, 2, 4, 5] ![1, 2, 3, 6] ![1, 2, 4, 5] ![1, 2, 4, 6] !
Len 4[0, 1, 2, 3, 6] ![0, 1, 2, 4, 6] ![0, 1, 2, 4, 5] !
Prime Paths
Simple paths
-
22/08/2011
54
160
Data flow test criteria
161
Data Flow CriteriaGoal: Try to ensure that values are computed and used correctly Definition (def) : A location where a value for a variable is stored
into memory Use : A location where a variables value is accessed def (n) or def (e) : The set of variables that are defined by node n or
edge e use (n) or use (e) : The set of variables that are used by node n or
edge e
0
2
1
63
5
4X = 42
Z = X-8
Z = X*2 Defs: def (0) = {X}def (4) = {Z}def (5) = {Z}
Uses: use (4) = {X}use (5) = {X}
162
DU Pairs and DU Paths DU pair : A pair of locations (li, lj) such that a
variable v is defined at li and used at lj Def-clear : A path from li to lj is def-clear with
respect to variable v if v is not given another value on any of the nodes or edges in the path
Reach : If there is a def-clear path from li to lj with respect to v, the def of v at li reaches the use at lj
du-path : A simple subpath that is def-clear with respect to v from a def of v to a use of v
du (ni, nj, v) the set of du-paths from ni to nj du (ni, v) the set of du-paths that start at ni
-
22/08/2011
55
163
Data Flow Test Criteria Three criteria
Use every def Get to every use Follow all du-paths
164
Data Flow Test Criteria
AllAll--defsdefs coveragecoverage (ADC)(ADC) :: ForFor eacheach setset ofof dudu--pathspaths SS == dudu ((nn,,vv),), TRTR containscontains atat leastleast oneone pathpath dd inin SS..
AllAll--usesuses coveragecoverage (AUC)(AUC) :: ForFor eacheach setset ofof dudu--pathspaths toto usesuses SS ==dudu ((nnii,, nnjj,, vv),), TRTR containscontains atat leastleast oneone pathpath dd inin SS..
AllAll--dudu--pathspaths coveragecoverage (ADUPC)(ADUPC) :: ForFor eacheach setset SS == dudu ((ni,ni, njnj,,vv),), TRTR containscontains everyevery pathpath dd inin SS..
Then we make sure that every def reaches all possible uses
Finally, we cover all the paths between defs and uses
First, we make sure every def reaches a use
165
Data Flow Testing Example
0
2
1
63
5
4X = 42
Z = X-8
Z = X*2
All-defs for X
[ 0, 1, 3, 4 ]All-uses for X
[ 0, 1, 3, 4 ][ 0, 1, 3, 5 ]
All-du-paths for X
[ 0, 1, 3, 4 ][ 0, 2, 3, 4 ][ 0, 1, 3, 5 ][ 0, 2, 3, 5 ]
-
22/08/2011
56
166
Graph Coverage Criteria Subsumption
Simple Round Trip Coverage
SRTCNode CoverageNC
Edge Coverage
EC
Edge-Pair Coverage
EPC
Prime Path Coverage
PPC
Complete Path Coverage
CPC
Complete Round Trip Coverage
CRTC
All-DU-Paths Coverage
ADUP
All-uses Coverage
AUC
All-defs Coverage
ADC
167
Graph Coverage for Source Code
168
Graph Coverage for Source Code The most common application of graph criteria is to
program source Graph : Usually the control flow graph (CFG) Node coverage : Execute every statement Edge coverage : Execute every branch Loops : Looping structures such as for loops, while
loops, etc. Data flow coverage : Augment the CFG
defs are statements that assign values to variables uses are statements that use variables
-
22/08/2011
57
169
Control Flow Graphs A CFG models all executions of a method by describing control
structures Nodes : statements or sequences of statements (basic blocks) Edges : Transfers of control Basic Block : A sequence of statements such that if the first
statement is executed, all statements will be (no branches)
CFGs are sometimes annotated with extra information branch predicates defs uses
Rules for translating statements into graphs
170
CFG : The if Statementif (x < y){
y = 0;x = x + 1;
}else{
x = y;}
4
1
2 3
x >= yx < y
x = yy = 0
x = x + 1
if (x < y){
y = 0;x = x + 1;
}3
1
2 x >= yx < y
y = 0x = x + 1
171
CFG : The if-Return Statement
if (x < y){
return;}print (x);return;
3
1
2 x >= yx < y
return
print (x)return
No edge from node 2 to 3.The return nodes must be distinct.
-
22/08/2011
58
172
Loops Loops require extra nodes to be added
Nodes that do not represent statements or basic blocks
173
CFG : while and for Loopsx = 0;while (x < y){
y = f (x, y);x = x + 1;
}
1x = 0
43y =f(x,y)x = x + 1
x >= yx < y
for (x = 0; x < y; x++){
y = f (x, y);}
1
x = x + 1
2
3 5
x >= yx < y
y = f (x, y)
4
2dummy node
x = 0implicitly
initializes loop
implicitly increments loop
174
CFG : The case (switch) Structure
read ( c) ;switch ( c ){
case N:y = 25;break;
case Y:y = 50;break;
default:y = 0;break;
}print (y);
5
1 read ( c );c == N
y = 0;break;
2 43
c == Y default
y = 50;break;
y = 25;break;
print (y);
-
22/08/2011
59
175
Example Control Flowpublic static void computeStats (int [ ] numbers){
int length = numbers.length;double med, var, sd, mean, sum, varsum;
sum = 0;for (int i = 0; i < length; i++){
sum += numbers [ i ];} med = numbers [ length / 2 ];mean = sum / (double) length;varsum = 0;for (int i = 0; i < length; i++){
varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean));}var = varsum / ( length - 1.0 );sd = Math.sqrt ( var );System.out.println ("length: " + length);System.out.println ("mean: " + mean);System.out.println ("median: " + med);System.out.println ("variance: " + var);System.out.println ("standard deviation: " + sd);
}
176
Control Flow Graph for Statspublic static void computeStats (int [ ] numbers){
int length = numbers.length;double med, var, sd, mean, sum, varsum;
sum = 0;for (int i = 0; i < length; i++){
sum += numbers [ i ];} med = numbers [ length / 2 ];mean = sum / (double) length;varsum = 0;for (int i = 0; i < length; i++){
varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean));}var = varsum / ( length - 1.0 );sd = Math.sqrt ( var );System.out.println ("length: " + length);System.out.println ("mean: " + mean);System.out.println ("median: " + med);System.out.println ("variance: " + var);System.out.println ("standard deviation: " + sd);
}
i = 0
i >= length
i < lengthi++
i >= lengthi < length
i = 0
1
2
3
54
6
87
Control Flow TRs and Test Paths PPC
1
2
3
54
6
87
TRA. [ 3, 4, 3 ]B. [ 4, 3, 4 ]C. [ 7, 6, 7 ]D. [ 7, 6, 8 ]E. [ 6, 7, 6 ]F. [ 1, 2, 3, 4 ]G. [ 4, 3, 5, 6, 7 ]H. [ 4, 3, 5, 6, 8 ]I. [ 1, 2, 3, 5, 6, 7 ]J. [ 1, 2, 3, 5, 6, 8 ]
Test Pathsi. [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ]ii. [ 1, 2, 3, 4, 3, 4, 3,
5, 6, 7, 6, 7, 6, 8 ]iii. [ 1, 2, 3, 4, 3, 5, 6, 8 ]iv. [ 1, 2, 3, 5, 6, 7, 6, 8 ]v. [ 1, 2, 3, 5, 6, 8 ]
Prime Path Coverage
-
22/08/2011
60
178
Data Flow Coverage for Source def : a location where a value is stored into memory
x appears on the left side of an assignment (x = 44;) x is an actual parameter in a call and the method changes its value x is a formal parameter of a method (implicit def when method
starts) x is an input to a program
use : a location where variables value is accessed x appears on the right side of an assignment x appears in a conditional test x is an actual parameter to a method x is an output of the program x is an output of a method in a return statement
If a def and a use appear on the same node, then it is only a DU-pair if the def occurs after the use and the node is in a loop
179
Example Data Flowpublic static void computeStats (int [ ] numbers){
int length = numbers.length;double med, var, sd, mean, sum, varsum;
sum = 0;for (int i = 0; i < length; i++){
sum += numbers [ i ];} med = numbers [ length / 2 ];mean = sum / (double) length;varsum = 0;for (int i = 0; i < length; i++){
varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean));}var = varsum / ( length - 1.0 );sd = Math.sqrt ( var );System.out.println ("length: " + length);System.out.println ("mean: " + mean);System.out.println ("median: " + med);System.out.println ("variance: " + var);System.out.println ("standard deviation: " + sd);
}
180
Control Flow Graph for Stats
8
1
2
4
3
5
6
7
( numbers )sum = 0length = numbers.length
i = 0
i >= length
i < length
sum += numbers [ i ]i++
med = numbers [ length / 2 ]mean = sum / (double) length;varsum = 0i = 0
i >= length
i < length
varsum =
i++
var = varsum / ( length - 1.0 )sd = Math.sqrt ( var )print (length, mean, med, var, sd)
-
22/08/2011
61
181
CFG for Stats With Defs & Uses
8
1
2
4
3
5
6
7
def (1) = { numbers, sum, length }
def (2) = { i }
def (5) = { med, mean, varsum, i }use (5) = { numbers, length, sum }
use (3, 5) = { i, length }use (3, 4) = { i, length }
def (4) = { sum, i }use (4) = { sum, numbers, i }
use (6, 8) = { i, length }use (6, 7) = { i, length }
def (7) = { varsum, i }use (7) = { varsum, numbers, i, mean }
182
Defs and Uses Tables for StatsNode Def Use
1 { numbers, sum, length }
2 { i }34 { sum, i } { numbers, i, sum }5 { med, mean,
varsum, i }{ numbers, length, sum }
67 { varsum, i } { varsum, numbers, i,
mean }8 { var, sd } { varsum, length, var,
mean, med, var, sd }
Edge Use
(1, 2)(2, 3)(3, 4) { i, length }(4, 3)(3, 5) { i, length }(5, 6)(6, 7) { i, length }(7, 6)(6, 8) { i, length }
DU Pairs for Statsvariable DU Pairs
numbers (1, 4) (1, 5) (1, 7)length (1, 5) (1, 8) (1, (3,4)) (1, (3,5)) (1, (6,7)) (1, (6,8))med (5, 8)var (8, 8)sd (8, 8)mean (5, 7) (5, 8)sum (1, 4) (1, 5) (4, 4) (4, 5)varsum (5, 7) (5, 8) (7, 7) (7, 8)i (2, 4) (2, (3,4)) (2, (3,5)) (2, 7) (2, (6,7)) (2, (6,8))
(4, 4) (4, (3,4)) (4, (3,5)) (4, 7) (4, (6,7)) (4, (6,8))(5, 7) (5, (6,7)) (5, (6,8))(7, 7) (7, (6,7)) (7, (6,8))
-
22/08/2011
62
184
DU Paths for Stats No DuplicatesThere are 38 DU paths for Stats, but only 12 unique
[ 1, 2, 3, 4 ][ 1, 2, 3, 5 ][ 1, 2, 3, 5, 6, 7 ][ 1, 2, 3, 5, 6, 8 ][ 2, 3, 4 ][ 2, 3, 5 ]
[ 4, 3, 4 ][ 4, 3, 5 ][ 5, 6, 7 ][ 5, 6, 8 ][ 7, 6, 7 ][ 7, 6, 8 ]
5 expect a loop not to be entered
5 require at least one iteration of a loop
2 require at least two iteration of a loop
185
Test Cases and Test PathsTest Case : numbers = (44) ; length = 1Test Path : [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ]Additional DU Paths covered[ 1, 2, 3, 4 ] [ 2, 3, 4 ] [ 4, 3, 5 ] [ 5, 6, 7 ] [ 7, 6, 8 ]The five stars that require at least one iteration of a loopTest Case : numbers = (2, 10, 15) ; length = 3Test Path : [ 1, 2, 3, 4, 3, 4, 3, 4, 3, 5, 6, 7, 6, 7, 6, 7, 6, 8 ]DU Paths covered[ 4, 3, 4 ] [ 7, 6, 7 ]The two stars that require at least two iterations of a loopOther DU paths require arrays with length 0 to skip loopsBut the method fails with divide by zero on the statement
mean = sum / (double) length; A fault wasfound
186
Testing State Behavior
-
22/08/2011
63
187
Testing State Behavior A finite state machine (FSM) is a graph that
describes how software variables are modified during execution
Nodes : States, representing sets of values for key variables
Edges : Transitions, possible changes in the state
Off On
switch up
switch down
188
Finite State Machine Two Variables
Tropical Depressioncirculation = yes
windspeed < 39mph
Tropical Stormcirculation = yes
windspeed = 39..73 mph
Major Hurricanecirculation = yes
windspeed >= 110 mph
Hurricanecirculation = yes
windspeed 74..109 mph
Something Elsecirculation = no
windspeed = 0..38 mph
Other variables may exist but not be part of state
189
Finite State Machines are Common FSMs can accurately model many kinds of software
Embedded and control software (think electronic gadgets) Abstract data types Compilers and operating systems Web applications
Creating FSMs can help find software problems Numerous languages for expressing FSMs
UML statecharts Automata State tables (SCR) Petri nets
Limitations : FSMs are not always practical for programs that have lots of states (for example, GUIs)
-
22/08/2011
64
190
Annotations on FSMs FSMs can be annotated with different types of actions
Actions on transitions Entry actions to nodes Exit actions on nodes
Actions can express changes to variables or conditions on variables
These slides use the basics: Preconditions (guards) : conditions that must be true for
transitions to be taken Triggering events : changes to variables that cause transitions to
be taken
This is close to the UML Statecharts, but not exactly the same
191
Example Annotations
Closed Open
open elevator door
pre: elevSpeed = 0
trigger: openButton = pressed
192
Covering FSMs Node coverage : execute every state (state coverage) Edge coverage : execute every transition (transition coverage) Edge-pair coverage : execute pairs of transitions (transition-pair)
Data flow: Nodes often do not include defs or uses of variables Defs of variables in triggers are used immediately (the next state) Defs and uses are usually computed for guards, or states are
extended FSMs typically only model a subset of the variables
Generating FSMs is often harder than covering them
-
22/08/2011
65
193
Deriving FSMs With some projects, a FSM (such as a statechart) was created
during design Tester should check to see if the FSM is still current with respect
to the implementation If not, it is very helpful for the tester to derive the FSM Strategies for deriving FSMs from a program:
1. Combining control flow graphs2. Using the software structure3. Modeling state variables4. Using implicit or explicit specifications
Discussion of these strategies based on a digital watch Class Watch uses class Time
194
Class Watchclass Watch// Constant values for the button (inputs)private static final int NEXT = 0;private static final int UP = 1;private static final int DOWN = 2;// Constant values for the stateprivate static final int TIME = 5;private static final int STOPWATCH = 6;private static final int ALARM = 7;// Primary state variableprivate int mode = TIME;// Three separate times, one for each stateprivate Time watch, stopwatch, alarm;
public Watch () // Constructorpublic void doTransition (int button) // Handles inputspublic String toString () // Converts values
class Time ( inner class )private int hour = 0;private int minute = 0;
public void changeTime (int button)public String toString ()
195
// Takes the appropriate transition when a button is pushed.public void doTransition (int button){
switch ( mode ){
case TIME:if (button == NEXT)
mode = STOPWATCH;else
watch.changeTime (button);break;
case STOPWATCH:if (button == NEXT)
mode = ALARM;else
stopwatch.changeTime (button);break;
case ALARM:if (button == NEXT)
mode = TIME;else
alarm.changeTime (button);break;
default:break;
}} // end doTransition()
// Increases or decreases the time.// Rolls around when necessary.public void changeTime (int button){
if (button == UP){
minute += 1;if (minute >= 60){
minute = 0;hour += 1;if (hour >= 12)
hour = 0;}
}else if (button == DOWN){
minute -= 1;if (minute < 0){
minute = 59;hour -= 1;if (hour
-
22/08/2011
66
196
1. Combining Control Flow Graphs The first instinct for inexperienced developers is to
draw CFGs and link them together
This is really not a FSM
Several problems Methods must return to correct callsites built-in
nondeterminism Implementation must be available before graph can be built This graph does not scale up
Watch example
197
CFGs for Watch
5 6 7 8
1
2 3 4
11 12 13
9 10
14
doTransition ()1
12
6
8
2
4
10
6
8
2
4
10
changeTime ()
198
2. Using the Software Structure A more experienced programmer may map methods
to states
These are really not states
Problems Subjective different testers get different graphs Requires in-depth knowledge of implementation Detailed design must be present
Watch example
-
22/08/2011
67
199
Software Structure for Watch
button
inMain
inChangeTimeinDoTransition
button==NEXT / change mode
button==UP or button==DOWN
button==DOWN / change hour, minute
button==UP / change hour, minute
??? Not clear what triggers this
??? Should inChangeTime transit back to inMain???
200
3. Modeling State Variables More mechanical State variables are usually defined early First identify all state variables, then choose
which are relevant In theory, every combination of values for the
state variables defines a different state In practice, we must identify ranges, or sets
of values, that are all in one state Some states may not be feasible
201
State Variables in WatchConstants NEXT, UP, DOWN TIME, STOPWATCH, ALARMNon-constants int mode Time watch, stopwatch, alarmTime class variables int hour int minute
Not relevant, really just values
Merge into the three Time variables
-
22/08/2011
68
202
State Variables and ValuesThese three ranges actually represent quite a bit of thought and semantic domain knowledge of the program
Relevant State Variables
mode : TIME, STOPWATCH, ALARM watch : 12:00, 12:01..12:59, 01:00..11:59 stopwatch : 12:00, 12:01..12:59, 01:00..11:59 alarm : 12:00, 12:01..12:59, 01:00..11:59
Total 3*3*3*3 = 81 states
But the three watches are independent, so we only care about 3+3+3 = 9 states
(still a messy graph )
203
State Variable Model for Watchmode = TIMEwatch = 12:00
mode = TIMEwatch = 12:01.. 12:59
mode = TIMEwatch = 01:00..12:59
mode = STOPstopw = 12:00
mode = STOPstopw = 12:01.. 12:59
mode = STOPstopw = 01:00..12:59
mode = ALARMalarm = 12:00
mode = ALARMalarm = 12:01.. 12:59
mode = ALARMalarm = 01:00..12:59
nextnext
next nextnext next
nextnext next
nextnext next
next
next
next
next
next
next
nextnext
nextnext next
nextnext
next next
204
NonDeterminism in the State Variable Model Each state has three outgoing transitions on next
This is a form of non-determinism, but it is not reflected in the implementation
Which transition is taken depends on the current state of the other watch
The 81-state model would not have this non-determinism
This situation can also be handled by a hierarchy of FSMs, where each watch is in a separate FSM and they are organized together
-
22/08/2011
69
205
4. Using Implicit or Explicit Specifications Relies on explicit requirements or formal specifications that
describe software behavior
These could be derived by the tester
These FSMs will sometimes look much like the implementation-based FSM, and sometimes much like the state-variable model For watch, the specification-based FSM looks just like the state-
variable FSM, so is not shown
The disadvantage of FSM testing is that some implementation decisions are not modeled in the FSM
206
Tradeoffs in Applying Graph Coverage Criteria to FSMs There are two advantages
1. Tests can be designed before implementation2. Analyzing FSMs is much easier than analyzing source
There are three disadvantages1. Some implementation decisions are not modeled in the
FSM2. There is some variation in the results because of the
subjective nature of deriving FSMs3. Tests have to mapped to actual inputs to the program
the names that appear in the FSM may not be the same as the names in the program
Static testing
-
22/08/2011
70
Contents Reviews and the test process Types of review Static analysis
208
Reviews and the test process
209
Static testing Static: Not Dynamic, i.e. you dont execute
the code This is the process of testing some
documentation products
The primary goal of Static Testing is to reduce defects in the software by reducing defects in the documentation from which the software is developed
210
-
22/08/2011
71
Static techniques Do not execute code
People techniques Specific techniques
Desk checking Developer tests his/her own documentation
Walkthroughs Developer walks the participants through the
documentation Peer Reviews/Inspections
More formal documentation is distributed beforehand
211
Benefits of reviews Development productivity improvement Reduced development timescales Reduced testing time and cost Lifetime cost reductions Reduced fault levels Improved customer relations
212
Reviews are cost-effective 10 times reduction in faults reaching test,
testing cost reduced by 50% to 80% Freedman & Weinberg, Handbook of
Walkthroughs, Inspections & Technical Reviews reduce faults by a factor of 10
Yourdon, Structured Walkthroughs 25% reduction in schedules, remove 80% -
95% of faults at each stage, 28 times reduction in maintenance cost, many others Gilb & Graham, Software Inspection
213
-
22/08/2011
72
What can be Inspected? Anything written down can be Inspected
policy, strategy, business plans, marketing or advertising material, contracts
system requirements, feasibility studies, acceptance test plans
test plans, test designs, test cases, test results system designs, logical & physical software code user manuals, procedures, training material
214
What can be reviewed? anything which could be Inspected
i.e. anything written down plans, visions, big picture, strategic
directions, ideas project progress
work completed to schedule, etc. Should we develop this marketing options
215
What to review / Inspect?
216
Tests
Tests
Tests
Tests
Requirements
Design
Code
Functions
Integration T
Unit Test
Accept. Test
System Test
-
22/08/2011
73
Costs of reviews Rough guide: 5%-15% of development effort
half day a week is 10% Effort required for reviews
planning (by leader / moderator) preparation / self-study checking meeting fixing / editing / follow-up recording & analysis of statistics / metrics process improvement (should!)
217
Types of review
218
Types of review of documents Informal Review (undocumented)
widely viewed as useful and cheap (but no one can prove it!). A helpful first step for chaotic organisations.
Technical Review (or peer review) includes peer and technical experts, no management
participation. Normally documented, fault-finding. Can be rather subjective.
Decision-making Review group discusses document and makes a decision
about the content, e.g. how something should be done, go or no-go decision, or technical comments
219
-
22/08/2011
74
Types of review of documents Walkthrough
author guides the group through a document and his or her thought processes, so all understand the same thing, consensus on changes to make
Inspection formal individual and group checking, using sources
and standards, according to generic and specific rules and checklists, using entry and exit criteria, Leader must be trained & certified, metrics required
220
Reviews in general 1 Objectives / goals
validation & verification against specifications & standards
achieve consensus (excluding Inspection) process improvement (ideal, included in
Inspection)
221
Reviews in general 2 Activities
planning overview / kickoff meeting (Inspection) preparation / individual checking review meeting (not always) follow-up (for some types) metrics recording & analysis (Inspections and
sometimes reviews)
222
-
22/08/2011
75
Reviews in general 3 Roles and responsibilities
Leader / moderator - plans the review / Inspection, chooses participants, helps & encourages, conducts the meeting, performs follow-up, manages metrics
Author of the document being reviewed / Inspected Reviewers / Inspectors - specialised fault-finding roles for
Inspection Managers - excluded from some types of review, need to
plan project time for review / Inspection Others: e.g. Inspection/ review Co-ordinator
223
Reviews in general 4 Deliverables
Changes (edits) in review product Change requests for source documents
(predecessor documents to product being reviewed / Inspected)
Process improvement suggestions to the review / Inspection process to the development process which produced the
product just reviewed / Inspected Metrics (Inspection and some types of review)
224
Reviews in general 5 Pitfalls (they dont always work!)
Lack of training in the technique (especially Inspection, the most formal)
Lack of or quality of documentation - what is being reviewed / Inspected
Lack of management support Failure to improve processes
225
-
22/08/2011
76
Inspection is different
the document to be reviewed is given out in advance
typically dozens of pages to review
instructions are "please review this"
226
not just product, sources
chunk or sample
training, roles
Inspection is different
some people have time to look through it and make comments before the meeting (which is difficult to arrange)
the meeting often lasts for hours
227
entry criteria to meeting, may not be worth holding
2 max., often much shorter
The Inspection Process
228
SoftwareDevelopment
Stage
.
.
Planning
Kickoff
IndChk Meet Edit
ChangeRequest
Process Improvement
Entry
Next SoftwareDevelopment
Stage
Exit
-
22/08/2011
77
Static analysis
229
What can static analysis do? Remember: static techniques do not execute the
code
A form of automated testing check for violations of standards check for things which may be a fault
Descended from compiler technology a compiler statically analyses code, and knows a lot
about it, e.g. variable usage; finds syntax faults static analysis tools extend this knowledge can find unreachable code, undeclared variables,
parameter type mis-matches, uncalled functions & procedures, array bound violations, etc.
230
Data flow analysis
This is the study of program variables variable defined* where a value is stored into it variable used where the stored value is accessed variable is undefined before it is defined or when it
goes out of scope
231*defined should not be confused with declared
x = y + zIF a > b THEN read(S)
x is defined, y and z are used
a and b are used, S is defined
-
22/08/2011
78
Data flow analysis faults
232
n := 0read (x)n := 1while x > y do
beginread (y)write( n*y)x := x - n
end
Data flow anomaly: n isre-defined without being used
Data flow fault: y is usedbefore it has been defined(first time around the loop)
Control flow analysis Highlights:
nodes not accessible from start node infinite loops multiple entry to loops whether code is well structured, i.e. reducible whether code conforms to a flowchart grammar any jumps to undefined labels any labels not jumped to cyclomatic complexity and other metrics
233
Cyclomatic complexity cyclomatic complexity is a measure of the
complexity of a flow graph (and therefore the code that the flow graph
represents) the more complex the flow graph, the greater
the measure it can most easily be calculated as:
complexity = number of decisions + 1
234
-
22/08/2011
79
Which flow graph is most complex?
235
1
2 3 5
What is the cyclomatic complexity?
Example of control flow graph
236
Result = 0Right = 0DO WHILE more Questions
IF Answer = Correct THEN Right = Right + 1
ENDIFEND DOResult = (Right / Questions)IF Result > 60% THEN
Print "pass"ELSE
Print "failENDIF
do
if r=r+1
end
init
if
res
pass
fail
end
Pseudo-code:
Other static metrics lines of code (LOC) operands & operators (Halsteads metrics) fan-in & fan-out nesting levels function calls OO metrics: inheritance tree depth, number
of methods, coupling & cohesion symbolic evaluation
237
-
22/08/2011
80
Limitations and advantages Limitations
cannot distinguish "fail-safe" code from programming faults or anomalies (often creates overload of spurious error messages)
does not execute the code, so not related to operating conditions
Advantages can find faults difficult to "see" gives objective quality assessment of code
238
Summary Reviews help to find faults in development
and test documentation, and should be applied early
Types of review: informal, walkthrough, technical / peer review, Inspection
Static analysis can find faults and give information about code without executing
239
Functional testing
-
22/08/2011
81
241
Contents Introduction to functional testing Functional testing techniques
Boundary Value testing Equivalence Class testing Special Value testing Decision Tables
242
Introduction to functional testing
243
Functional testing Functional testing: Deriving test cases from
program specifications Functional refers to the source of information used in
test case design, not to what is tested Also known as:
specification-based testing (from specifications) black-box testing (no view of the code)
Functional specification = description of intended program behavior either formal or informal
-
22/08/2011
82
244
Systematic vs Random Testing Random (uniform):
Pick possible inputs uniformly Avoids designer bias
A real problem: The test designer can make the same logical mistakes and bad assumptions as the program designer (especially if they are the same person)
But treats all inputs as equally valuable Systematic (non-uniform):
Try to select inputs that are especially valuable Usually by choosing representatives of classes that are apt
to fail often or not at all Functional testing is systematic testing
245
Why Not Random? Non-uniform distribution of faults Example: Java class roots applies quadratic
equation
Incomplete implementation logic: Program does not properly handle the case in which b2 - 4ac < 0 and a=0
Random sampling is unlikely to choose a=0.0 and b=0.0
246
Systematic Partition TestingFailure (valuable test case)No failure
Failures are sparse in the space of possible inputs ...
... but dense in some parts of the space
If we systematically test some cases from each part, we will include the dense parts
Functional testing is one way of drawing pink lines to isolate regions with likely failures
The
sp
ace
o
f po
ssib
le in
put v
alu
es
(the
ha
ysta
ck)
-
22/08/2011
83
247
Functional testing: exploiting the specification Functional testing uses the specification (formal or
informal) to partition the input space E.g., specification of roots program suggests division
between cases with zero, one, and two real roots Test each category, and boundaries between
categories No guarantees, but experience suggests failures often lie
at the boundaries (as in the roots program)
248
Why functional testing? Timely
Often useful in refining specifications and assessing testability before code is written
Effective finds some classes of fault (e.g., missing logic) that can
elude other approaches Widely applicable
to any description of program behavior serving as spec at any level of granularity from module to system testing.
Economical typically less expensive to design and execute than
structural (code-based) test cases
249
Early functional test design Program code is not necessary
Only a description of intended behavior is needed Even incomplete and informal specifications can be used
Although precise, complete specifications lead to better test suites
Early functional test design has side benefits Often reveals ambiguities and inconsistency in spec Useful for assessing testability
And improving test schedule and budget by improving spec Useful explanation of specification
or in the extreme case (as in XP), test cases are the spec
-
22/08/2011
84
250
Functional versus Structural:Classes of faults
Different testing strategies (functional, structural) are most effective for different classes of faults
Functional testing is best for finding design faults
Structural testing is best for finding programming faults
251
Functional vs structural test: granularity levels Functional test applies at all granularity levels:
Unit (from module interface spec) Integration (from API or subsystem spec) System (from system requirements spec) Regression (from system requirements + bug history)
Structural (code-based) test design applies to relatively small parts of a system: Unit Integration
252
Steps: From specification to test cases 1. Decompose the specification
If the specification is large, break it into independently testable features to be considered in testing
2. Select representatives Representative values of each input, or Representative behaviors of a model
Often simple input/output transformations dont describe a system. We use models in program specification, in program design, and in test design
3. Form test specifications Typically: combinations of input values, or model behaviors
4. Produce and execute actual tests
-
22/08/2011
85
253
From specification to test cases
254
Simple example: Postal code lookup
Input: ZIP code (5-digit US Postal code)
Output: List of cities What are some
representative values (or classes of value) to test?
255
Example: Representative values
Simple example with one input, one output
Note prevalence of boundary values (0 cities, 6 characters)
and error cases
Correct zip code With 0, 1, or many cities
Malformed zip code Empty; 1-4 characters; 6 characters; very long Non-digit characters Non-character data
-
22/08/2011
86
256
Functional testing techniques Boundary Value testing Equivalence Class testing Special Value testing Decision Tables
257
Examples
258
The Triangle ProgramIn order for 3 integers a, b, and c to be
the sides of a triangle, we must have c1 a + b > cc2 a + c > bc3 b + c > a
A triangle is: Equilateral if all 3 sides are equal Isosceles if 2 sides are equal Scalene if no two sides are equal
a b
c
-
22/08/2011
87
259
The Triangle Program The Triangle Program reads in 3 integers and
decides if they form an equilateral triangle, an isosceles triangle, a scalene triangle, or if they dont form a triangle at all
The logic of the program is clear, but complex, due to the relationships between the inputs and the outputs
We assume that each side length is between 1 and 200
260
NextDate Function This program reads in a date in a certain
format and prints out the next days date. For example, an input of 03 31 1998 gives an
output of 04 01 1998. The year is constrained to lie between 1812
and 2012 inclusive
261
The Commission Problem A salesman sells locks, stocks and barrels On a monthly basis, the salesperson reports how
many of each he or she has sold Due to manufacturing constraints, we have:
1 #locks 701 #stocks 801 #barrels 90
Goal: Determine total sales and the commission
-
22/08/2011
88
262
The Commission Problem The parts sell for the following:
Locks $45 eachStocks $30 eachBarrels $25 each
Commission is computed on sales:10% on sales up to and including $1000; 15% on the next $800; and 20% on any amount over $1800
This program has calculation logic to test inputs and outputs are straightforward
263
Boundary Value testing
264
Boundary/Limits Testing Test values, sizes or quantities near the design
limits Value limits Length limits Volume limits Null strings vs Empty strings
Errors tend to occur near the extreme values of inputs
Robustness: How does the software react when boundaries are exceeded?
-
22/08/2011
89
265
Single Fault AssumptionThe Single Fault assumption from reliability
states:Failures are rarely the result of the
simultaneous occurrence of two (or more) faults.
266
Extensions of Boundary Testing Boundary value testing
Stay within the limits Robustness testing
Exceed the limits Worst-case testing
Like boundary value testing above, but discard Single Fault Assumption
Robust worst-case testing Like robustness testing above, but discard Single Fault
Assumption
267
Boundary Value TestingThe basic idea of boundary value analysis is to
use input variable values at their minimum, just above the minimum, at a nominal value, just below the maximum, and at the maximum
-
22/08/2011
90
268
Input Boundary Value TestingInput Boundary Value Testinga b
Test cases for a variable x, where a x b
Experience shows that errors occur more frequently for extreme values of a variable
x
x(min) x(min+) x(nom) x(max -) x(max)
269
Input Boundary Value Testing Input Boundary Value Testing ((2 2 Variables)Variables)
Test cases for variables x1 and x2, wherea x1 b and c x2 d
As in reliability theory, two variables rarelyassume their extreme values simultaneously.
a b
c
d
x2
x1
270
Robustness Testing An extension of Boundary Value testing:
Include values below the minimum value and above the maximum value
This is a form of what we call negative testing -- how does the program handle errors in the input?
This type of testing may not be possible with some strongly-typed languages, GUIs with fixed palette values or drop-down lists, etc.
-
22/08/2011
91
271
Robustness TestingRobustn