1
CENG 557
Software Quality Assurance
&
Testing
Spring 2015-16
Dr. Sadık Eşmelioğlu
General Information
Dr. Sadık Eşmelioğlu CENG 557 2
E-Mail : [email protected]
Phone : +90 (312) 233 13 43
Hours
Class : Tuesday 18:00 – 20:50
Office : Tuesday 20:50 – 21:50
Room
Class : A202
Office : A202 – New Campus L-215
Text Book and References
Dr. Sadık Eşmelioğlu CENG 557 3
Text Book:
“Software Quality Assurance: From Theory to
Implementation”, D. Galin, Addison Wesley, 2004, ISBN 978-0-
201-70945-2.
Reference Material:
Software Testing Techniques, B. Beizer, Van Nostrand Reinhold,
1992.
Software Engineering Body of Knowledge, Chapter 5 Software
Testing, 2004.
http://www.swebok.org/ironman/pdf/SWEBOK_Guide_2004.p
df
2
Grading Components
Dr. Sadık Eşmelioğlu CENG 557 4
Midterm 25%
Will cover the concepts learned thus far.
Closed books, notes, etc.
Presentation 10%
15-Minute Presentation
Each student will pick a test technique as topic
Research using references
PC or Overhead Projector will be used, copies to be distributed to the class
Should include examples
50% of the grade will be given by the classmates (Anonymously)
Assignments 10%
4-5 assignments (Approx. every other week)
Project 20%
Pick a small piece of software
A list of Requirements for the software should be available
Write test plan, test design specs., and test cases
Final 35%
Will cover all the topics learned
Closed books, notes, etc.
Course Outline
Dr. Sadık Eşmelioğlu CENG 557 5
SQA Concepts
Basic notions: Quality Assurance, Detection vs. Prevention, Verification & Validation, testing
Testing Concepts
Definition, Types and Levels of testing, Black vs. White Box testing
Test Techniques
White Box techniques
Black Box techniques
Test Planning
Test Plans
Test Design Specifications
Test Cases
Test Metrics
Pre-process metrics: Estimation
In-process metrics: Process Management
End-process metrics: Process Improvement
Test Management
Test planning, resource management, test reporting, tools
Test Automation
What and How to automate
Calendar
Dr. Sadık Eşmelioğlu CENG 557 6
Wk Date Subject
1 09-Feb Introduction
2 16-Feb SQA Concepts
3 23-Feb Testing Concepts
4 01-Mar Test Techniques
5 08-Mar Test Techniques
6 15-Mar Test Techniques
7 22-Mar Test Planning
8 29-Mar Midterm
9 05-Apr Presentations
10 12-Apr Presentations
11 19-Apr Test Metrics
12 26-Apr Test Metrics
13 03-May Project Presentations
14 10-May Test Management and Automation
3
CMM Levels – Characteristics
Dr. Sadık Eşmelioğlu CENG 557 7
Level Features
1 Initial Add hoc, no defined processes
Success by heroes
2 Repeatable Basic project management
Repeatable success (same team)
3 Defined Processes defined
Consistently used, applied
4 Managed Metrics gathered
Used in process management
5 Optimizing Metrics analyzed
Used in process improvement
CMM Levels – Processes
Dr. Sadık Eşmelioğlu CENG 557 8
Level Processes
1 Initial Saving the day, act according to the situation
2 Repeatable
Requirements management
Software project planning
Software project monitoring
Subcontractor management
Software quality assurance
Software configuration management
3 Defined
Organizational process focus
Organizational process definition
Training Program
Integrated software management
Software product engineering
Coordination between groups
Peer reviews
4 Managed Quantitative project management
Software quality management
5 Optimizing
Defect prevention
Technology change management
Process change management
Software Development Process
Dr. Sadık Eşmelioğlu CENG 557 9
Requirements
Design
Coding
Testing
Maintenance
Defect
SQA
4
Concepts – Testing in SQA
Dr. Sadık Eşmelioğlu CENG 557 10
Process
and
Procedures
SQA
Change
Control … Standards
and
Templates
Verification
and
Validation
Inspection Review Audit Test
Software Dev – Waterfall Model
Dr. Sadık Eşmelioğlu CENG 557 11
R
D
C
T
Software Dev – Incremental Model
Dr. Sadık Eşmelioğlu CENG 557 12
R D C T
R D C T
R D C T
PH 1
PH 2
PH 3
5
Software Dev – Cyclic Model
Dr. Sadık Eşmelioğlu CENG 557 13
R
D
C
T
R
D
C
T
SQA, V&V, and Testing
Dr. Sadık Eşmelioğlu CENG 557 14
SQA is a set of activities and processes to ensure that an
end product is of desired quality.
V&V ensures that the product of each phase of software
development confirms to the requirements of that phase.
Testing is a process to detect faults in the end product.
V&V is a part of the QA process and Testing is a part of the
V&V process.
Detection vs. Prevention
Dr. Sadık Eşmelioğlu CENG 557 15
QA focuses not only on detection but prevention of
faults (a.k.a. defects, failures, errors, etc.) as well.
Establish a suitable development environment
Define processes and procedures to be followed
Define how the effectiveness of the processes and
procedures are to be measured and improved
Etc.
V&V and Testing focus on detection only.
Inspections, Audits, Reviews, Walkthroughs (Static)
Unit, Integration, System, and Acceptance Testing (Mostly
Dynamic)
6
Cost of Fixing a Defect
Dr. Sadık Eşmelioğlu CENG 557 16
12
5
10
20
0
2
4
6
8
10
12
14
16
18
20
Cost
Requirements
Design
Coding
Testing
Maintenance
Software Testing
Dr. Sadık Eşmelioğlu CENG 557 17
Software testing consists of the dynamic verification of the
behavior of a program on a finite set of test cases, suitably
selected from the usually infinite executions domain, against
the expected (“specified and implied”) behavior.
(SWEBOK, 2004)
Dynamic : Code is executed
Finite : Limited # of Test Cases, not exhaustive
Selected : Appropriate set of Test Cases
Expected : Describing the desired outcome
(Pass/Fail Criteria – Test Oracle)
Scope of Testing
Dr. Sadık Eşmelioğlu CENG 557 18
The purpose of testing is to find faults. If no faults are
found, then the testing cannot be considered successful!
Testing should cover not only the expected scenarios
(Sunny Day Scenarios), but unexpected scenarios (Rainy
Day Scenarios) as well.
Testing starts with requirements.
7
System Under Test (SUT)
Dr. Sadık Eşmelioğlu CENG 557 19
Mod
A
Mod
B
Mod
D
Mod
C
Mod
E
Sys X
SUT
Sys Y
Input/Output
The function of each line is described in documents called
Specifications/Requirements
UI
XI
Levels of Testing wrt Target
Dr. Sadık Eşmelioğlu CENG 557 20
Unit Testing
Testing a module wrt a low level design specification
Debuggers, Simulators, and Stubs are used for testing
Integration Testing
Testing several modules together wrt a high level design specification
Debuggers, Simulators, Drivers, and Stubs are used for testing
Top-down, Bottom-up, or Architecture driven approaches can be used
System Testing
Testing the SUT wrt a system specification
Simulators are used for stress and capacity testing
Levels of Testing wrt Objective
Dr. Sadık Eşmelioğlu CENG 557 21
Installation (OA&M)
Cannot use the system if cannot install properly. Other typical administrative function are also tested, backup, restore, user administration, etc.
Alfa & Beta
Friendly users. No specific plan for testing. Users report the problems they encounter.
Functional
Heart of testing. Conformity to specifications
Performance
Users are not very patient. Testing response time under different load conditions
Stress
Break SUT. Increase the load and capacity until the system breaks.
Back-to-back
Just compare results. Same tests executed on two version of a product and results are compared.
8
Levels of Testing wrt Objective (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 22
Recovery
System crashed. Can system be restarted without any major loss of data?
Configuration
Same SW different environments. Unix/Windows, ORACLE/SYBASE, etc.
Usability
User Friendly. On-line Help available, selecting instead of typing, comprehensible error messages, etc.
Acceptance
Before I pay. Customer needs to agree to what they are getting is what they asked for.
Reliability
This is not what I got before! System behaves in a consistent manner. When a test is executed twice under the same conditions, results should be the same.
Regression
New version old functionality. When new functionality is working, what used to work before should still work.
Test Techniques
Dr. Sadık Eşmelioğlu CENG 557 23
Intuition-Based
Ad Hoc
Specification-Based
Equivalence partitioning
Boundary-value analysis
Decision table
Finite-state machine-based
Testing from formal specifications
Random testing
Code-Based
Reference models for code-based testing (flow graph, call graph)
Control flow-based criteria
Data flow-based criteria
Test Techniques (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 24
Fault-based
Error guessing
Mutation testing
Usage-based
Operational profile
SRET
Based on nature of application Object-oriented testing
Component-based testing
Web-based testing
GUI testing
Testing of concurrent programs
Protocol conformance testing
Testing of distributed systems
Testing of real-time systems
Testing of scientific software
9
Black-Box Testing
Dr. Sadık Eşmelioğlu CENG 557 25
Equivalence partitioning
Boundary-value analysis
Decision table
Finite-state machine-based
Testing from formal specifications
Error guessing
Random testing
Operational profile
SRET
White-Box Testing
Dr. Sadık Eşmelioğlu CENG 557 26
Reference models for code-based testing (flow graph, call
graph)
Control flow-based criteria
Data flow-based criteria
Mutation testing
Black Box/White Box Testing
Dr. Sadık Eşmelioğlu CENG 557 27
Black Box Testing
SUT is treated as a closed box
Does not pay attention to how the software is written
System Testing is mostly Black Box
White Box Testing
SUT is treated as an open box
Knowledge of code and design is needed
Unit and Integration testing is mostly White Box
10
An Example
Dr. Sadık Eşmelioğlu CENG 557 28
Requirements for “Stock Purchase Wizard” (SPW)
1. Stock Selection(ss)
1. Screen will include a select box which lists all the stocks available for purchase
2. The user will be able to select only one stock
3. The selection will be done either by double clicking on a stock or single clicking and pressing “Ekle” button.
4. The selected stock symbol will be automatically populated in the “Hisse” text box.
2. Number of Stocks (ns)
1. The number of stocks to be purchased will be determined by two entries:
1. A box for entering the number “Miktar” and
2. Another box to indicate whether the number denotes lots or individual number of stocks.
3. Purchase Price (pp)
1. The purchase price will be selected from a box of valid numbers. The valid values will be automatically generated and can be “Serbest” or a number rounded to the nearest 0.01 between 10% below or above the opening price of the stock.
2. Once the purchase price is selected, the total purchase amount (# of stocks * purchase price) will be automatically populated into the “Tutar” box
An Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 29
4. Number of Sessions (sn)
1. The user will be able to select from a pull down menu, the number of sessions the
order will be in the market.
2. The valid values in this menu are
1. “Bir” if the order is entered during the afternoon session
2. “Bir” and “Iki” if the order is entered during the morning session
3. The order will be canceled if the stock did not reach the purchase price indicated by
the user during the number of sessions selected by the user
5. Confirming the order (co)
1. Once the user fills all the boxes and clicks on “Devam”, the program will respond
with
1. A six-digit confirmation number if the order is valid
2. An error message if one of the fields is missing or invalid
An Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 30
11
An Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 31
TC Id Req. Id Priority Description
SPW-FN-ss-01 1.1 H Check stock list to make sure it contains all the stocks avail. for purchase
SPW-FN-ss-02 1.2 H Click one stock and check if highlighted
SPW-FN-ss-03 1.2 M Try to click on more than one stock with clk, shift-clk and cntl-clk
SPW-FN-ss-04 1.3, 1.4 H Click on one stock, press “Ekle” and check “Hisse” box
SPW-FN-ss-05 1.3, 1.4 H Dbl-clk on one stock and check “Hisse” box
SPW-FN-ns-01 2.1.1 L Enter a negative number in “Miktar”
SPW-FN-ns-02 2.1.1 L Enter zero in “Miktar”
SPW-FN-ns-03 2.1.1 M Enter a positive number in “Miktar”
SPW-FN-ns-04 2.1.2 M Check if selection for unit of stocks contain only “Lot” and “Adet”
SPW-FN-pp-01 3.1 M Check if “Emir Fiyati” selection contains only Serbest and valid values
SPW-FN-ns-02 3.2 M Select “Serbest” and check the “Tutar” box
SPW-FN-ns-03 3.2 M Select lowest value and check the “Tutar” box
SPW-FN-ns-04 3.2 M Select highest value and check the “Tutar” box
SPW-FN-ns-05 3.2 M Select a value in the middle and check the “Tutar” box
Test Scenarios for “Stock Purchase Wizard”
An Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 32
TC Id Req. Id Priority Description
SPW-FN-se-01 4.1 Check if there is a pull down menu to select the number of sessions
SPW-FN-se-02 4.2 Check if “Gecerli Seans” box contains only “Bir” before noon
SPW-FN-se-03 4.2 Check if “Gecerli Seans” box contains only “Bir” and “Iki” after noon
SPW-FN-se-04 4.3 Enter an order for one session with a price which is not reached, check if
order still exists during the second session
SPW-FN-se-05 4.3 Enter an order for two sessions with a price which is not reached, check if
order still exists during the second session
SPW-FN-se-06 4.3 Enter an order for two sessions with a price which is not reached, check if
order still exists after the second session
SPW-FN-co-01 5.1.1 Enter a valid order, click “Devam” and check the confirmation number
SPW-FN-co-02 5.1.2 Enter an invalid order, click “Devam” and check the error message
Test Scenarios for “Stock Purchase Wizard”
Another Example
Dr. Sadık Eşmelioğlu CENG 557 33
Requirements for “Vending Machine”
1. The vending machine will accept only ¢5 and ¢10 coins
2. The cost of a can of soda is ¢25
3. When the coins inserted reach ¢25 or more, a can of soda will be dropped and excess coins will be returned.
4. After a soda can is dropped, the machine will be reset to initial state.
12
Another Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 34
0
30
10
5
15
20 25
25
20
25 30
15
20
25 30
25
15
10
20
25 30
25
20
25 30
Decision Tree
13 TCs
Another Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 35
State Diagram
TC #1 5 + 5 + 5 + 5 + 5 3 TCs
Each State and path must be
traversed at least once.
0 5 10 15 20 5/00
5/00 5/00
5/00
5/10
10/15
10/00 10/00
10/00 10/10
TC #2 5 + 10 + 10
TC #3 10 + 10 + 10
x/yz
x 5 – 10
y 0 no soda
1 soda
z 0 no coin
5 5 kurus back
Yet Another Example
Dr. Sadık Eşmelioğlu CENG 557 36
Requirements for “Home Alarm System”
1. The system is activated and deactivated by entering the security code
2. The system has three sensors: One is connected to the main door, one to the living room window and the last one to the bedroom window.
3. If at least one of the sensors indicate an entry, the system will sound the alarm if it is in active mode.
4. If the system is not in the active mode, no alarm will be sounded.
13
Yet Another Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 37
SA DR W1 W2 Alarm
N N N N Off
N N N Y Off
N N Y N Off
N N Y Y Off
N Y N N Off
N Y N Y Off
N Y Y N Off
N Y Y Y Off
Y N N N Off
Y N N Y On
Y N Y N On
Y N Y Y On
Y Y N N On
Y Y N Y On
Y Y Y N On
Y Y Y Y On
Test all scenarios: 16 TCs
Yet Another Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 38
SA DR W1 W2 Alarm
N N N N Off
N N N Y Off
N N Y N Off
N N Y Y Off
N Y N N Off
N Y N Y Off
N Y Y N Off
N Y Y Y Off
Y N N N Off
Y N N Y On
Y N Y N On
Y N Y Y On
Y Y N N On
Y Y N Y On
Y Y Y N On
Y Y Y Y On
Testing for single-mode
faults: 2 TCs
Each input value is tested
once. This testing
guarantees finding single-
mode faults.
Yet Another Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 39
SA DR W1 W2 Alarm
N N N N Off
N N N Y Off
N N Y N Off
N N Y Y Off
N Y N N Off
N Y N Y Off
N Y Y N Off
N Y Y Y Off
Y N N N Off
Y N N Y On
Y N Y N On
Y N Y Y On
Y Y N N On
Y Y N Y On
Y Y Y N On
Y Y Y Y On
Orthogonal Array Testing: 6 TCs
Orthogonal Array Testing
(a.k.a. Pairwise Testing)
means, each input value is
tested with each other input
value. This testing
guarantees finding double-
mode faults.
14
Another Decision Table Example
Dr. Sadık Eşmelioğlu CENG 557 40
Requirements for “FlyHigh Airline Reservation System”
The SUT is an online reservation program for an airline, called FLYHIGH.
The requirements for this software are listed below:
FLYHIGH flies from Istanbul to only four destinations, Ankara, Izmir,
Antalya, and Paris.
Flights to domestic cities have only Business and Economy classes, but
international flights have First Class as well.
The customers can select an isle, middle or a window seat.
The customers can also request vegetarian or regular food.
Once the user selects the Destination, Class, Seat Type, and Food Type, the
software will either make the reservation if the selections are valid or
print an error message if the selections are not valid.
Total scenarios = 4x3x3x2 = 72
Another Decision Table Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 41
Dest Class Seat Food Out
Pairwise scenarios = 12
ANK ANK
IZM
ANT
PAR
BUS
ECO
FST
BUS
ECO
FST
ANK
ANK
IZM
IZM
IZM
BUS
ECO
FST
ANT
ANT
BUS
ECO
FST
PAR
PAR
PAR
BUS
ECO
FST
ISL
MDL
WIN
ISL
MDL
WIN
ISL
MDL
WIN
ISL
MDL
WIN
ISL
MDL
WIN
REG
VEG
REG
VEG
VEG
REG
REG
VEG
VEG
REG
ANT
REG
VEG
REG
VEG
VAL
VAL
VAL
VAL
Err
VAL
Err
VAL
Err
VAL
VAL
VAL
DEST CLASS SEAT FOOD
One More Example
Dr. Sadık Eşmelioğlu CENG 557 42
Requirements for “ATM – Withdrawing Money”
1. ATM allows cash to be withdrawn in TL20 increments.
2. Maximum cash limit for withdrawals in a day is TL600
3. If a valid amount is entered, the transaction is successful, if not an error message is displayed according to the error
1. E1: The amount entered should be in TL20 increments
2. E2: The amount entered exceeds daily limit
3. E3: Invalid number (All other errors)
15
One More Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 43
TC ID Input Expected
Result
ATM-FN-mw-1
20 S
ATM-FN-mw-2 300 S
ATM-FN-mw-3 600 S
ATM-FN-mw-4 10 E1 (or E3?)
ATM-FN-mw-5 620 E2
ATM-FN-mw-6 250 E1
Equivalence Partitioning
1. Valid Values (20, 300, 600)
2. Invalid Values
1. <20 (10)
2. >600 (620)
3. Invalid Number (250)
An Additional Example
Dr. Sadık Eşmelioğlu CENG 557 44
Requirements for “Calculate Commission”
1. Minimum commission is TL1
2. If Amount > TL500, then commission is 0.2%
3. If Amount > TL5K, then commission is 0.15%
4. If Amount > TL50K, then commission is 0.1%
Pseudo Code for “Calculate Commission”
1 If Amount > 50K
2 then Commission = Amount * 0.001
3 Else If Amount > 5K
4 then Commission = Amount * 0.0015
5 Else If Amount > 0.5K
6 then Commission = Amount * 0.002
7 Else Commission = 1
8 Return Commission
An Additional Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 45
1
>50K
3
>5K
7
<=.5K
5
>.5K
8
Return
2
4
6
n n n
y
y
y
Flow Graph Cyclomatic Complexity
V(G) = E – N + 2
= 10 – 8 + 2 = 4
V(G) = R = 4
E: Edges (Links)
N: Nodes
R: Regions
R1
R2
R3
R4
16
An Additional Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 46
TC ID Input Expected
Result
CC-FN-fg-1
0.5K 1.0
CC-FN-fg-2
5K 10
CC-FN-fg-3
50K 75
CC-FN-fg-4 60K 60
Paths:
a) 1-3-5-7-8 (<=0.5K)
b) 1-3-5-6-8 (>0.5K)
c) 1-3-4-8 (>5K)
d) 1-2-8 (>50K)
TCs
Apply Mutation Testing technique to this
example.
M1: Change the first > to <
M2: Change the second > to <
M3: Change the third > to >=
M4: Change all the commission % to 0.002
Etc.
An Additional Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 47
TC ID Input Expected
Result
CC-FN-bv-1 499 1.0000
CC-FN-bv-2 500 1.0000
CC-FN-bv-3 501 1.0020
CC-FN-bv-4 4,999 9.9980
CC-FN-bv-5 5,000 10.0000
CC-FN-bv-6 5.001 7.5015
CC-FN-bv-7 49,999 74.9985
CC-FN-bv-8 50,000 75.0000
CC-FN-bv-9 50,001 50.0010
If we do not know the code, we need to use a Black Box Technique. Best one is Boundary Value:
TCs
0.5K 5K 50K
1M .2% .15% .1%
Compare the two techniques.
If the developer used “>=“ instead of “>”
comparisons in the code, would both
techniques uncover this error?
An Additional Example (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 48
TC ID Input Expected
Result
CC-FN-bv-1 495 1.00
CC-FN-bv-2 500 1.00
CC-FN-bv-3 505 1.01
CC-FN-bv-4 4,995 9.99
CC-FN-bv-5 5,000 10.00
CC-FN-bv-6 5.007 7.51
CC-FN-bv-7 49,993 74.99
CC-FN-bv-8 50,000 75.00
CC-FN-bv-9 50,010 50.01
If we do not know the code, we need to use a Black Box Technique. Best one is Boundary Value:
TCs
0.5K 5K 50K
1M .2% .15% .1%
Compare the two techniques.
If the developer used “>=“ instead of “>”
comparisons in the code, would both
techniques uncover this error?
17
Planning for the Test Phase
Dr. Sadık Eşmelioğlu CENG 557 49
Identify Roles and Responsibilities
Become Familiar with Requirements and Design
Write a Test Plan
Write Test Design Specifications
Write Test Cases
Have Test Plan and Design Spec. Reviewed and Approved
Prepare the Test Environment
Prepare the Test Tools
Prepare the Test Data
Test Documentation
Dr. Sadık Eşmelioğlu CENG 557 50
Project Plan Master
Test Plan Quality Plan
Test Plan 1 Test Plan K . . .
Test
Design 1
Test
Design L
Test Case 1 Test Case M
. . .
. . .
SRS
Design
Test Plan
Dr. Sadık Eşmelioğlu CENG 557 51
1. Test plan identifier: Specifies the unique identifier assigned to the test plan.
2. Introduction: Summarizes the software items and software features to be tested; Defines what this test plan covers and provides references to the documents relevant for testing (overall project plan, quality assurance plan, configuration management plan, applicable standards…); Provides a list of terms and abbreviations along with their definitions.
3. Test items: Identifies the items to be tested, including their version/revision level; provides references to the relevant item documentation (requirements specification, design specification, user’s guide, operations guide, installation guide …); also identifies items which are specifically excluded from testing.
4. Features to be tested: Identifies all software features and their combinations to be tested, identifies the Test Design Specification (TDS) associated with each feature and each combination of features.
5. Features not to be tested: Identify all features and significant combinations of features which will not be tested, and the reasons for this.
6. Approach: Describes the overall approach to testing (the testing activities and techniques applied, the testing of non-functional requirements such as performance and security, the tools used in testing); specifies completion criteria (for example, error frequency or code coverage); identifies significant constraints such as testing-resource availability and strict deadlines; serves for estimating the testing efforts.
18
Test Plan (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 52
7. Item pass/fail criteria: Specifies the criteria to be used to determine whether each test item has passed or failed testing.
8. Suspension criteria and resumption requirements: Specifies the criteria used to suspend all or portion of the testing activity on the test items (at the end of working day, due to hardware failure or other external exception …), specifies the testing activities which must be repeated when testing is resumed.
9. Test deliverables: Identifies the deliverable documents, typically test-design specifications, test-case specifications, test-procedure specifications, test-item transmittal reports, test logs, test-incident reports, description of test-input data and test-output data, description of test tools.
10. Testing tasks: Identifies the set of tasks necessary to prepare and perform testing (description of the main phases in the testing process, design of verification mechanisms, and plan for maintenance of the testing environment …).
11. Environmental needs: Specifies both the necessary and desired properties of the test environment (hardware, communications and systems software, software libraries, test support tools, level of security for the test facilities, drivers and stubs to be implemented, office or laboratory space …).
Test Plan (Cont’d)
Dr. Sadık Eşmelioğlu CENG 557 53
12. Responsibilities: Identifies the groups of persons responsible for managing, designing, preparing, executing, witnessing, checking, and resolving the testing process; identifies the groups responsible for providing the test items (section 3) and the environmental needs (section 11).
13. Staffing and training needs: Specifies the number of testers by skill level, and identifies training options for providing necessary skills.
14. Schedule: Includes test milestones (those defined in the overall project plan as well as those identified as internal ones in the testing process), estimates the time required to do each testing task, identifies the temporal dependencies between testing tasks, and specifies the schedule over calendar time for each task and milestone.
15. Risks and contingencies: Identifies the high-risk assumptions of the test plan (lack of skilled personnel, possible technical problems …), specifies contingency plans for each risk (employment of additional testers, increase of night shift, exclusion of some tests of minor importance …).
16. References: Lists all the reference documents.
17. Approvals: Specifies the persons who must approve this plan.
Test Design Specifications
Dr. Sadık Eşmelioğlu CENG 557 54
1. Subfeatures to be tested
Provide a list of subfeatures to be tested
2. Subfeatures not to be tested
Provide a list of subfeatures not to be tested and why
3. Approach
List refinements or deviation from the Test Plan.
4. Item Pass/Fail Criteria
List refinements or deviation from the Test Plan.
5. Environmental Needs
Provide any special environmental needs, such as data, tools, etc.
6. Test Cases
TC_ID Req Scenario Description
OM.LG.fn.01 1.1.1 Login with a valid id and password
OM.LG.fn.02 1.1.1 Login with a valid id and an invalid password
OM.LG.fn.03 1.1.1 Login with an invalid id and a valid password
19
Test Cases
Dr. Sadık Eşmelioğlu CENG 557 55
TC ID : MO.LG.fn.01
Purpose : Login with a valid id and password
Requirements : 1.1.1
Priority High (High, Medium, Low)
Category Acceptance (Acceptance, Regression, ...)
Est. Time : 5 Minutes (Estimated Time to run the test case)
Dependency : None (List any test cases that need to be run first, if any)
Setup : Bring up Netscape Web Browser (List the needs prior to executing this test case)
Procedure : 1. Go to http://online.metu.edu.tr web site
2. Enter the valid id and password into the corresponding fields
3. Click on the Login button
4. Metu-Online main screen should appear
Cleanup : Logoff (Describe what needs to be cleaned for next set of test cases)
Need for Measurement - Metrics
Dr. Sadık Eşmelioğlu CENG 557 56
Why do we need metrics?
Pre-process Estimation
How much effort is needed?
How many defects do we expect to find?
In-process Monitoring/Managing
How much efforts is being spent?
Are we finding enough defects?
Post-Process Process Improvement
Is the quality of the material sufficient to release to the next phase?
What can be done to improve the next project?
Update benchmark data.
Types of Metrics – Pre-Process
Dr. Sadık Eşmelioğlu CENG 557 57
Helps us estimate the size of testing effort
Size of SUT
# of Requirements
# of Lines of Code
Function Points
Complexity
Coding Effort in Staff-Months
Quality of SUT
Requirements/Code Inspected
# of Faults Found During
Requirements Inspection
Code Inspection
Unit Testing
Integration Testing
Results from Previous Projects
20
Metrics DB
Date Size
(KLOC)
Effort
(Staff-Months)
Defects
2005 120 18 240
2006 160 24 310
2007 200 29 450
2008 250 35 490
2009 300 38 550
2013 200 30 350
TOTAL 1030 144 2040
Dr. Sadık Eşmelioğlu 58 CENG 557
Types of Metrics – In-Process
Dr. Sadık Eşmelioğlu CENG 557 59
Helps us track progress and determine quality of software
Number of Test Cases Scheduled Run
Passed Failed Inconclusive
Blocked
Number of Faults Found Severity 1/2/3/4
Faults on Fixes Fault Density
Time Spent on Test Execution CPU Hours Staff Hours
Testing Effectiveness Requirements Coverage/Traceability Fault Seeding
Mutation Score
Types of Metrics – End-Process
Dr. Sadık Eşmelioğlu CENG 557 60
Helps us determine final quality of product and improve our process
Totals of In-Process Metrics
# and Severity of Faults Outstanding (not Fixed)
Faults Found in Test Cases
Defect Removal Efficiency (DRE)
FT
DRE = -------------- * 100
FT + FC
FT: Faults found during System Test
FC: Faults found by Customer
21
Reporting Faults Found
Dr. Sadık Eşmelioğlu CENG 557 61
Severity 1: The whole system or a major functionality does not work and
there is no workaround.
Severity 2: A major functionality does not work, but there is a
workaround or a minor functionality does not work and there is no
workaround.
Severity 3: A minor functionality does not work, but there is a
workaround.
Severity 4: Spelling and cosmetic type of faults or enhancements that are
not explicitly stated in the requirements
These definitions need to be included in the Project Plan or Master
Test Plan and agreed to by all the parties involved.
Classification of Faults – What and When
Dr. Sadık Eşmelioğlu CENG 557 62
Source
Requirements
Design
Code
Hardware
Testing
Phase Detected
Requirements
Design
Code
Unit Testing
Integration Testing
System Testing
Field
Defects by Phases – Case 1
Dr. Sadık Eşmelioğlu CENG 557 63
Req Des Code Unit
TstInteg
TstSys
TestCust
Code
Req0
5
10
15
20
25
30
35
40
22
Defects by Phases – Case 2
Dr. Sadık Eşmelioğlu CENG 557 64
Code
Req0
10
20
30
40
50
60
Re
q
Co
de
Inte
gT
st
Cu
st
Classification of Faults - Nature
Dr. Sadık Eşmelioğlu CENG 557 65
Logic
Standards
Interface
Functionality
Performance
Data
Maintainability
Testability
Human Factors
Input/Output
Documentation
Duplicate
Not Fault
Tracking the Progress of Testing
Dr. Sadık Eşmelioğlu CENG 557 66
0
100
200
300
400
500
600
700
800
900
1000
0 1 2 3 4 5 6 7 8 9 10
Planned
Executed
Passed
23
Tracking the Quality– Faults Found by Week
Dr. Sadık Eşmelioğlu CENG 557 67
0
20
40
60
80
100
120
0 1 2 3 4 5 6 7 8 9 10
Sev1
Sev2
Sev3
Sev4
Total
Tracking the Quality–Faults Found by CPU Hrs
Dr. Sadık Eşmelioğlu CENG 557 68
0
20
40
60
80
100
120
0 1 2 3 4 5
Sev1
Sev2
Sev3
Sev4
Total
Tracking the Quality– Fault Density (Faults/KLOC)
Dr. Sadık Eşmelioğlu CENG 557 69
0
0,5
1
1,5
2
2,5
1 2 3 4 5 6 7 8 9 10
Benchmark
Actual
24
Tracking the Quality – Faults Outstanding
Dr. Sadık Eşmelioğlu CENG 557 70
0
5
10
15
20
25
30
35
40
45
1 2 3 4 5 6 7 8 9 10
Sev1
Sev2
Sev3
Sev4
Total
Deciding When to Stop – Exit Criteria
Dr. Sadık Eşmelioğlu CENG 557 71
100% of the test cases are executed
95% of the test cases passed
All High Priority test cases passed
Sev 1 Faults = 0
Sev 2 Faults ≤ 2
Sev 3 Faults ≤ 5
Sev 4 Faults ≤ 10
These criteria need to be included in the Project Plan or Master Test Plan and agreed to by all the
parties involved.
Questions
Dr. Sadık Eşmelioğlu CENG 557 72
1. When can we stop testing?
2. How many bugs can we expect?
3. Which test technique is more effective?
4. Are we testing hard or are we testing smart?
5. Do we have a strong program or a weak test suite?
25
Lessons Learned – Postmortem
Dr. Sadık Eşmelioğlu CENG 557 73
Conducted with the SQA Engineers
Can be for only Testing or Project-wide
Present all the data
Brainstorm strengths and weaknesses
Praise the strengths
Identify areas of improvement
Use a voting technique to identify the importance of each area
Use a Pareto diagram to prioritize the results
Select 3 top priority areas
Establish a QIT for each area
Test Management
Dr. Sadık Eşmelioğlu CENG 557 74
Test Plan
Test Design Specifications
Test Cases
All the fields in the template and more
Traceability to Requirements
Schedule
When/What
Results
Run/Pass/Fail
Defects (Traceability to Test Cases)
History
Test Automation
Dr. Sadık Eşmelioğlu CENG 557 75
Why Can be run multiple times
Can be run 7x24
Faster execution
When/What Stable feature set
Frequent regression testing
Automation Degree of Difficulty (Cost/Benefit)
How Scripts (Semi/Full Automation)
Tools (TSL, Capture/Replay)
26
Test Automation – Road Blocks
Dr. Sadık Eşmelioğlu CENG 557 76
Management
Wants automation, but not willing to spend the money
(Tool, Training, Learning Curve)
Time
Effort is similar to Code development
Changes in Requirements need to be reflected
Constant Maintenance
Technical
Synchronization
Windows pop up in different locations
THE END
Dr. Sadık Eşmelioğlu CENG 557 77
QUESTIONS
Top Related