David Capocci, CQA, CSTE 1 Test Requirements: The Basis of Testing David Capocci, CQA, CSTE Sr. QA...
-
Upload
quintin-clemmens -
Category
Documents
-
view
225 -
download
2
Transcript of David Capocci, CQA, CSTE 1 Test Requirements: The Basis of Testing David Capocci, CQA, CSTE Sr. QA...
David Capocci, CQA, CSTE
1
Test Requirements:The Basis of Testing
David Capocci, CQA, CSTE
Sr. QA Systems Analyst
SAFECO Corporation
David Capocci, CQA, CSTE
2
Agenda• Defining Test Requirements (TR)
– What, Why, Where
• Fitting TR’s into the testing picture– What’s within our testing process– Generating TR’s
• Organizing & Decomposing TR’s
• Test Requirements Hierarchy Samples
• Setting the stage for measuring test coverage
David Capocci, CQA, CSTE
3
Background• “Test Requirements Hierarchy” is a term coined
from Rational’s SQA Team Test software.• The principle of identifying, organizing, and
measuring test requirements is universal to many test processes and methodologies
• Much of this in-service is distilled from:• Rational methodologies (we are an SQA Team Test
house after all)
• QAI Workbench model
• Seminar topics from– 19th annual International Software Testing Conference
– Star ‘98 West Conference
• Ed Kit’s “Software Testing in the Real World”
David Capocci, CQA, CSTE
4
Test Requirements
• What exactly is a Test Requirement?
• Why identify Test Requirements?
• Where does a Test Requirement come from?
Defining TR’s: What, Why, Where
David Capocci, CQA, CSTE
5
What exactly is a Test Requirement?
Identifies the “WHAT” of testingWhat needs to be tested, ANDWhat are you going to validate about it
Includes both normal and error conditionsCovers business rules, functionality, non-
functional standardsDo NOT have case specific data values
assigned to them yet (data appears in test cases, the “How” of testing) examples…
Defining TR’s: What, Why, Where
David Capocci, CQA, CSTE
6
Example 1: Testing the inserting of a record to a table
“Validate that you can insert an entry”
“Validate that insertion fails if entry already present”
“Validate that insertion fails if table already full”
“Validate that you can insert an entry to an empty table (initial)”
• These are test requirements NOT tests because they do not describe the data element being inserted
• The data is irrelevant at this level, it will appear in the test cases used to cover these test requirements
• “Validate you can insert ‘John Doe’” is a test case not a test requirement
Test Requirements Identified (among others):
Defining TR’s: What, Why, Where
David Capocci, CQA, CSTE
7
Why identify Test Requirements? It’s what QC “owns” in our workbench:
Requirements-based or Function-based testing It’s the basis for establishing the completion of testingHelps determine the scale of the testing effortGoverns the types of resources you will needServes to identify automation strategies you can useBecomes a roadmap for your testing effortCan be a tool for leverage and dialog with developers
and business analystsDev Team can sign off on them (Verification!)
Defining TR’s: What, Why, Where
David Capocci, CQA, CSTE
8
Where does a TR come from?Traditional: Business Requirements,
functionality, internal logic…Marketing specs, Functional specs, Technical specs
Reality:“Interview Analysis”, Non-Functional Checklists
(standards & compliance), Use Cases (from business scenarios and users), Discovery during testing, any other deliverables from previous workbenches (diagrams, modeling, flowcharts, etc.)
Defining TR’s: What, Why, Where
David Capocci, CQA, CSTE
9
How do Test Requirements relate to the Test Plan?
Traditionally, the Test Plan has represented “what” was going to be tested, even including test cases.
Paradigm is shifting: Test Plan should relate what your testing process (and deliverables) will be for a particular project.
A Test Plan should build confidence in the testing process for a project, making approaches clear.
A Test Plan can include the Test RequirementsHowever, if the Test Requirements are too lengthy,
they should become their own document: A Test Requirements Hierarchy Defining TR’s: What, Why, Where
David Capocci, CQA, CSTE
10
Agenda
• Defining TR’s: What, Why, Where
• Fitting TR’s into the testing picture– What’s within our testing process– Generating TR’s
• Organizing & Decomposing TR’s
• Test Requirements Hierarchy Samples
• Setting the stage for measuring test coverage
David Capocci, CQA, CSTE
11
Drilling down: Where test requirements fit into the picture
BusinessRequirement
TestRequirement
TestScenarios/Cases
TestProcedure/Script
Fitting TR’s into the testing picture
Generates
1
M
Generates
1
M
Executes/Runs
1
M
David Capocci, CQA, CSTE
12
Drilling Down
BusinessRequirement
TestRequirement
TestScenarios/Cases
TestProcedure/Script
Fitting TR’s into the testing picture
First, Let’s look
at this relationship:
What’s within our
testing process
Then we’ll look
at this relationship:
Gernerating TR’s
from what feeds into
our testing process
David Capocci, CQA, CSTE
13
ATM Example: Practice Writing Test Requirements
Business Requirements:
- “ATM must do withdrawals”
- “Withdrawals are between $20-$300”
- “Withdrawals are in $20 multiples”
Group Exercise!
1. Limit the scope to these 3 requirements.
2. What will you validate (test for)?
3. Are there any implied requirements that may not be written out?
What’s within our testing process
David Capocci, CQA, CSTE
14
Example 2: Testing Withdrawals on an ATM
“Validate that a withdrawal option is available”
"Validate that a withdrawal of a multiple of $20, between $20-$300 can be done"
"Validate that <$20 is not allowed"
"Validate that >$300 is not allowed"
"Validate that $20 multiples >$300 is not allowed"
"Validate that non-$20 multiples between $20-$300 not allowed"
"Validate strange numeric amounts/combinations not allowed (all zero's, all 9's, 20.0000)"
“Validate that the withdrawal received is equal to the amount requested”
"Validate that a valid withdrawal amount must be below the account balance”• These are test requirements NOT tests because they do not describe the data element being
used (like $20, $40, $60, $1)• The data is irrelevant at this level, it will appear in the test cases used to cover these test
requirements
Test Requirements Identified (among others):
What’s within our testing process
David Capocci, CQA, CSTE
15
Test Scenarios/Cases for - “Validate that a withdrawal of a multiple of $20,
between $20-$300 can be done”
Case # P/F $ entered ExpectedResults
ActualResults
WD01 Pass 20 $20 withdrawnWD02 Pass 40 $40 withdrawnWD03 Pass 60 $60 withdrawnWD04 Pass 80 $80 withdrawn
WD05 Pass 100 $100 withdrawn: : : :
WD13 Pass 260 $260 withdrawnWD14 Pass 280 $280 withdrawnWD15 Pass 300 $300 withdrawn
What’s within our testing process
David Capocci, CQA, CSTE
16
Test Procedure & Script for previous example
Step 1: Insert Card
Step 2: Enter PIN
Step 3: Select Withdraw option
Step 4: Enter dollar amount
Step 5: Validate amount received
Do until EOF ‘until end of data file
Input data record
Senddata CARDINFO to “Cardfield”
Senddata “Enter”
Senddata PIN to “PINFfield”
Senddata “Enter”
Senddata “W” to “SelectionField”
Senddata AMOUNT to “DollarField”
Senddata “Enter”
If ErrorMsg > 0 then print ErrorMsg
Print “DollarAMTgiven”
Loop
Procedure: Script: (in pseudo-code )
What’s within our testing process
ThinkThinkManual !
Manual !ThinkThink
Automated
Automated
!!
David Capocci, CQA, CSTE
17
Agenda
• Defining TR’s: What, Why, Where
• Fitting TR’s into the testing picture– What’s within our testing process– Generating TR’s
• Organizing & Decomposing TR’s
• Test Requirements Hierarchy Samples
• Setting the stage for measuring test coverage
David Capocci, CQA, CSTE
18
The Workbench Concept
DODO Check
Standards
Tools
Rework
Entrance
CriteriaExit Criteria
Product Input
Product Output
Generating TR’s
Our workbench is called “Generating Test Requirements”
David Capocci, CQA, CSTE
19
Entrance Criteria for Business Requirements to generate Test Requirements
Visible ?Clear? (unambiguous)Complete?Consistent? (conflicting requirements must be prioritized)Reasonable? (achievable)Measurable? (quantifiable)Modifiable? (will it change or is it stable?)Traceable? (the source is known)Dependent requirements identified?Testable? (given current environment, resources, skills)
Generating TR’s
David Capocci, CQA, CSTE
20
Exit Criteria for Test RequirementsCan another tester create test cases/scenarios from these?Does a Test Requirement specify what is being tested and what
about it we are validating? (Clear?) Are the Test Requirements…
- Complete?
- Consistent? (conflicting requirements must be prioritized)
- Reasonable? (achievable)
- Measurable? (quantifiable for measuring test coverage)
- Modifiable? (will it change or is it stable?)
- Traceable? (the source is known)
- Testable? (given current environment, resources, skills)
Do the test requirements cover the complete scope of the project?Are all the test requirements verified and signed off by the
Development Team?Generating TR’s
David Capocci, CQA, CSTE
21
When creating Test Requirements (“Do”)...• Use “action” verbs & words
- “Validate that…”- “Check for…”- “Test that…”
• Trace them back to the source• Remember that different applications arrange in
different ways- Think of MF, batch, C/S, web, e-commerce, GUI, etc.- Use “testing considerations” checklists that generally cover what kinds of
things should be considered when testing your specific situation
• Make your Test Requirements document a “living document”
Generating TR’s
David Capocci, CQA, CSTE
22
Also...• Maintain a level of balance between too much &
too little- Too High level: won’t be useful, vague, can’t generate test cases from it.
- Too low level: Over-process, over documentation, no productivity
- General rule: 5-7 levels deep in an outline format
• Organize them!- Outline/Hierarchy format recommended
- Look at how the BA breaks down the project into areas
- Look at how the PA breaks down the project into areas
- Organize by functional areas
- Organize by “types” of testing (Function vs. System vs. Non-Functional)
Generating TR’s
David Capocci, CQA, CSTE
23
Agenda
• Defining TR’s: What, Why, Where• Fitting TR’s into the testing picture
– What’s within our testing process
– Generating TR’s
• Organizing & Decomposing TR’s
• Test Requirements Hierarchy Samples• Setting the stage for measuring test coverage
David Capocci, CQA, CSTE
24
Distinguishing the types of testing….
I. Function-Based Tests
II. User Interface Tests
III. Security Tests
IV. Installation Tests
V. Configuration Tests
VI. Performance Tests (Response)
VII. Load Tests (simultaneous users, lots of small transactions)
VIII. Volume Tests (Big transactions)
IX. Stress Tests (breaking point: memory, resources)
X. Resource Usage Tests
XI. Documentation Tests
XII. Compatibility Tests
XIII. Recovery Tests
XIV. Serviceability Tests
and others…
*III - XIV are all “Systems-based tests”
Organizing TR’s
David Capocci, CQA, CSTE
25
Organizing by Functional areas….
• Most testers perform function-based, or requirements-based tests
- Tests on functionality that the system will provide
- Usually scenario/case based- includes normal and alternate (invalid) cases
Organizing TR’s
David Capocci, CQA, CSTE
26
Organizing by Functional areas….
• Most testers also perform User Interface Style Tests
- These are generally separate from the functionality that the software will provide
- Usually encompass the architectural standards & compliance (like Windows Design Standards)
- Also includes tests of navigation, menus, admin functions (like printing, saving)
Organizing TR’s
David Capocci, CQA, CSTE
27
Agenda
• Defining TR’s: What, Why, Where• Fitting TR’s into the testing picture
– What’s within our testing process
– Generating TR’s
• Organizing & Decomposing TR’s
• Test Requirements Hierarchy Samples• Setting the stage for measuring test coverage
David Capocci, CQA, CSTE
28
Remember this?…Drilling down
BusinessR equirem ent
TestRequirement
TestScenarios/Cases
TestProcedure/Script
Fitting TR’s into the testing picture
David Capocci, CQA, CSTE
29
Decomposing: Drilling down within a Test Requirement
BusinessRequirement
TestRequirement
TestScenarios/Cases
TestProcedure/Script
Fitting TR’s into the testing picture
Keep the
function-based
perspective
in mind!
Business Function
Tasks within theFunction
Data Entry Typesfor transactions
Transactions toperform a task
Field Validation
David Capocci, CQA, CSTE
30
Test Requirement Decomposition
Business Function
Tasks with in theFunction
D ata Entry Typesfor transactions
Transactions toperform a task
F ie ld Validation
The "what" that the software is in tended todo, like "Rent a Car to a custom er"
The tasks involved: 1) query car rates,2) check custom er policy coverages, ....
The database transactions necessaryto perform the task.
The edits on the data included w ith inthe d ifferent types
The types of data entry thetransaction can process: Add,
Change, Delete, Inquire
Decomposing TR’s
David Capocci, CQA, CSTE
31
Test Requirement Decomposition
BusinessFunction
Tasks within theFunction
Data Entry Typesfor transactions
Transactions toperform a task
Field Validation
High level Functional Areas:
usually from
“Functional Spec” type
docs, or BA work
Lower level
Functional Areas:
usually from
“Technical Spec” type
docs regarding
internal logic,
or PA workDecomposing TR’s
David Capocci, CQA, CSTE
32
Test Requirement Decomposition
• You can start on the high level functional areas early into the project & build in lower level areas as they become available
• Any level can contain a test requirement which can also be made up of (or broken down into) lower level test requirements
Decomposing TR’s
David Capocci, CQA, CSTE
33
Business Function Level• Try to identify “groups” of functions or
functions connected by similar themesfile management functions, printing functions, help
functions, car rental functions, reservation functions, ticket purchase functions, claim reporting functions
• Be sure all areas of the system are covered. If something is left out or doesn’t fit into a group, it becomes its own group.
• It may be easier to identify functional areas by “window” instead of by function.
Decomposing TR’sFunction Task TransactionTrans Data
Type
Field
Validation
David Capocci, CQA, CSTE
34
Business Function Level
• At this level, the idea is seeing the connections, integration, and interactions of the system’s functionality.
• May not necessarily be identifying a test requirement at this level as much as just identifying the functional area.
Decomposing TR’sFunction Task TransactionTrans Data
Type
Field
Validation
David Capocci, CQA, CSTE
35
Task Level
• Break down each Function area into the tasks within the function
• For complex tasks, it is possible to break them down further into sub-tasks
• Some Business Functions can not decompose into further tasks (example: Check Writing function)
Decomposing TR’sFunctio
nTask Transaction
Trans Data
Type
Field
Validation
David Capocci, CQA, CSTE
36
Transaction Level• From this level down, we start to address
the internal things that occur to make the functions and tasks happen
• Identify any logical transactions that ties the task to the database or any other transactions necessary to perform the task.
• Identify any data processing, calculation, data formatting type transactions
Note: A screen or window may cause the execution of several different logical transactions
Decomposing TR’sFunctio
n Task Transaction Trans Data
Type
Field
Validation
David Capocci, CQA, CSTE
37
Transaction Data Type Level
• Identify which of the four types the transaction can become: Add, Change, Delete, Inquire
• It is entirely possible that a transaction can be multiple types.
• If a transaction is only one type, you can wrap this level up into the higher level.
Decomposing TR’sFunctio
n Task Transaction
Trans Data
TypeField
Validation
David Capocci, CQA, CSTE
38
Field Validation Level• Most testers like to jump directly to this
level. It’s the most obvious at times.
• Field validation covers all edits & cross edits on fields and data.
• Be careful of the detail you document at this level. Remember the separation between test requirement and test case.
Decomposing TR’sFunctio
n Task TransactionTrans Data
Type
Field
Validation
David Capocci, CQA, CSTE
39
Field Validation Level• Not all test requirements (especially at this
level) fit well to be documented in this manner.- Example: Testing all the stated properties of windows
objects
- Use “Validate that the properties of all the objects in this window match the properties listed on the Object Properties Reference Table in Appendix B upon initialization of the window”
- Don’t list each property check as a separate test requirement if it can be summarize under one test requirement
- This is a judgement call YOU make for your given project.
Decomposing TR’sFunctio
n Task TransactionTrans Data
Type
Field
Validation
David Capocci, CQA, CSTE
40
Example 3: Rental Car Application
1. Validate that a Rental can occur.
1.1 Check Customer policy coverage
1.2 Query Car availability
1.3 Query Car rates
1.4 Open a Rental ticket
1.4.1 Validate that a customer record can be entered
1.4.2 Validate that credit card approval is obtained
1.4.3 Validate that status on the car record is changed
from “waiting” to “rented”
2. Billing Function
3. Reservation Function
Let’s look at
the lower levels
for this one
Decomposing TR’s
Then we’ll try it
on this one
David Capocci, CQA, CSTE
41
Example 3: Rental Car Application
1. Validate that a Rental can occur.
1.4 Open a Rental ticket
1.4.1 Validate that a customer record can be entered
1.4.1.1 Validate that a new customer can be added to the customer table
1.4.1.1.1 Validate that the first name is all alpha
1.4.1.1.2 Validate that the age is > 21.
1.4.1.1.3 Validate that the phone number is numeric
1.4.1.1.4 Validate area code is an existing area code number.
1.4.1.2 Validate changing an existing customerDecomposing TR’s
David Capocci, CQA, CSTE
42
Example 3: Rental Car Application
1. Validate that a Rental can occur.
1.4 Open a Rental ticket
1.4.2 Validate that credit card approval is obtained
…fill in the lower level test requirements!
First, Identify any sub-areas (further tasks, or even
separate transactions within this)
Then, Identify the lowest level field validation test
requirements (think about what is typically
involved with credit card authorizations)
Decomposing TR’s
David Capocci, CQA, CSTE
43
What did you come up with?1. Validate that a Rental can occur.
1.4 Open a Rental ticket
1.4.2 Validate that credit card approval is obtained
_________________________________________
_________________________________________
_________________________________________
_________________________________________
_________________________________________
_________________________________________
_________________________________________
_________________________________________
Decomposing TR’s
David Capocci, CQA, CSTE
44
Possible Test Requirements...1. Validate that a Rental can occur.
1.4 Open a Rental ticket
1.4.2 Validate that credit card approval is obtained
1.4.2.1 Validate the expiration date is a valid future date
1.4.2.2 Validate the expiration date is not within 1 month of
expiring.
1.4.2.3 Validate that the CC# is 12 digits
1.4.2.4 Validate that the $ amount is <= credit balance available
1.4.2.5 Validate that an authorization # is received.
Decomposing TR’s
Function
Task Transaction
David Capocci, CQA, CSTE
45
Agenda
• Defining TR’s: What, Why, Where
• Fitting TR’s into the testing picture– What’s within our testing process– Generating TR’s
• Organizing & Decomposing TR’s
• Test Requirements Hierarchy Samples
• Setting the stage for measuring test coverage
David Capocci, CQA, CSTE
46
TRH Samples
Let’s look at a few examples of how test requirements can be organized:
- Sample 1: Excerpt from a Personal Finance app
- Sample 2: actually organizing Function requirements, but good representative of the concept
- Sample 3: QBS, from Rational’s sample project (adds types of testing into the hierarchy)
- Sample 4: another view of the Personal Finance app
- Sample 5: A mainframe batch system
David Capocci, CQA, CSTE
47
Test Coverage Measures
• Test Requirements are the “what” of testing & are the basis for establishing the completion of testing
• TR’s give us the point of measurement for test coverage
• Each TR should receive a Priority, Risk, and Weight
• Each TR should be tracked for Verification () and Validation (%)
Test Coverage Measures
David Capocci, CQA, CSTE
48
Summary & Recap• Defined: What, Why, Where
• Described: How TR’s “fit” into the big picture
• Decomposed: Organizing TRs & Generate TR’s into a measurable format
• Demonstrated: Sampling some TRH
• Determined: The Measurement Connection