55th세미나 발표자료
Transcript of 55th세미나 발표자료
Stuart Reid PhD, FBCS ([email protected])
© 2015 Stuart Reid
Making Your Reviews More Effective
Using the New ISO Standard (NIPA – 2nd July 2015)
• Context - ISO/IEC 20246 & the ISO 29119 standards
• The generic review process
• Individual review techniques
• Review types vs review attributes
• Current status
• Question?
Scope
Putting ISO 20246 in Context
ISO 29119/33063/20246 – Structure
BS 7925-1
BS 7925-2 IEEE 829
Concepts & Vocabulary
Part 1
Testing Techniques
Part 4
Documentation
Part 3 Part 2
Processes
Keyword-Driven Testing
Part 5
Process Assessment
ISO/IEC 33063
Reviews
ISO/IEC 20246 IEEE 1028
ISO 29119 Part 2: Testing Processes
TEST MANAGEMENT PROCESSES
ORGANIZATIONAL TEST PROCESS
DYNAMIC TEST PROCESSES
TEST MANAGEMENT PROCESSES
STATIC TEST
PROCESSES
ORGANIZATIONAL TEST PROCESS
DYNAMIC TEST
PROCESSES
Testing Processes – the intention
REVIEWS STATIC
ANALYSIS
V&V Taxonomy – ISO/DoD Washington, 2010
Verification & Validation
Formal Methods
Model Checking
Proof of Correctness
Specification-Based
Structure-Based
Experience-Based
V&V Analysis Simulation
Evaluation Quality Metrics
Model Verification
Static Analysis
Reviews
Static Testing
Dynamic Testing
Testing
ISO/IEC/IEEE 29119
TEST MANAGEMENT
PROCESSES
ORGANIZATIONAL TEST
PROCESS
DYNAMIC TEST
PROCESSES
Testing Processes – the end result
ISO/IEC/IEEE 20246
REVIEW PROCESS
ISO/IEC/IEEE 29119
TEST
MANAGEMENT
PROCESSES
ORGANIZATIONAL
TEST PROCESS
DYNAMIC TEST
PROCESSES
Testing Processes – the next step?
ISO/IEC/IEEE 20246
REVIEW PROCESS
ISO/IEC/IEEE ?????
STATIC ANALYSIS
ORGANIZATIONAL TEST PROCESS
TEST MANAGEMENT PROCESSES
TEST
PLANNING
TEST
MONITORING
& CONTROL
TEST
COMPLETION
ORGANIZATIONAL
TEST
DOCUMENTATION
FEEDBACK ON
ORGANIZATIONAL TEST
DOCUMENTATION
TEST PLAN UPDATES
TEST
PLAN
TEST
COMPLETION
REPORT
DYNAMIC TEST
PROCESSES
TEST
MANAGEMENT
PROCESSES
TEST PLAN,
TEST COMPLETION
REPORT,
TEST MEASURES
TEST
MEASURES
TEST PLAN,
CONTROL
DIRECTIVES
TEST PLAN,
CONTROL
DIRECTIVES
Overall Test Management Processes
WORK PRODUCT
REVIEW
PROCESS
REVIEW
MEASURES
TEST PLAN,
CONTROL
DIRECTIVES
Organise
Test Plan
Development
Identify &
Estimate Risks
Design Test
Strategy
Determine
Staffing and
Scheduling
Document
Test Plan
Schedule, Staffing
Profile
Test
Strategy
Estimated
Risks
Scope
Identify Risk
Mitigation
Approaches
Gain
Consensus on
Test Plan
Approved
Test Plan
Draft
Test Plan
Test
Plan Publish
Test Plan
Understand
Context
Treatment
Approaches
ISO 29119 Part 2 - Test Planning Process
Understand
Context
MUST INCLUDE
REVIEWS
ISO/IEC 20246 Work Product
Reviews
• Anything can be reviewed – in parts or the whole thing
• For example:
– Contracts – Requirements Specification – Design Specification – Code – Test Specification – User Manual – User Procedures – Project Plan – Test Plan – Development Plan
Why ‘Work Product’ Reviews?
• What do you think?
• Find defects
• Measure quality
• Educate reviewers
• Gain consensus
• Generate new ideas
• Motivate authors to improve their practices
Software Reviews – Main Purpose
Generic Review Process
Work Product Reviews - Generic Process
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation & Analysis
Fixing &
Reporting
• Identify review scope – what is being reviewed – the purpose of the review (there may be more than one) – relevant supporting information, such as standards – the timeframes for the review
• Identify review characteristics – review activities – individual review techniques – checklists
• Distribute review materials (as early as possible) – product to be reviewed – baseline specifications – checklists
Planning & Preparation
Planning &
Preparation
Overview
Meeting
Individual Review
Issue
Collation & Analysis
Fixing &
Reporting
• Choose appropriate reviewers
– agreed with Project Manager
– assign specific roles
• Agree with each reviewer
– availability (and a substitute, just in case)
– the review role and focus
Planning & Preparation – Reviewers
Planning &
Preparation
Overview
Meeting
Individual Review
Issue
Collation & Analysis
Fixing &
Reporting
• Optional
• Review Leader
– presents the scope of the review to the review participants
– presents related documents to reviewers
– explains the role, responsibilities and focus of each review participant
– details the timeline for the review
• Reviewers (and author)
– formally commit to the review
Overview Meeting
Planning &
Preparation
Overview
Meeting
Individual Review
Issue
Collation & Analysis
Fixing &
Reporting
• Each reviewer identifies issues with the work product
• Identified issues shall be communicated to the review leader / author
Individual Review
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation & Analysis
Fixing &
Reporting
• The author collates the Issues identified by the reviewers • Collated issues are analysed and a decision made on what
to do with each issue • Issue Review Status
– ‘rejected’ – ‘issue to be noted but no action’ – ‘issue to be addressed’ – ‘issue updated due to analysis’ – etc.
• Issues are then assigned based on their review status. – to authors – to other responsible individuals (e.g. to update earlier
specifications, organization standards)
Issue Collation & Analysis – 1 (of 2)
Planning &
Preparation
Overview
Meeting
Individual Review
Issue
Collation &
Analysis
Fixing &
Reporting
• Finally the review decision is made on the reviewed product:
– used as is
– updated based on the identified issues and used
– reworked and re-reviewed
– discarded
• The BIG Question – is this analysis and decision-making done by the author or done by a group of reviewers as part of a Review Meeting???
Issue Collation & Analysis – 2 (of 2)
Planning &
Preparation
Overview
Meeting
Individual Review
Issue
Collation &
Analysis
Fixing &
Reporting
• The author – addresses the outstanding issues – organizes for others to fix their outstanding issues in related
documents • in ‘earlier’ specifications • in organizational standards
– checks that all issues have been addressed – either:
• publishes the updated work product • submits the updated work product for re-review
• The reviewers – sign off on the final version (confirming that issues addressed)
• A Review Report is generated
Fixing & Reporting
Planning &
Preparation
Overview
Meeting
Individual Review
Issue
Collation &
Analysis
Fixing &
Reporting
Review Timelines
Planning &
Preparation
Overview
Meeting
Individual Review
Issue
Collation &
Analysis
Fixing &
Reporting
3 days
5-10 days
3 days
5-10 days
2 days
They vary!!
But need to be set and agreed.
Generic Review - Roles
Reviewer
Author
Planning &
Preparation
Overview
Meeting
Individual Review
Issue
Collation &
Analysis
Fixing &
Reporting
Review Leader
Reader Recorder/Scribe
Moderator
QA
SME
Customer
Management
Technical Lead
Individual Reviews
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation & Analysis
Fixing &
Reporting
• Ad Hoc Reviewing
• Checklist-Based Reviewing
• Scenario-Based Reviewing
• Perspective-Based Reading (PBR)
Individual Review Techniques
• This is the most common approach (completely unstructured)
• Each reviewer is expected to find as many defects as possible of any type
• Little or no guidance on how to review is provided
• This approach is highly-dependent on reviewer skills
• Often the same issues are identified by different reviewers
Ad Hoc Individual Review
“60% of reviewers don’t prepare at all”
Individual Reviews -
Checklist-based Reviewing
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation & Analysis
Fixing &
Reporting
• A systematic approach to identifying defects
• Typically review checklists take the form of a set of questions based on potential defects (and risk)
• Assign different reviewers to different checklists to – get wider coverage
– prevent the duplication inherent in the ad hoc approach
• There is a danger that some reviewers limit themselves to only considering the checklist entries
• Will normally take longer than using an ad hoc approach – often multiple passes are needed to cover all questions
(especially if checklist is long)
– Requires a systematic approach
Checklist-based Review
“50% OF REVIEWERS USE CHECKLISTS”
• Checklist defects are derived from experience – within the project – within the organization – across the industry as a whole
• Checklists should be specific to – the product under review (be wary of generic questions) – the methodology used to develop the work product (e.g. there may
be different checklist questions for requirements in the form of plain text to those in the form of use cases or user stories)
– the application domain of the work product (e. g. a checklist for a banking work product may be based on banking regulations while a checklist for an avionics work product would be based on avionics standards)
• Checklists should be based on risk – to ensure that the most important (and the most frequently-
occurring) risks are reviewed in more depth
Creating Checklists
• Many checklists are too long and never change
• The ideal checklist should be constrained to about 10 entries and regularly updated
– as entries become stale and find fewer issues (hopefully because the authors have learned and improved)
– new entries should reflect issues missed in the recent past
Managing Checklists
Individual Reviews –
Scenario-based Reviewing
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation & Analysis
Fixing &
Reporting
• Used where multiple scenarios can be identified for different users
• Reviewers perform ‘dry runs’ on the work product to check functionality and handling of error conditions – following scenarios rather than reading the document from top to
bottom
• Assign different reviewers to different scenarios to: – get wider coverage – prevent the duplication inherent in the ad hoc approach
• There is a danger that some reviewers limit themselves to only considering the provided scenarios – and will miss defects of omission, where required functionality is not
included in the work product under review
• Research indicates that scenario-based reviews – can be more cost effective than checklist-based reviews – but they do take longer
Scenario-based Reviewing
• Where requirements, designs (or tests) are documented in a scenario-type format (e.g. use cases) then the existing specifications are a good starting point – the review can then use the modelled scenarios to drive the review
• However, scenarios typically also need to be created (we need multiple scenarios to cover everything): – overview scenario – most common scenarios – alternative scenarios – error condition scenarios – growth scenarios (e.g. for an architecture review) – input handling scenarios – output generating scenario – enquiry scenarios (getting specific results)
• Scenarios should be based on risk – to ensure that the most important (and the most frequently-used)
scenarios are reviewed in more depth
Identifying Scenarios
Individual Reviews -
Perspective-based Reading
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation & Analysis
Fixing &
Reporting
Perspectives and Roles
PRODUCT UNDER REVIEW
View 2
View 5
View 7
Perspective-based Reading (PBR)
Identify Stakeholders
Create/Identify PBR Scenarios
Perform Review Perform Review Perform Review Perform Review • Understand Stakeholder Perspective • Attempt to Create a Work Product • Use Checklist on the Work Product
Describes the stakeholder role
the reviewer is taking for this
review – and their interest in
the work product under review
Describes the high
level work product
that the stakeholder
would be expected to
develop
A checklist of questions
specific to the high-level
work product developed
Choosing your Review
• Milestone Reviews
• Inspections
• Technical Reviews
• Peer Reviews
• Walkthroughs
• Informal Reviews
• Author Check
• Buddy Check
• Pair Review
• Peer Deskcheck
Work Product Reviews – Types or Bespoke
Planning &
Preparation
Overview
Meeting
Individual
Review
Issue
Collation &
Analysis
Fixing &
Reporting
Chosen
Attributes
OR
• Purpose • Roles • Individual review techniques • Optional activities • Number of reviewers • Planned number of reviews • Work product type • Work product format • Formal reporting • Training required • Review improvement • Entry and exit criteria • Geographic distribution of reviewers
Work Product Reviews - Attributes
• Factors
– purpose
– product type
– product format
– risks
– reviewer availability
– reviewer skills (and availability of training)
– life cycle model
– budget (time and cost)
Choosing your Review
• Foreword • Introduction • 1 Scope • 2 Conformance • 3 Normative References • 4 Terms and Definitions • 5 Work Product Reviews - Introduction • 6 Software Review Process • 7 Review Techniques • Annex A (informative) Review Documentation • Annex B (informative) Review Roles • Annex C (informative) Review Attributes • Annex D (informative) Review Types (and attribute mapping) • Annex E (informative) Mapping to IEEE 1028 • Annex F (informative) Reviews - Life Cycle Mapping • Annex G (informative) Review Measurement & Improvement • Annex H (informative) Tool Support • Annex I (informative) Bibliography
ISO/IEC 20246 - Work Product Reviews - Contents
ISO/IEC 20246 Development
Working Draft (WD)
Committee Draft (CD)
Draft International Standard (DIS)
Final Draft International Standard (FDIS)
Final International Standard (FIS)
May
2014
May
2015
May
2016
WD1
DIS
FDIS
WD2
DIS2
ISO 29119/30363/20246 – Current Status
Testing Concepts & Vocabulary
29119 Part 1
Testing Techniques
29119 Part 4
Test Documentation
29119 Part 3 29119 Part 2
Keyword-Driven Testing
29119 Part 5
Process Assessment
ISO/IEC 33063
AUG ‘13
AUG ‘13 AUG ‘13 @ FDIS
@ DIS2 @ FDIS
Work Product Reviews
ISO/IEC 20246
DIS
Test Processes
• Context - ISO/IEC 20246 & the ISO 29119 standards
• The generic review process
• Individual review techniques
• Review types vs review attributes
• Current status
• Question?
Summary
Thank you for listening
Any Questions?
• Join ISO Working Group 26
– representing your national standards body
– 6 day meetings, every 6 months
– contribute between meetings
• Join a WG26 mirror group
– for your national standards body
• Contribute materials
• Review drafts
• Trial the standards on real projects
Do you want to be involved?
– if you have any questions on the standards
– if you are interested in trialling the standard on a project, reviewing drafts or writing examples
• http://softwaretestingstandard.org/
– WG26 website
• http://www.jtc1-sc7.org/
– access to official documents released by WG 26
Finally…