Training Review Technique

8/13/2019 Training Review Technique

1/30

Training

Review Technique

for Object Reviews


2/30

2 / Hubert Gast / 14.Feb.2008 © Continental AG

Agenda

Benefits

Review Types

Review Planning, Review Execution

Hints for effective Reviews

Guidelines and best practices

Focus of this Training are Object Reviews (reviewing Documents

according to P730103, not Milestone Reviews acc. to P730128) !


3/30


Benefits

Earlier detection of defects

less risk in late project phases

cheaper defect removal

much lower maintenance costs

better quality of documents, esp. basic documents like System Design, Functional Spec

comparable quality level of all reviewed documents in projectguaranteed appliance of project wide Guidelines & Rules

Know-how sharing within project

Disadvantage: Time spent for Review (Admin, Preparation, Meeting) is lost if Review

results does not lead to Document update


4/30


Benefit of Reviews: Development Costs

0

1

2

3

4

5

6

Analysis Design Coding Mod test Sys test Use

Normalized Cost for Defect Detection and Removal

RequireDesign

Code

the earlier defects are detected, the cheaper is the defect removal


5/30


Without

reviews

Using

reviews

P e o p l e R e s o u r c e

Design Code Testing Ship

Schedule

Source: M. Fagan, IBM 1986

In the 1st half of a project

reviews can achieve 20-30% of

the development effort

Planning/

Requirements

Benefits: Resources with and without reviews


6/30


Review types, main goals

Note: terms and their meaning vary in literature - this is our definition in PDMK

Intensive Inspection: thorough check of a document, reader, by preparation andmeeting (èfind defects)

Inspection: thorough check of a document by preparation and meeting

(èfind defects)

Reading: no meeting, only collecting issues found during reading

(è find defects)

4 Eye Review: meeting, no preparation, informally collect issues

(è find defects e.g. after changes)

Walkthrough: "walk through" document in a meeting

(è understanding, find defects, know-how)

Workshop: discussion about document contents in a meeting

(è decision making)

f o r m a l l e v e l


7/30


Review types: pro & contra

Intensive +: most effective finding major defects of a document

Inspection: +: additional items are often found during meeting-: most time consuming review method (reader)

Inspection: +: very effective finding major defects of a document+: additional items are often found during meeting-: takes more time than reading

Reading: +: efficient if all participants take time for checking

+: independent of location of participants-: no additional issues found which would come up in logging meeting

4 Eye Review: +: efficient, few participants, informal logging-: not central documents, only for minor changes

Workshop: +: structured discussion and logging of issues, results, solutions-: not very efficient for finding defects

Walkthrough:+: deep understanding for all of difficult parts of a document-: takes much time, not efficient for finding defects


8/30


new Review type: Intensive Inspection(additional items to Inspection in orange). According to P730103d, this review type must be used for very complex documentsof high importance (central documents like SW System Design etc.)

In the Intensive Inspection the following roles must be used: – Moderator

– Author (must not be the same person as the Moderator)

– Reader: is reviewer, with the special task to explain the review object in his own words

– Reviewer with special understanding of the review object, e.g. designer, developer, tester

It follows a defined multistage process with the following steps:

– Planning: check of the review object against entry criteria, select reviewer, distribute invitation.

– Introduction: Active introduction of the review object (topics, structure) by the author. This can be a meeting separate fromthe review meeting.

– Preparation: every reviewer checks the review object, especially following his special role. (if available, he uses thechecklist).

– Inspection Meeting: check, if the reviewer are sufficiently prepared, the reader explains the review object in his words,discussion of open topics, different understanding, the Moderator records all comments/findings and open topics. Decideupon re-inspection.

– Error analysis (Development and Review Process Improvement): The causes for errors allowed by the developmentprocess should be determined and described in the Review Protocol. Problems encountered or improvement suggestionsin the use of the Intensive Inspection method should be described in the Review Protocol.

– Re-Work/Update of the review object.

– Acceptance: Moderator checks, if the author as the responsible has corrected all findings, has clarified all open items andinitiated the handling of findings in external documents.


9/30


new Review type: Four Eyes Inspection

The Four Eyes Inspection is an informal inspection with at least two participants: authorand expert.

No preparation is necessary.

The use of the Review Minutes Template is not mandatory, also a handwritten list or acommented papercopy of the document can serve as Review Minutes. Nevertheless theauthor is responsible to file the Review Minutes, even if there's no need to use theconfiguration management therefor.è it is advised either to file the papercopies in a central 4-Eye-Review binder or –evenbetter- check out the document, change it online while doing the 4-Eye-Review and

check it in again after the 4-Eye-Review with an History Log entry like "after 4-Eye-Review"

The review result (statistical data) is entered into the Review List.

As this review type is very informal, it is usable only for documents without centralimportance or to verify minor changes.


10/30


Review Planning - Aim

• Which Document is reviewed wh en , how and by

whom ?

•select documents for review

•set review date soon after document should be available•choose appropriate review type

•name the review participants

•enter all this data in Review List


11/30


Review Planning Process

• see PDMK:

Methodsà Quality Management: Object Review Planning

• Object Reviews are also done on System level and in ME + EE

è common Object Review List


12/30


Review Planning

plan reviews according to PDMK / SMK tables by:

• phase• document with priority (tailoring!)

• recommended reviewers

• recommended Review Type

• see PDMK: Methodsà Quality Management: Choosing Review Type, Object

and Participants, Object Review Plan

• see SMK: Methodsà Quality Management: Software Object Review Plan


13/30


Review Planning: Review List

• see PDMK: Documentsà

Quality Management: Review List• see Review List Tab "Guidelines" for fill out instructions

• the planned review is entered into the Review List with:

•review object, document type, date, reviewers, moderator

• enter all data thoroughly and completely, as this list is not only the

"Review Database" for the project, but also serves as data source for

annually Review Process improvements


14/30


Review Planning: Review List usage

Review Status:• See PDMK: Methodsà Quality Management: Object Review List Usage

• The Review Status is set to PLA after the review has been planned, then it changes to TDO afterinvitation, followed by REV after the review meeting (or reading completion).

• DNE is the last state if the review has been completed successfully.• If a review has to be deferred, its state changes to DEF, until a new review date is assigned. This state

change should only be done, if a review date has to be shifted for more than 2 weeks. If a review ispostponed only for some days (e.g. review participant not available), the review state can remain in

TDO, just the updated Planned Review Date is added in the Review List, the first Planned Review Dateis set in brackets behind the updated one.

• Cancelling a review can be done in the states PLA, TDO, DEF or REV and is documented by setting itsstate to REJ (rejected) in the Review List.


15/30


Review Execution

• how a review is carriedout:

• see PDMK: Methods à

Quality Management:

Object Review Execution


16/30


Review Execution: Review Minutes

• see PDMK: Documentsà Quality Management: Object Review Minutes

• Fill out title page, footer and Overview section completely and thoroughly

• enter all comments in the list, so that the document author understands them

• assign one of the types F,I,D,O,K to each comment, see PDMK: Methodsà Quality

Management: Object Review Minutes

• after each item is commented by the author, the moderator can spot check the

implementation and set the Review State to DNE

• Review Minutes have to be checked in (demanded by P730103)


17/30


Review Execution: Review Minutes: "D" issues

• Benchmark Subproject: Process Optimization - Cost Reduction Measure #1 demanded from

01.04.2007 on:

• Leave out all D (Improvement of presentation) entries in Object ReviewMinutes and Listsè PDMK Method: Quality Management: Object Review Minutes:

ncontinue to bring up documentation issues (D points) in the review meeting

nbad readability remains a violation of review entry criteriaà postpone review

nD points are only listed in Review Minutes if document may be read by customer, for all other

documents D issues are not recorded any more

nfor all non-customer relevant documents, D count remains empty in Review Minutes andReview List

ndocument author should note and implement important D issues on his own responsibility toimprove readability of the document, however implementation of those is not checked by Review

Process

è PDMK Template Object Review Minutes:

footnote 2: ... D=Improvement of presentation (readability), ...: no change!

è PDMK Template Object Review List:

empty D count fields are not colored any more (was indicator for missing entry)


18/30


Review Execution: Test Specifications

• only review the "Test Objectives" when reviewing Test Specifications

èthe Test Objectives should give a good representation of what is tested

èitems forgotten in the test will show up quickly when focusing on Test Objectives

èthe details of each single test case are not so important and likely to be correct

(otherwise test fails), so checking those is not needed and not efficient

èthis is a "cost reduction measure", but already has been shown to raise

effectiveness in the past (before 01.April 2007)


19/30


Hints for effective Reviews

Use reviews to check all important documents in a project

more detailed planning of reviews: what / who / when?

set a focus: what should be checked in the review?

Split document if it is too big!

choose most effective review type

do a Walkthrough of critical parts during a document Inspection

select right participants (document owner, experienced people)

build up review experience for all project members (newbies!)

apply best practices based on lessons learned from earlier reviews (use laptop, beamer, BeyondCo.)

use checklists where they make sense

QA people are not needed for each review

at least 50% of all reviews should be made without QA support

QA does spot-checks of review planning, protocol and follow-up


20/30


effective Reviews: Planning: Choice of objects

late changes: short time to delivery or SOP, 'quick & dirty‘

complex changes, new features with influence throughout many other documentshigh change request rate, many changes since last review (see MR DB)

new developer (internal or external)

modules with many warnings generated by automated code analysis (see QA C results)

complex modules or interfaces, with great influence on the system (see Understand Cresults, product metrics)

modules using data from more than 1 interrupt or task level, and 1 level can interruptanother in a pre-emptive way: data consistency/ shared memory access problem

more frequent or severe problem known in the project


21/30


effective Reviews: Planning: size of objects

Plan Review Meeting no longer than 2 hours, include breaks

Consider optimum checking rate for a deep and efficient exploration of about 100-140 NLOC/h or3-5 pages/h (300 words of content per page) (Only a rough guideline)

Divide documents (tasks) into suitable chunks:

assign different (parts of) checklists to roles

concentrate on changes, or other special parts of documents

changes should be marked with changes bars, use a ‘diff’ toolcheck many modules, but for only 1 checklist item or problem, a more frequent or expensive

one reported in a late development phase

use more than 1 inspection or checking phase

A good design with small modules is a very great help - Design for Testability! (see Understand C /product metrics)


22/30


15

12

9

6

3

20 40 60 80 100

D e f e c t d e n s i t y ( d

e f e c t s / p a g e )

Inspection rate (pages/hour)Source: Tom Gilb, Denise Leigh: Software Inspection p 334,230 inspections of Sema Group (GB)

you will find more defects when inspecting pages longer

not more than 20 pages should be inspected per hour,

or defects are not found!

one review should not last longer than 2 hours

=> if a document has more than 40 pages,

chunk document in parts and inspect each

part separately!

effective Reviews: Planning: size of objects


23/30


effective Reviews: Planning: Choice of participants

Persons who are responsible for the object (module, document) or others where a good

quality of the object is in their own interest, e.g. users of objects,

Experienced designers and integrators, who have the overview over the design, the

system and its requirements. Might be a bottleneck!

Do not change the whole review team between reviews concerning 1 module, e.g.

between specification - design - test specification - code.

Participants can also be persons from other departments, e.g. from system testing or

from the customer. It can be useful to have some more perspectives to the object, e.g. auser or legal perspective.

New people which are not yet routine-blinded may bring in new ideas or raise interesting

questions.


24/30


effective Reviews: Experience Data

The following statistical Experience Data has been determined by yearly analysis of our

review data:

the average duration of the Logging Meeting is 2 hours

the average Review Effort is about 8 hours

(Review Effort = # of participants*Logging meeting time + Σ preparation time + Administration time)

the average number of Review participants is around 3


25/30


80

40

23

8

0

1 2 3 4 5

Experience as Checker: Also otherdocuments produced by Gary became better

after participating several inspections.

Gary was “documentauthor” at points 1 to 5

Issues IdentifiedSource: Douglas Aircraft, 1988,

private report to Tom Gilb

effective Reviews: Personal Learning Curve


26/30


Guidelines and best practicesExtra motivation (Kick-off)

Moderator

•should encourage beginners and shy persons, ask for help and give tips.Beginners should start small, not with all rules, but with important.

should ask - not wait - for suggestions for improvement

Communicate!Known problems or open items should be solved before a review takes place!

If there are major problems encountered all participants have to be informed

immediately so that the review procedure can be aborted as quickly as possible!

If a participant cannot join the team, the moderator should contact his/her boss tochange priority.


27/30


Guidelines and best practicesThe 'hot seat'

Documents are checked, not the author:

'choice of words', e.g. ‘here you made another mistake: ...’

no big boss

No improvement suggestions to the author during logging meeting, but

afterwards

Planning of review participants should avoid possible personal friction between

author and checkers: checkers should dare to raise items

Consider: Everyone will have this role!


28/30


Guidelines and best practicesChecking

Focus on ‚majors‘ = economical important defects, which cannot be found by

tools - e.g. static code analysis - nor module test. Checklists should reflect this.

In reality this is difficult, many minors are found much more often. Though it is a

good feeling to find a high number of issues, it is of course much better to find 1

major instead of 100 minors only.

Minors like syntax errors in a comment should be sent to the author beforehand,

without an extra logging.

The checking rate used as a guideline for planning reviews and the plannedduration is no law: stop checking when run dry

choose best environment, with no interrupts like telephone calls possible,

set a goal: 70% majors!


29/30


Guidelines and best practicesLogging meeting

§Try to express the content of a logging item in a few precise words.

§The author should watch the log (‘real time log validation’), or should express thecontent of a logging item in his own words. The author is normally the editor, so he

must understand what is logged.

§Use the auto-save function when logging to a sheet on a PC !

§'No' discussion, just log it!

If an item cannot be solved in a short time, or a question cannot be answered

quickly, the discussion should be deferred to a ‚3rd hour‘, only with the peopleinvolved.

§Use Beamer and Laptop to show document and protocol


30/30

Thank you for your attention

– are there further questions?

Training Review Technique

Documents

Transcript of Training Review Technique