Managing Reviews and Inspection

7/29/2019 Managing Reviews and Inspection

1/57

Software Review and Audit


2/57

Software Review and Inspection

Reviews provide a powerful way to improve quality by

providing a means by which defects can be detected

(and corrected) early in development.


3/57

Defects and Correctness

Defect:Anything that detracts from a

programs ability to completely and

effectively meet the users needs (HumphreyPSP)

Correctness: A program is said to be

correct only if it contains no defects. Correctness is only definable with respect to

the users needs.


4/57

Verification and Validation


5/57

Verification and Validation

Validation: Are we building the right

system?

Verification: Are we building the systemright?


6/57

Types of Review

There are a number of types of reviewranging in formailty and effect. Theseinclude:

Buddy Checking

having a person other than the author informallyreview a piece of work.

generally does not require collection of data

difficult to put under managerial control

generally does not involve the use of checklists toguide inspection and is therefore not repeatable.


7/57

Types of Review

Walkthroughs

generally involve the author of an artifact presenting that

document or program to an audience of their peers

The audience asks questions and makes comments on theartifact being presented in an attempt to identify defects

often break down into arguments about an issue

usually involve no prior preparation on behalf of the audience

usually involve minimal documentation of the process and of

the issues found process improvement and defect tracking are therefore not easy


8/57

Types of Review

Review by Circulation

similar in concept to a walkthrough

artifact to be reviewed is circulated to a group of the

author(s) peers for comment avoids potential arguments over issues, however it

also avoids the benefits of discussion

reviewer may be able to spend longer reviewing the

artifact there is documentation of the issues found, enabling

defect tracking

usually minimal data collection


9/57

Types of Review

Inspection (Fagan 76)

formally structured and managed peer reviewprocesses

involve a review team with clearly defined roles specific data is collected during inspections

inspections have quantitative goals set

reviewers check an artifact against an unambiguous

set of inspection criteria for that type of artifact The required data collection promotes process

improvement, and subsequent improvements inquality.


10/57

Software Inspection

The inspection process comprises three broad

stages:

preparation

collection

Repair

Gilb and Graham [GilbGraham93] expand this

three stage process into the inspection steps;Entry, Planning, Kickoff Meeting, Individual

Checking, Logging Meeting, Root Cause Analysis

Edit, Follow Up, Exit.


11/57

Benefits of Inspection

30% to 100% net productivity increases;

Overall project time saving of 10% to 30%;

5 to 10 times reduction in test execution costs and time;

Reduction in maintenance costs of up to one order of

magnitude;

Improvement in consequent product quality;

Minimal defect correction backlash at systems integration

time.

In addition to these tangible benefits, less tangible benefits

such as a training effect for inspectors are also evident.


12/57

Disadvantages

Up front costs (although far outweighed by

benefit):

Training

Implementation Support

Ongoing allocation of staff resources

Not strictly repeatable


13/57

Typical Inspection Process

Entry

The author of the artifact requests that it beinspected.

The artifact to be inspected is checked by theinspection moderator to ensure that certainentry criteria are met.

The primary purpose of this stage is to ensurethat inspection time is not wasted on artifactsthat contain defects which the author shouldrightly have found.


14/57


Planning

The moderator determines the practical aspects

of the inspection. This may include: Determining the size and composition of the

inspection team

Determining the goals of the inspection.

Determine the timing and purpose of the meetings.


15/57


Kickoff Meeting

Roles for the inspection team are assigned and clarified(generally the moderator does this).

Documents, including the artifact, its source document(eg SRS for Design), the inspection checklist, andinspection rules are distributed and checked (Defects inthe source document and checklist are sometimes foundin these meetings, however, this meeting should be kept

short). In some variations, the author(s) of the artifact may be

required to give a quick walkthrough of the artifact tobe inspected and its relation to the other documentation.


16/57


Individual Checking

The final stage of preparation is individual checking(although it could also be considered the main stage of

collection). The majority of defects found in inspection processes

are found in the individual checking stage.

During this stage an individual reviewer reads theartifact and with the guidance of an inspection checklistattempt to find defects in the artifact.

The reviewer should record any issues found.

The reviewer should also make some effort todetermine what they consider the severity of a defect to

be and classify the defect.


17/57

Defect Severity & Classification

Severity

Major - Will cause problem if not corrected

Minor - Will not

If in doubt its major

Classification (for now)

Missing

Extra

Wrong


18/57

A Sample Form


19/57


Logging Meeting

A planned and moderated meeting with the primarypurpose of logging the issues found by the reviewers.

all reviewers should be given a chance to raise theirissues as a scribe logs the issues being raised

It is important that an issue is only logged once.

moderator should ensure that discussion about issues iskept to a minimum in order to maintain the continuityof the meeting.

Some variations of this process include group defectfinding as an activity at the end of this meeting.


20/57


21/57


Edit

The editor (usually the author) is resposible for

addressing all logged issues in the inspectedartifact.

The editor decides if something is a defect or

not.

All defects must be corrected.

All non-defects should also be addressed in

some way.


22/57


Follow Up and Exit

moderator checks that all defects have been addressed(and all non-defect issues addressed if required).

moderator must also ensure that any defects found in asource document during inspection are forwarded to theowner of that document for correction.

moderator may calculate certain metrics in this stage tobe analysed to assess the effectiveness of an inspection.

may also be used to hold a meeting to evaluate andrecommend inspection process improvement

An inspection will be exit when pre-defined set ofinspection exit criteria have been satisfied.


23/57

Inspection Roles

Moderator / Leader

Author / Producer

Reviewer / Reader

Scribe


24/57

Using the Log Form (1)

The first section describes the artifact being inspected, and

summarises the individual reviewers data (from the review

form in the previous stage if used).


25/57


This section of the form contains information about major

defects found and indicates which reviewers found them.

Some variations also log minor defects here.


26/57


The next section of the found simply totals the data in the

columns for the reviewers.


27/57


The final section of the formis used for metrics calculation.


28/57

Inspection Metrics

Total Defects Found = A + B - C, where A and B are the

number found by reviewer A and B respectively and C is the

number found by both A and B.

Estimated Total Defects = AB/C

Yield = Total Defecs Found / Estimated Total Defects * 100%

Defect Density = Total Defects Found / Size

Inspection rate = size / total inspection hours


29/57

Estimating Total Defects

Capture / Recapture Method Capture a number of fish, tag them and release them (let this

number be S1).

Allow time for the first sample population to redistribute.

Capture a second number of fish (let this number be S2).

Count the number of tagged fish in the second population (let this

number be ST).

Calculate the proportion of tagged fish in the second population

(let this number be T, then T=ST/S2).

We assume that T is representative of the proportion of tagged fish

in the total population (POP), so T*POP=S1, or for our purposes

POP=S1/T.


30/57

I like fish but what about the defects

Let the number of defects found by one reviewer be the tagged

population (A).

Assume an even likelyhood of finding all defects (even distribution,...)

Count the number of defects found by the second reviewer (B).

Count the number of defects found by the second reviewer that were

also found by the first (C the common defects).

Calculate the proportion of common defects in the second reviewers

defects (T=C/B).

We assume that T is representative of the proportion of commondefects in the total number of defects (EstTot), so T * EstTot=A, or for

our purposes EstTot = A/T = (A*B)/C.


31/57

Calc. yield with > 2 reviewers


32/57

Sample Re-Inspection Criteria

Inspection rate too high

Yield too low

Majors found / Total found too low

Unusual defect distribution

High defect density

Should be specified in plan


33/57

AUDIT

The goal is to provide a guide to those responsible forsoftware-related auditing and how best to achieve the finaloutcome of a fair, objective, and useful software-relatedaudit that improves the situation as found.

An independent examination of a work product or set ofwork products to assess compliance with specifications,standards, contractual agreements, or other criteria IEEE

Purpose: to provide an independent evaluation of

conformance of software products and processes toapplicable regulations, standards, guidelines, plans, and

procedures


34/57

Reasons

A specific project milestone has been reached and

an audit is initiated as planned or as required by

the auditing organizations charter.

External parties or customers request an audit of a

specific item, at a specific date, or at a project

milestone. This could be part of a contract

agreement. An internal organization has requested the audit,

establishing a clear and specific need.


35/57

Software-related Audit

The client, person, or organization that

requests the audit;

The auditor or team who performs the audit; The auditee whose work is being examined.


36/57

Auditors Responsibilities

Determining the team size;

Briefing team members on the audit scope and areas to beaudited;

Providing background about the organization being

audited; Assigning the workload of who will audit what areas;

Determining the audit schedule;

Notifying and briefing the audited organization on thescope of the audit and materials that need to be provided;

Ensuring that the audit team is prepared to conduct theaudit;

Ensuring that the audit plan or procedures are performed;

Issuing reports in accordance with the audit plan or

procedures.


37/57

Auditees Responsibilities

Establishing a professional, positive attitude aboutthe audit among the members of the auditedorganization;

Participating in the audit; Providing all relevant materials and resources tothe audit team;

Understanding the concerns of the auditors andverifying their factual accuracy;

Providing a response to the audit report;

Correcting or resolving deficiencies cited by theaudit team.


38/57

Types of Software Audits

Software piracy audit

Security audit

Information systems audit ISO 9001:2000 software audit

CMMI-DEV appraisal

Personal audit experiences

Automated audits


39/57

Software Piracy Audit


40/57

Security Audit

Issues

Backups

Antivirus. Firewall

Access control.


41/57

Security Audit

ISO 17799

It is organized into 10 sections

Business continuity planning;

Systems access control;

System development and maintenance;

Physical and environmental security;

Compliance;

Personnel security;

Security organization; Computer and operations management;

Asset classification and control;

Security policy.


42/57

Security Audit


43/57

Security Audit


44/57

Information Systems Audit

Information systems auditing evaluates whether

computer-based information systems safeguard

assets, maintain data integrity, achieve

organizational objectives effectively, and consumeresources efficiently.

Ron Weber


45/57


The information systems lead auditor must be sensitive tothe following

All audits should be conducted only with prior approval of themanagement.

Consult an advocate for all the applicable legislationthat is, ifyou do not want to be caught by surprise later.

Ensure that one does not violate the software copyright in anyform.

Ask the accounts department how long they retain financial

records. Ensure that there is no misuse of the information processing

facilities by any person, insider as well as outsider.

If you are traveling with your notebook PC where you have storedencrypted files, you may be breaking a few laws of the land.


46/57



47/57



48/57


The information systems lead auditor must be sensitive tothe following

All audits should be conducted only with prior approval of themanagement.

Consult an advocate for all the applicable legislationthat is, ifyou do not want to be caught by surprise later.

Ensure that one does not violate the software copyright in anyform.

Ask the accounts department how long they retain financial

records. Ensure that there is no misuse of the information processing

facilities by any person, insider as well as outsider.

If you are traveling with your notebook PC where you have storedencrypted files, you may be breaking a few laws of the land.


49/57

ISO 9001:2000 Software Audit

ISO 9001:2000 Software


50/57


Audit



51/57


Audit

ISO/IEC 90003:2004 explains how ISO9001:2000 can be applied to software relatedservices.

Sample Checklist Provide quality infrastructure Identify infrastructure needs.

Provide needed infrastructure.

Maintain your infrastructure.

Maintain the tools you need in order to manage software.

Control software design and development Plan software design and development

Plan software design and development.



52/57


Audit

Identify infrastructure needs. Identify the infrastructure you need in order to develop software.

Identify the hardware you need in order to develop software.

Identify the software you need in order to develop software.

Identify the facilities you need in order to develop software.

Identify the tools you need in order to manage software.

Identify the tools you need in order to develop software.

Identify the tools you need in order to support software.

Identify the tools you need in order to protect software.

Identify the tools you need in order to control software.


53/57

CMMI-DEV appraisal


54/57

CMMI-DEV appraisal

Phase Process

1 Plan and Prepare for Appraisal

1.1 Analyze Requirements

1.2 Develop Appraisal Plan

1.3 Select and Prepare Team

1.4 Obtain and Inventory Initial Objective Evidence

1.5 Prepare for Appraisal Conduct

2 Conduct Appraisal

2.1 Prepare Participants

2.2 Examine Objective Evidence

2.3 Document Objective Evidence2.4 Verify Objective Evidence

2.5 Validate Preliminary Findings

2.6 Generate Appraisal Results

3 Report Results

3.1 Deliver Appraisal Results

3.2 Package and Archive Appraisal Assets


55/57

CMMI-DEV appraisal

Benefits to the organization: Improved accuracy in appraisal results delivered by

external appraisal teams (i.e., clear understanding ofimplemented processes, strengths, and weaknesses);

Detailed understanding of how each project or supportgroup has implemented CMMImodel practices, andthe degree of compliance and tailoring of organizationalstandard processes;

Assets and resources for monitoring process

compliance and process improvement progress; Residual appraisal assets that can be reused on

subsequent appraisals, minimizing the effort necessaryfor preparation.


56/57

Personal Audit Experiences


57/57

Automated Audits

Managing Reviews and Inspection

Documents

Transcript of Managing Reviews and Inspection