Capability Maturity Model Integration: Implications for Quality Assurance and Testing Presentation...

Capability Maturity Model Integration: Implications for

Quality Assurance and Testing

Presentation to SCQAA 20 June 2002

Rick Hefner, TRW [email protected]

(310) 812-7290

SCQAA - Rick Hefner 2

Agenda

• Project History

• CMMI Structure

• Comparisons with SW-CMM v1.1, SE-CMM, and EIA/IS 731

• Assessment Methodology

• Transition Timelines and Strategies

• Implications for Quality Assurance Professionals


What is a CMM?

• Capability Maturity Model: A reference model of mature practices in a specified discipline -- industry best practices

• CMMs differ by– Discipline (software, systems, acquisition, etc.)– Structure (staged versus continuous)– How Maturity is Defined (process improvement path)– How Capability is Defined (institutionalization)

• “Capability Maturity Model®” and CMM® are used by the Software Engineering Institute (SEI) to denote a particular class of maturity models

Capability Maturity Model®, CMM®, CMM Integration, and CMMI are service marks and registered trademarks of Carnegie Mellon University


Commonly Used CMMs

Software CMM staged software development

System Engineering CMM continuous system engineering

System Engineering Capability Model continuous system engineering

Software Acquisition CMM staged software acquisition

System Security Engineering CMM continuous security engineering

Personal Software Process stagedindividual software development

FAA-CMM continuous software engineering, systems engineering, and acquisition

IPD-CMM continuousintegrated product development

People CMM stagedworkforce

SPICE Model continuous software development


SoftwareCMM

SoftwareCMM

SystemsSecurity

Engr CMM

SystemsSecurity

Engr CMM

SystemsEngrCMM

SystemsEngrCMM

PeopleCMM

PeopleCMM

SecuritySystems

EngrCMM

SecuritySystems

EngrCMM

PeopleCMM

PeopleCMM

ZZZCMMZZZ

CMM

FAAiCMMFAA

iCMM

IPDCMMIPD

CMM

SoftwareAcqCMM

SoftwareAcqCMM

CMMI

What is the Problem?

• Different structures, formats, terms, ways of measuring maturity• Causes confusion, especially when using more than one CMM• Hard to integrate them in a combined improvement program• Hard to use multiple models in supplier selection


The CMMI Project

• DoD sponsored collaboration between industry, Government, academia

• Over 100 people involved

• U.S. Army, Navy, Air Force• Federal Aviation Administration• National Security Agency• Software Engineering Institute• ADP, Inc.• AT&T Labs• BAE• Boeing• Computer Sciences Corporation• EER Systems• Ericsson Canada• Ernst and Young• General Dynamics• Harris Corporation

• Honeywell• KPMG• Litton• Lockheed Martin• Motorola• Northrop Grumman• Pacific Bell• Q-Labs• Raytheon• Rockwell Collins• Sverdrup Corporation• Thomson CSF• TRW


CMMI Models

Disciplines– systems engineering– software engineering– integrated process and product

development (IPPD)– supplier sourcing– combinations of the above

Representations– staged– continuous

CMMI Models

Source Models

• Capability Maturity Model for Software V2, draft C (SW-CMM V2C)

• EIA Interim Standard 731, System Engineering Capability Model (SECM)

• Integrated Product Development Capability Maturity Model, draft V0.98 (IPD-CMM)


Maturity Levels

Process unpredictable, poorly controlled and reactive

Process characterized for projects and is often reactive

Process characterized for the organization and is proactive

Process measuredand controlled

Focus on processimprovement

Level 4QuantitativelyManaged

Level 1Performed

Level 2Managed

Level 5Optimizing

Level 3Defined

• Each level is a layer in the foundation for continuous process improvement


Causal Analysis and ResolutionOrganizational Innovation and Deployment5 Optimizing

4 Quantitatively Managed

3 Defined

2 Managed

Continuous process improvement

Quantitativemanagement

Processstandardization

Basicprojectmanagement

Quantitative Project ManagementOrganizational Process Performance

Organizational Process FocusOrganizational Process DefinitionOrganizational Training Integrated Project ManagementRisk ManagementDecision Analysis and ResolutionRequirements DevelopmentTechnical SolutionProduct IntegrationVerificationValidation

Requirements Management Project PlanningProject Monitoring and ControlSupplier Agreement Management Measurement and AnalysisProcess and Product Quality AssuranceConfiguration Management

1 Performed

Process AreasLevel Focus

CMMI Process Areas


Process Area Group Relationships

Process Management

Organizational Process FocusOrganizational Process Definition

Organizational TrainingOrganizational Process Performance

Organizational Innovation and Deployment

Project Management

Project PlanningProject Monitoring and Control

Supplier Agreement ManagementIntegrated Project Management

Risk ManagementQuantitative Project Management

Engineering

Requirements DevelopmentRequirements Management

Technical SolutionProduct Integration

VerificationValidation

Support

Configuration ManagementProcess and Product Quality Assurance

Measurement and AnalysisDecision Analysis and ResolutionCausal Analysis and Resolution

analyze

empower analyze

employ measure & assist

standardize processes


Common Features For Each Process Area

• Commitment to Perform – Establish an Organizational Policy

• Ability to Perform – Plan the Process– Provide Resources– Assign Responsibility– Train People– Establish a Defined Process

• Directing Implementation– Manage Configurations– Identify and Involve Relevant Stakeholders– Monitor and Control the Process– Collect Improvement Information

• Verification– Objectively Evaluate Adherence– Review Status with Higher-Level Management


CMMI Staged Model Structure

Commitmentto Perform

Maturity Levels

Specific Goals

Process Area 2

Common Features

Process Area 1 Process Area n

Abilityto Perform

DirectingImplementation

VerificationActivities

Performed

Generic Goals

...

SpecificPractices

SubpracticesAmplificationsElaborations



SpecificPractices

SpecificPractices Specific

Practices




SpecificPractices

GenericPractices


Required, Expected, Informative

Commitmentto Perform

Maturity Levels

Specific Goals

Process Area 2

Common Features

Process Area 1 Process Area n

Abilityto Perform

DirectingImplementation

VerificationActivities

Performed

Generic Goals

SpecificPractices

...


RequiredNeeded to satisfy

CMMI

ExpectedTypical practices to

meet goals;Used as a starting

point in assessments

InformativeIdeas to considerSubpractices

AmplificationsElaborations


SpecificPractices

SpecificPractices Specific

Practices




SpecificPractices

GenericPractices

SW-CMM v1.1 vs. CMMIProcess Areas

Defect Prevention Causal Analysis and ResolutionTechnology Change Mgmt Organizational Innovation & DeploymentProcess Change Management

Quantitative Process Mgmt Organizational Process PerformanceSoftware Quality Mgmt Quantitative Project Management

Organization Process Focus Organization Process Focus Organization Process Definition Organization Process DefinitionTraining Program Organizational TrainingIntegrated Software Mgmt Integrated Project Management

Risk ManagementSoftware Product Engr Requirements Development

Technical SolutionProduct Integration

Intergroup Coordination VerificationPeer Reviews Validation

Decision Analysis and Resolution

Requirements Management Requirements ManagementSoftware Project Planning Project PlanningSoftware Project Tracking & Oversight Project Monitoring and ControlSoftware Subcontract Mgmt Supplier Agreement ManagementSoftware Quality Assurance Product & Process Quality Assurance Software Configuration Mgmt Configuration Management

Measurement and Analysis

LEVEL 5OPTIMIZING

LEVEL 4MANAGED

LEVEL 3DEFINED

LEVEL 2REPEATABLE

14

SW-CMM v1.1 vs. CMMICommon Features

15


EngineeringPA06 Understand

Customer Needs and Expectations

PA02 Derive and Allocate Requirements

PA03 Evolve System Architecture

PA01 Analyze Candidate Solutions

PA05 Integrate SystemPA07 Verify and Validate

SystemPA04 Integrate

Disciplines

SE-CMM

Engineering

CMMI

Technical1.1 Define

Stakeholder and System Level Requirements

1.2 Define Technical Requirements

1.3 Define Solution

1.4 Assess and Select

1.5 Integrate System

1.6 Verify System

1.7 Validate System

SECM

2.3

SE-CMM and SECM (EIA/IS 731) vs. CMMI Common Features - Engineering

Requirement Mgmt

RequirementDevelopment

Technical Solution

Decision Analysis and Resolution

Product Integration

Verification

Validation


Project ManagementProject Planning

Project Mon and Con

Supplier Agree Mgt

Integrated Proj Mgt

Risk Management

* Not in SE-CMM

SE-CMM

Management

2.1 Plan and Organize2.2 Monitor and

Control 2.3 Integrate

Disciplines 2.4 Coordinate with

Suppliers2.5 Manage Risk2.6 Manage Data2.7 Manage

Configurations2.8 Ensure Quality

SECM

*

Project

PA12 Plan Technical Effort

PA11 Monitor and Control Technical Effort

PA10 Manage RiskPA09 Manage

ConfigurationsPA08 Ensure Quality

CMMI

PA04

PA18

SE-CMM and SECM (EIA/IS 731) vs. CMMICommon Features - Project Management

SupportConfiguration Mgt

Proc and Prod QA

Meas and Analysis

L2 GP


SE-CMM

Environment

3.1 Define and Improve the SE Process

3.2 Manage Competency

3.3 Manage Technology

3.4 Manage SE Support Environment

SECM

OrganizationPA13 Define Organization’s

SE ProcessPA14 Improve

Organization’s SE Process

PA17 Provide Ongoing Knowledge and Skills

PA15 Manage Product Line Evolution

PA16 Manage SE Support Environment

PA18 Coordinate With Suppliers

CMMI

Org Process FocusOrg Process Def

Organizational Tng

Quant Project Mgt

Org Process Perf

Causal An & Res

Org Innovation and Deployment

(IPPD Extension) Organizational Environment Integration

Process Management

2.4

SE-CMM and SECM (EIA/IS 731) vs. CMMICommon Features - Process Management

L4 GP

L4 GP

L5 GP


Summary of Changes

• Organizations that are currently using SW-CMM v1.1, SE-CMM, or EIA/IS 731 should be able to smoothly transition

• Will require additional infrastructure– Identification and involvement of relevant stakeholders– Configuration management of intermediate work products

• Will require additional processes– Measurement and Analysis– Decision Analysis and Resolution– Requirements Development– Risk Management

• Additional focus on engineering processes

Software organizations adopting CMM-based improvements for the first time should use the CMMI


Appraisal Requirements for CMMI (ARC)

• Similar to the current CMM Appraisal Framework (CAF) V1.0– A guide to assessment method developers

• Specifies the requirements for classes of assessment methods– Class A: Full, comprehensive assessment methods– Class B: Initial, incremental, self-assessments– Class C: Quick-look

• Method developers can declare which class their method fits

• Method buyers will understand the implications of the method proposed


Standard CMMI Assessment Method for Process Improvement (SCAMPI) v1.1

• Similar to CBA IPI method

• Led by authorized Lead Assessor

• Team of 4-9 trained assessors

• Tailorable to organization and model scope

• Products:– SCAMPI Method Description Document– Maturity questionnaire(s), electronic work aids, templates

The method was updated to version 1.1 to improve efficiency


CMMI Transition Plan

Development Phase• Development of CMMI products• Verification and validation of CMMI products

Transition Phase• Approval of initial CMMI products for

public release• Evidence of sufficient use• Transition planning to help organizations

use CMMI products

Sustainment Phase• Upkeep and continuous improvement

of the product suite• Additional evidence of adoption and

useV1.0

Aug 2000V1.1

Dec 2001SW-CMM, EIA 731 phased out

Aug 2003V0.2

Aug 1999

Pilots


Role of the Quality Assurance Professional

• QA has a defined role in every process area, and a process area of their own


Generic Practices

• GP 2.9 (VE1) Objectively Evaluate Adherence– Objectively evaluate adherence of the process and the work

products and services of the process to the applicable requirements, objectives, and standards, and address noncompliance.

• Quality assurance must be performed in all processes, work products, and services– Engineering– Support– Project Management– Organizational process management


Process and Product Quality Assurance - 1

• Purpose– Objectively review activities and work products for their adherence to

applicable requirements, process descriptions, standards, and procedures, and communicate the results to staff and management.

• P&PQA ensures planned processes are implemented; Verification ensures the specified requirements are satisfied.

• QA can be embedded in the process, but:– Objectivity must be maintained– All QA performers must be trained– An independent reporting channel must be maintained

• QA should participate in establishing the plans, processes, standards and procedures to ensure they:– Fit the project’s needs– Will be useable for performing quality assurance evaluations.



• Projects must designate which processes and work products they intend to audit

– What processes and work products drive overall quality?

– Where is insight into quality the most valuable?

– Assessor typically expect all processes to be audited

SG 1 Objectively Evaluate Processes and Work Products

SP 1.1 Objectively Evaluate ProcessesObjectively evaluate the designated performed processes against the applicable process descriptions, standards and procedures.

SP 1.2 Objectively Evaluate Work Products and ServicesObjectively evaluate the designated work products and services against the applicable process descriptions, standards, and procedures.



• Requires a independent reporting channel for noncompliance issues

– Makes senior management responsible for adjudicating quality versus schedule conflicts

SG 2 Provide Objective Insight

SP 2.1 Communicate and Ensure Resolution of Noncompliance IssuesCommunicate quality issues and ensure resolution of noncompliance issues with the staff and managers.

SP 2.2 Establish RecordsEstablish and maintain records of the quality assurance activities.



• Quality assurance activities must be fully supported by the project and organizational infrastructure

GG 2 Institutionalize a Managed Process

GP 2.1 (CO 1) Establish an Organizational Policy

GP 2.2 (AB 1) Plan the Process

GP 2.3 (AB 2) Provide Resources

GP 2.4 (AB 3) Assign Responsibility

GP 2.5 (AB 4) Train People

GP 2.6 (DI 1) Manage Configurations

GP 2.7 (DI 2) Identify and Involve Relevant Stakeholders

GP 2.8 (DI 3) Monitor and Control the Process

GP 2.9 (VE 1) Objectively Evaluate Adherence

GP 2.10 (VE 2) Review Status with Higher-Level Management


Verification - 1

• Purpose– Assure that selected work products

meet the specified requirements (e.g., peer reviews, testing)

• Key Issues– Recognizing verification is an

incremental process that occurs throughout the development of the product and work products» Verification of the Requirements» Verification of the evolving work

products» Verification of the completed

product– Identifying where verifications will

occur (as part of program strategy)

User needs

System requirements

Software requirements

Software design

Software code

Test cases

Test results

Code peer review

Code testing


Verification - 2

• Projects identify (and document) the work products they intend to verify

• Verification environment typically refers to a test bed

– Could also imply the facilities and tools needed for peer reviews

• “Procedures” are not typically called out in the CMMI – implies the need for more detailed planning

SG 1 Prepare for VerificationPreparation for verification is conducted.

SP 1.1 Select Work Products for VerificationSelect the work products to be verified and the verification methods that will be used for each.

SP 1.2 Establish the Verification EnvironmentEstablish and maintain the environment needed to support verification.

SP 1.3 Establish Verification Procedures and CriteriaEstablish and maintain verification procedures and criteria for the selected work products.


Verification - 3

• Different verification methods: peer review, inspection, demonstration, test, analysis

• Preparing for peer review includes identification of:

– Staff involved, roles– Review criteria– Schedule

• Peer review data are analyzed to improve review effectiveness

– Size of the product– Composition of review team– Type of peer review– Preparation time per reviewer– Length of the review meeting– Number of defects found, type and

origin of defect, etc.

SG 2 Perform Peer ReviewsPeer reviews are performed on selected work products.

SP 2.1 Prepare for Peer ReviewsPrepare for peer reviews of selected work products.

SP 2.2 Conduct Peer ReviewsConduct peer reviews on selected work products and identify issues resulting from the peer review.

SP 2.3 Analyze Peer Review DataAnalyze data about preparation, conduct, and results of the peer reviews.

SG 3 Verify Selected Work ProductsSelected work products are verified against their specified requirements.

SP 3.1 Perform VerificationPerform verification on the selected work products.

SP 3.2 Analyze Verification Results and Identify Corrective ActionAnalyze the results of all verification activities and identify corrective action.


Verification - 4

• Verification activities must be fully supported by the project and organizational infrastructure

GG 2 Institutionalize a Managed Process

GP 2.1 (CO 1) Establish an Organizational Policy

GP 2.2 (AB 1) Plan the Process

GP 2.3 (AB 2) Provide Resources

GP 2.4 (AB 3) Assign Responsibility

GP 2.5 (AB 4) Train People

GP 2.6 (DI 1) Manage Configurations

GP 2.7 (DI 2) Identify and Involve Relevant Stakeholders

GP 2.8 (DI 3) Monitor and Control the Process

GP 2.9 (VE 1) Objectively Evaluate Adherence

GP 2.10 (VE 2) Review Status with Higher-Level Management


Implications for the Quality Assurance Professional

• More companies are adopting the CMMI (and CMM)– Required by customers– Improves predictability and efficiency; reduces rework and cost– A competitive discriminator

• The CMMI/CMM requires a significant QA presence– Involvement in all projects, all processes– Significant support required to satisfy PPQA process area

• CMMI/CMM provides an opportunity to educate the community on the proper role of QA


Summary

• CMMI is a natural evolution in industry maturity

• Organizations that are currently using SW-CMM v1.1, SE-CMM, or EIA/IS 731 should be able to smoothly transition

• Quality assurance professionals have an important role in the CMMI

See http://www.processimprovement.com for more information and resources

http://www.processimprovement.com/

Capability Maturity Model Integration: Implications for Quality Assurance and Testing Presentation...

Documents

Transcript of Capability Maturity Model Integration: Implications for Quality Assurance and Testing Presentation...