Standard CMMI Appraisal Method for Process Improvement

(SCAMPI)

• ARC Class A Compliant

• Version 1.1 in 2002

• Version 1.2 in 2006

Approach

• Discovery

• Verification

SCAMPI Phase Structure

• Phase I Plan and Prepare For Appraisal

• Phase II Conduct Appraisal

• Phase III Report Appraisal Results

Key Preparation Questions

• Should we have an appraisal?

• Should we do it now or later?

• Should SE and SW be appraised together with separate results, or should their appraisals be integrated?

• Which appraisal class should be used?

• What should be included in each appraisal?

Preparation Considerations

• Organization scope

• Model Scope

• Size

• Disciplines

• Improvement Status

Successful appraisal teams require:

• a plan, including objectives, constraints, scope, outputs, tailoring, resources, cost, schedule, logistics, risks, and commitment

• clearly defined goals

• clearly defined roles

• established rules of engagement

• well-defined decision procedures

• understanding of the group process

• clear communications

• proper team behavior

• balanced participation

• training

Decision Making - Consensus

• Decision making on appraisal teams is done by consensus, not unanimous vote nor majority vote

• Consensus is finding a proposal acceptable enough that all team members can support it or live with it, and no team member opposes it

• The consensus process can require several cycles through the decision cycle as the team continues to gather and analyze factual information bearing on a decision

• If consensus cannot be obtained, a decision should not be forced, e.g., through majority vote - the item under consideration must be left not rated

Method Definition Document

MDD 1.2 Outline

SCAMPI MDD V1.2 Processes - 1• 1 Plan and Prepare for Appraisal

– 1.1 Analyze Requirements• 1.1.1 Determine Appraisal Objectives• 1.1.2 Determine Appraisal Constraints• 1.1.3 Determine Appraisal Scope• 1.1.4 Determine Outputs• 1.1.5 Obtain Commitment to Appraisal Input

– 1.2 Develop Appraisal Plan• 1.2.1 Tailor Method• 1.2.2 Identify Needed Resources• 1.2.3 Determine Cost and Schedule• 1.2.4 Plan and Manage Logistics• 1.2.5 Document and Manage Risks• 1.2.6 Obtain Commitment to Appraisal Plan

– 1.3 Select and Prepare Team• 1.3.1 Identify Appraisal Team Leader• 1.3.2 Select Team Members• 1.3.3 Prepare Team

– 1.4 Obtain and Inventory Initial Objective Evidence• 1.4.1 Obtain Initial Objective Evidence• 1.4.2 Inventory Objective Evidence

– 1.5 Prepare for Appraisal Conduct• 1.5.1 Perform Readiness Review• 1.5.2 Prepare Data Collection Plan• 1.5.3 Replan Data Collection

SCAMPI MDD V1.2 Processes - 2• 2 Conduct Appraisals

– 2.1 Prepare Participants II-65• 2.1.1 Conduct Participant Briefing

– 2.2 Examine Objective Evidence• 2.2.1 Examine Objective Evidence from Documents• 2.2.2 Examine Objective Evidence from Interviews

– 2.3 Document Objective Evidence• 2.3.1 Take/Review/Tag Notes• 2.3.2 Record Presence/Absence of Objective Evidence• 2.3.3 Document Practice Implementation• 2.3.4 Review and Update the Data Collection Plan

– 2.4 Verify Objective Evidence• 2.4.1 Verify Objective Evidence• 2.4.2 Characterize Implementation of Model Practices

– 2.5 Validate Preliminary Findings• 2.5.1 Validate Preliminary Findings

– 2.6 Generate Appraisal Results• 2.6.1 Derive Findings and Rate Goals• 2.6.2a Determine Process Area Capability Level• 2.6.2b Determine Satisfaction of Process Areas• 2.6.3a Determine Capability Profile• 2.6.3b Determine Maturity Level• 2.6.4 Document Appraisal Results

SCAMPI MDD V1.2 Processes - 3

• 3 Report Results– 3.1 Deliver Appraisal Results

• 3.1.1 Deliver Final Findings• 3.1.2 Conduct Executive Session(s)• 3.1.3 Plan for Next Steps

– 3.2 Package and Archive Appraisal Assets• 3.2.1 Collect Lessons Learned• 3.2.2 Generate Appraisal Record• 3.2.3 Provide Appraisal Data Package to CMMI Steward• 3.2.4 Archive and/or Dispose of Key Artifacts

Determination of Practice Implementation

• The basic approach to determination of practice implementation is to:

– characterize the degree to which the practice is implemented, and

– note any factors which should be considered in addition as the instantiation data is aggregated (strengths, weaknesses, etc.).

Characterizing Practice Implementation -1

• Assign characterization values reflecting the extent of practice implementation for each instance

• Aggregate practice characterizations to organizational unit level using defined method aggregation rules

• Iterate and focus revisions to data collection plan

• Generate findings based on aggregation of weaknesses and strengths

Rating Goals

• Goal ratings are a function of the extent to which the corresponding practices are present in the planned and implemented processes of the organization.

• Team judgment is used to rate goals where the objective evidence doesn’t render the outcome plainly obvious.

– If all practices are FI, then the goal must be satisfied.– If all practices are NI, the goal can’t be satisfied.– The conditions between the two extremes require

professional judgment.

Data Collection and Rating Concepts

• Corroboration– Must have direct artifacts, combined with either indirect

artifact or affirmation

• Coverage– Must have sufficient objective evidence for implementation

of each practice, for each instance– Must have face-to-face (F2F) affirmations (avoid “paper-only

appraisals”):

• At least one instance for each practice (“one column”)• At least one practice for each instance (“one row”)• Or 50% of practices for each PA goal, for each project, have at least

one F2F affirmation data point

Completion of Pre – Assessment Questionnaires - 1

• Site Information Questionnaire– The Site Information Questionnaire helps the

assessment team to rapidly understand the culture and language of the site's software practitioners and the context within which they operate

• Change Readiness Questionnaire – The Change Readiness Questionnaire assists an

organization in determining its readiness for change

Completion of Pre – Assessment Questionnaires - 2

• Organizational Structure Questionnaire– What is the real structure that gets things done? What is

the communication structure? What is the relationship with the clients? How are the requirements gathered? Are there multiple customers? Are the customers internal or external? Are there end users involved?

• The Project Questionnaire – The Project Questionnaire is intended to help collect

information for use in selecting projects or other work efforts. It helps the assessment team organize contextual background information.

Completion of Pre – Assessment Questionnaires - 3

• Documented Process Questionnaire– The Documented Process Questionnaire looks at all

of the documented policies, procedures, standards, guidelines, and templates as well as the artifacts that exist from the use of those defined processes and maps them to the requirements of the CMMI

Reviewing - 1

• Schedule 1-2 days to review documentation – entire assessment team is present

• Request that “process owners” or at least process lead developers present or “walk through” the specific process area processes

Reviewing - 2

• Assessment team members may ask the following questions during the presentation:

– Would you please clarify or expand on a point?

– Would you please jump to the referenced procedure, guideline, template, or checklist?

– Would you please show project examples that have followed that procedure or used that template?

– Why was this information placed in this document and not another one that is more closely aligned to the CMMI way of organization?

– Would you please print a hard copy of that section of the procedure or please print out the entire procedure?

Reviewing - 3

• Assessment team members can ask “interview-like” questions to gain more insight as to how these processes are actually being used on the projects

• Having “experts” describe their documented processes and other site assessment team members witnessing the answers eliminates or reduces the risk of long debates over the value of the documented processes in the later phases of the assessment

Consolidating - 1• Each assessment team member reviews his/her own

notes and makes observations

• These observations may make use of the sticky note format or may simply be placed on a sheet of paper with appropriate notations that will later be shared with the rest of the assessment team

• After each assessment team member has had a chance to develop individual observations, the Assessment Team Leader leads or facilitates the team in a brainstorming process to get the observation candidates out in front of the team

• A category is picked. The facilitator asks one person to offer a strength or weakness for that category

Consolidating - 2

• An observation is offered and the facilitator writes it on a flip chart or white board

• The facilitator asks the next person for an input:– This next person looks at what has already been placed on the

flip chart or white board and what observations they have written down for themselves

– If an observation already written on the flip chart or white board is close enough, they either pass to the next person or offer an observation for that category that they feel is different enough to be considered by their teammates

• The facilitator continues in a “Round Robin” fashion. (each person gets a turn to offer one input for a category and this process continues until everyone has run out of new ideas)

Consolidating - 3

• The assessment team then “takes a step back” and reviews all of the observations again that are in front of them

• Like observations may be further consolidated with the consensus of the assessment team