3/14/2006 USC-CSE 1

Ye Yang, Barry Boehm

Center for Software EngineeringUniversity of Southern California

COCOTS Risk Analyzer and Process Usage

Annual Research ReviewMar. 14th, 2006

3/14/2006 USC-CSE 2

Outline

• Motivation• COCOTS Model• COCOTS Risk Analyzer• Evaluation• Process Usage: Risk-Based Prioritization• Conclusions

3/14/2006 USC-CSE 3

Motivation

• Enable COTS integration risk analysis with COCOTS cost estimation inputs

• Identify relative risk levels of COTS-based development (CBD)

• Provide recommendations to improve risk management practices

3/14/2006 USC-CSE 4

COCOTS Model- Calibrated to 20 industry projects

3/14/2006 USC-CSE 5

COCOTS Glue Code Sub-modelCost

FactorsName Definition

Size Driver Glue Code Size The total amount of COTS glue code developed for thesystem.

Scale Factor

AAREN Application Architectural Engineering

ACIEP COTS Integrator Experience with ProductACIPC COTS Integrator Personnel CapabilityAXCIP Integrator Experience with COTS Integration ProcessesAPCON Integrator Personnel ContinuityACPMT COTS Product MaturityACSEW COTS Supplier Product Extension WillingnessAPCPX COTS Product Interface ComplexityACPPS COTS Supplier Product SupportACPTD COTS Supplier Provided Training and DocumentationACREL Constraints on Application System/Subsystem ReliabilityAACPX Application Interface ComplexityACPER Constraints on COTS Technical PerformanceASPRT Application System Portability

Effort

Multiplier

3/14/2006 USC-CSE 6

COCOTS Risk Analyzer

Knowledge Base

Knowledge Base

Risk Rules

Risk Level Scheme

Mitigation Strategy

User

5. Assess Overall Risk

6. Provide Risk Mitigation Advices

Input (Cost Factor Ratings)

Output (Risk Summary)

Knowledge Base

Knowledge Base

Risk Rules

Risk Level Scheme

Mitigation Strategy

User

2. Evaluate Risk

Probability

3. Analyze RiskSeverity

1. Identify risks of rating

combinations

4. Assess Overall Risk

5. Provide Risk Mitigation Advices

Input (Cost Factor Ratings)

Output (Risk Summary)

3/14/2006 USC-CSE 7

Knowledge Base

• Contents– Risk Rules (RR)– Risk level scheme– Common risk mitigation strategy

• Constructing approach– Expert Delphi Survey– Empirical study results– Literature review

3/14/2006 USC-CSE 8

Risk Rule

• A CBD risk situation – a combination of two cost attributes at their

extreme ratings

• Risk Rule (RR)– An identified risk situation is formulated as a risk

rule. E.g. one example RR:IF ((COTS Product Complexity > Nominal)

AND (Integrator’s Experience on COTS Product < Nominal))

THEN there is a project risk.

3/14/2006 USC-CSE 9

Risk Situation Identification

Total # of Delphi responses: 5

# of responses

% of responses

# of risk situations

>=3 >50% 24

2 40% 26

1 20% 28

24 Risk Rules formulated in the knowledge base

AS

PR

T

AC

PE

R

AA

CP

X

AC

RE

L

AC

PT

D

AC

PP

S

AP

CP

X

AC

SE

W

AC

PM

T

AP

CO

N

AX

CIP

AC

IPC

AC

IEP

AA

RE

N

SIZ

E

SIZEAARENACIEPACIPCAXCIPAPCONACPMTACSEWAPCPXACPPSACPTDACRELAACPXACPERASPRT

AS

PR

T

AC

PE

R

AA

CP

X

AC

RE

L

AC

PT

D

AC

PP

S

AP

CP

X

AC

SE

W

AC

PM

T

AP

CO

N

AX

CIP

AC

IPC

AC

IEP

AA

RE

N

SIZ

E

>=50% 40% 20%(Percentage of responses over total)

3/14/2006 USC-CSE 10

Risk Potential Rating for Cost Factors

Cost Factors Cost Factor Rating Risk Probability Rating

Very Low Worst CaseLow Risk Prone

Nominal ModerateHigh OK

Very High OKVery Low OK

Low OKNominal Moderate

High Risk ProneVery High Worst Case

AAREN, ACIEP, ACIPC, AXCIP,

APCON, ACPMT, ACSEW, ACPPS,

ACPTD

APCPX, ACREL, AACPX, ACPER,

ASPRT

Mapping between cost factor’s rating to its risk potential rating:

3/14/2006 USC-CSE 11

Risk Level Scheme

Worst Case Risk Prone Moderate OKWorst Case Severe Significant General

Attribute 2 Risk Prone Significant GeneralModerate GeneralOK

Attribute 1

Assignment of risk probability levels:

Risk level Quantifier

Severe 0.4

Significant 0.2

General 0.1

Quantitative weighting scheme:

3/14/2006 USC-CSE 12

Productivity Range

1.14

1.22

1.22

1.42

1.43

1.48

1.48

1.69

1.79

1.80

2.09

2.10

2.51

2.58

0.00 0.50 1.00 1.50 2.00 2.50 3.00

ASPRT

ACSEW

ACPER

AXCIP

ACPTD

ACREL

ACPPS

AACPX

ACIEP

APCPX

AAREN

ACPMT

APCON

ACIPC

Co

st F

acto

r

Productivity Range

• Reflects the cost consequence of risk occurring• Combines both expert judgment and industry data calibration

3/14/2006 USC-CSE 13

Project Risk Quantification

• Project Overall Risk:

– Riskprobij corresponds to the nonlinear relative probability of the risk occurring

– The product of PRi and PR j represents the cost consequence of the risk occurring

• Risk interpretation:– Normalized scale: 0 ~ 100– 100 represents the situation where each cost factor is

rated at its most expensive extremity– 0 ~ 5: low risk; 5 ~ 15: medium risk; 15 ~ 50: high risk; 50

~ 100: very high risk

3/14/2006 USC-CSE 14

Risk Mitigation Recommendations

• Knowledge base built on previous empirical study results, e.g.:

Risk Rule Risk Situation Mitigation Advice

APCPX_ACIPC

(High, Very Low)

Complex integration with inexperienced personnel

Consider more compatible COTS; re-staffing; training; consultant mentoring

ACREL_ACPMT

(High, Low)

High-reliability application dependent on immature COTS

Consider more mature COTS; reliability-enhancing COTS wrappers; risk-based testing

ACPER_AAREN (High, Very Low)

Unvalidated architecture with COTS performance shortfalls

Benchmark current and alternative COTS choices; reassess performance requirements vs. achievables

3/14/2006 USC-CSE 15

Evaluation Results

y = 0.6749x - 2.3975

R2 = 0.8948

0

5

10

15

20

25

30

35

40

45

0 10 20 30 40 50 60

Reported Risks

An

alyz

ed R

isks

Data: 9 USC e-services projects

y = 45.75x + 0.6143

R2 = 0.6283

0

5

10

15

20

25

30

35

40

45

50

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Reported Prob.(Risk)

An

alyz

ed R

isk

Data: 7 COCOTS calibration projects

USC e-services Industry

Domain

Web-based campus-wide e-services applications such as library services

Generally large scale comminication, control systems

# COTS 1 ~ 6 1 ~ 53Duration 24 weeks 1 ~ 56 monthsEffort 6 person by 24 weeks 1 ~ 1411 person-monthSize 0.2 ~ 10 KSLOC 0.1 ~ 390 KSLOC

3/14/2006 USC-CSE 16

Process Usage – An Example

• COTS A and B are our strongest COTS choices– But there is some chance that they have

incompatible HCI’s– Probability of loss P(L)

• COTS C is almost as good as B, and it is compatible with A

3/14/2006 USC-CSE 17

Risk-Driven CBD Process Framework

P1: Identify Objective, Constraints and

Priorities (OC&Ps)

P2: Do Relevant COTS Products Exist?

P3: Assess COTS Candidates

P4: Tailoring Required?

Single Full-COTS solution satisfies all OC&Ps

Yes or Unsure

P6: Can adjust OC&Ps?No

No acceptable or risky COTS-Based Solution

P5: Multiple COTS cover all OC&Ps?Partial COTS solution best

P7: Custom Development

NoYes

P10: Develop Glue Code

P8: Coordinate custom code and glue

code development

P9: Develop Custom Code

No, Custom codeRequired to satisfy

all OC&Ps

Yes

P11: Tailor COTSP12: Productize,

Test and Transition

NoYes

Deploy

Deploy

A

Process Area

Decision/Review

Assess-ment

Tailoring

Glue-Code

T

No

G

G

T

A

Start

Custom code

C

C

C

3/14/2006 USC-CSE 18

Different Risk Strategy Resulting in Different ProcessChoose COTS C

Integrate COTS A, C

Develop Application

Deliver(a) Risk Avoidance: COTS C adequate

Choose COTS B

Develop Application,

Integrate A & B

Develop Application

Deliver(b) Risk Transfer:

COTS C not adequate

OK

Use risk reserve to fix problem

Problem

Choose COTS B

Develop parts of application, use

wrappers to integrate A and B

Develop rest of application

Deliver(c) Risk Reduction:

Custom $, IP

(d) Risk Acceptance: Developer $, IP Package

wrappers for future use

3/14/2006 USC-CSE 19

Conclusions

• CBD brings a host of unique risk items• Many risk techniques/tools require intensive user

inputs• COCOTS Risk Analyzer provides a handy way to

automate the CBD risk analysis by leveraging on existing knowledge and expertise in both cost estimation and risk mgmt.

• Case study shows how it supports process decisions following the risk based prioritization strategy

3/14/2006 USC-CSE 20

Backup Slides

3/14/2006 USC-CSE 21

Risk Potential Rating

• Captures the underlying relation between cost attributes and the impact of their specific ratings on project risk – 4 Levels

• OK, Moderate, Risk Prone, and Worst Case

• Two types of treatments– Transforming continuous Size representation into

discrete risk potential ratings– Mapping cost driver ratings into risk potential

ratings

3/14/2006 USC-CSE 22

Risk Potential Rating for Size

Rating OK Moderate Risk Prone Worse CaseResponse 1 1 2 10 50Response 2 2 5 10 25Response 3 1 3 10 10Response 4 1 2 10 50Response 5 1 2 10 50

Median 1 2 10 50Stdev 0.447214 1.30384 0 18.5741756

Delphi Responses for Size Rating (Size in KSLOC):

3/14/2006 USC-CSE 23

Risk Based Prioritization Strategy

Risk Strategy

Step

Spiral Quadrants

CBD process Decision

Framework Step

Description

S1 Q1 P1, P2 Identify OC&Ps, COTS/other alternatives S2 Q2a P3 Evaluate COTS vs. OC&Ps (incl.

COCOTS) S3 Q2a P3 Identify risks, incl. COCOTS risk analysis S4 Q2b P3 Assess risks, resolution alternatives; If

risks manageable, go to S7 S5 Q2b, Q1 P6 Negotiate OC&P adjustments; If none

acceptable, drop COTS options (P7) S6 Q2a P3 If OC&P adjustments successful, go to

S7; If not, go to S5 S7 Q3 P4 or P5 Execute acceptable solution