COSOSIMO October 2005 Workshop Jo Ann Lane [email protected] University of Southern California Center...

COSOSIMO October 2005 Workshop

Jo Ann Lane

[email protected]

University of Southern California

Center for Software Engineering

COCOMO Forum – October 2005© USC CSE 2005

COSOSIMO Workshop Results © USC CSE 2005

COCOMO Forum 2005 2

Agenda

• COSOSIMO Workshop July 2005 Survey Results (~30 minutes)– Overview of results– How to interpret– Call for additional responses

• Using COSYSMO to estimate SoS LSI technical effort (~1 hour)– Tutorial Case Study– Call for data

• Break (~20 minutes)• SoS LSI management model ideas (~1 hour)

– Candidate ideas—Gary Constantine, Raytheon– Discussion

• SoS/FoS WBS ideas—Gan Wang (~1 hour)

System-of-Systems Cost Modeling: COSOSIMO July 2005 Workshop

Results

Jo Ann Lane

[email protected]





COCOMO Forum 2005 4

July 2005 Workshop Objectives and Summary

• Objectives– Clarify the LSI activities to be estimated by the COSOSIMO model– Begin converging on a relevant and complete set of parameters for

the COSOSIMO model that are easily discerned in the early stages of SoS development

• Summary of workshop activities– Attended by 20 people representing 12 organizations– Provided overview on research conducted to date on LSI activities– Discussed/updated list of key LSI activities– Conducted survey to better determine the differences between LSI

activities and more traditional SE activities– Discussed size drivers and factors that impact LSI effort– Conducted survey to identify relevant LSI size drivers and scale

factors


COCOMO Forum 2005 5

Workshop Recommendations and Results

• LSI Activities– Recommendation: LSI

activities should be broken out into the following categories

• Management activities• Technical activities

– Results: Updated list of LSI activities

• No initial activities or issues dropped from list

• Additional activities added to list

• Additional issues added to list

LSI Effort

ManagementEffort

TechnicalEffort

Discussions to continue in this area ….

Proposed scope for COSOSIMO if technical can be estimated using

COSYSMO SoS calibration

Investigate possibility of COSYSMO calibration

to estimate this part


COCOMO Forum 2005 6

Workshop Recommendations and Results (continued)

• Suggested additional activities– Configuration management– Common infrastructure

alignment/supply • Development common vocabulary

• Management processes

• Technical processes/tools

– Managing interoperability with external systems

– Verification and validation of the total system

• Development of SoS test bed/integration lab

• SoS level requirements in addition to oversight of lower level V&V

• Suggested additional activities (continued)

– Transition plans– Ensure communications between

various SoS orgs– Provide logistics, support centers, other

-ilities– Security approach (developmental and

operational)– Safety plans– Training – Post implementation communications– Disaster recovery– Tradeoffs on level of service reqs – Development of SoS infrastructure?


Activities to be analyzed with respect to size and cost drivers….


COCOMO Forum 2005 7

Workshop Results (continued)

• Suggested additional issues that may impact LSI effort– Conflict of interest– Sharing of proprietary info– Import/export concerns with international teams– Subcontractor process maturity– Supplier stabilization and synchronization– Diversity of supplier processes, methods, and tools– Synchronization of tools– Level of component independence


Issues to be analyzed with respect to size and cost drivers….


COCOMO Forum 2005 8

COSOSIMO Survey Overviews

• LSI activity survey – Description

• Typical LSI effort was compared to the typical systems engineering effort defined in COSYSMO

• Comparison was done for the 33 activities defined in EIA632

– 12 responses received to date

• Size Driver and Scale Factor Survey– Description

• Survey contained a list of current COSYSMO size drivers and scale factors as well as the proposed COSOSIMO size drivers and scale factors

• Respondents were asked to indicate which parameters applied to LSI management activities (by entering an M) and which parameters applied to LSI technical activities (by entering a T)

– 12 responses received to date


COCOMO Forum 2005 9

Survey Results: LSI Activities

• General observations– Number of EIA/632 activities

evaluated: 33

– Highest average rating: 0.67 Lowest average rating: 0.00

– Number of activities with average rating 0.5 or higher: 15

– Number of activities with average rating between 0.3 and 0.5: 9

– Number of activities with average rating 0.3 or lower: 9

-1 (less) 0 (same) 1 (more)

Survey Scale:

High46%

Medium27%

Low27%


COCOMO Forum 2005 10

Survey Results: LSI Activities (continued)

• Activities with average rating 0.5 or higher– Product supply – Product acquisition– Supplier performance – Process implementation strategy– Technical effort definition – Outcomes management– Information dissemination – Other stakeholder requirements– Transition to use – Effectiveness analysis– Risk analysis – Requirements statements validation– Logical solution validation – End product verification– End product validation

• Activities with average rating between 0.3 and 0.5– Schedule and organization – Progress against plans and schedules– Technical reviews – System technical requirements– Implementation – Tradeoff analysis– Acquirer requirements validation – Other stakeholder requirements validation– Enabling product readiness

• Activities with average rating 0.3 or lower– Technical plans – Work directives– Progress against requirements – Acquirer requirements– Design: Logical solutions – Design: Physical solutions– Design: Specified requirements – System technical requirements validation– Design solution verification



Survey Results: LSI Activities (continued)

• LSI activity survey conclusions (so far)– All EIA 632 activities are applicable to the LSI effort (there

were only two responses that indicated a specific activity was not applicable)

– Most EIA 632 activities (75%) require the same or more effort in the LSI environment than in the more traditional SE projects

– No survey identified additional activities to be included in this list – however, during workshop discussions, others were identified

Interpretation: the other activities identified during discussions were either a clarification of the emphasis of the activity or a set of lower level activities



Survey Results: Size Drivers and Scale Factors

• Management Size Drivers– Average response “high”: # of independent system

component orgs

– Average response “medium: # of component systems

– Average response “low: # system requirements

# system interfaces

# operational scenarios

Subsystem software size

# SoS interface protocols

– Average response “N/A”: # algorithms

Survey Rating Scale: High Applicability. Medium Applicability, Low Applicability, or Not Applicable (N/A)



Survey Results: Size Drivers and Scale Factors (continued)

• Technical Size Drivers– Average response “high”: # system interfaces

# operational scenarios

– Average response “medium”: # system requirements

# of component systems

# SoS interface protocols

– Average response “low”: # algorithms

Subsystem software size

# of independent system component orgs

– Average response “N/A”: (none)





• Management Scale Factors– Average response “high”: Stakeholder team cohesion

Multi-site coordination Cost/schedule compression

– Average response “medium”: Requirements understandingArchitecture understandingMigration complexityTechnology risk# and diversity of installations/platformsPersonnel/team capabilityPersonnel experience/continuityProcess capabilityTool supportIntegration risk resolutionIntegration stabilityComponent readinessComponent system maturity/stability

Survey Rating Scale: High Applicability Medium Applicability Low Applicability Not Applicable (N/A)




• Management Scale Factors– Average response “low”: Level of service requirements

Documentation

Recursive levels in the design

Integration simplicity

– Average response “N/A”: (none)





• Technical Scale Factors– Average response “high”: Requirements understanding

Personnel/team capability

Personnel experience/continuity

– Average response “medium”: everything else….





• Size driver survey conclusions (so far)– Management Size Drivers: Most important appear to be key drivers

currently defined for COSOSIMO– Technical Size Drivers:

• Most of the high and medium applicability drivers are those currently defined for COSYSMO (not COSOSIMO)

• Exceptions– COSOSIMO “# of system components” included under medium applicability – COSOSIMO “# of SoS level interfaces” included under medium applicability

(Possible interpretation: respondents may have thought this related to COSYSMO “# of system interfaces”)

– COSYSMO “# of algorithms” thought to be low applicability

– Rated low in both categories• # of algorithms• Sub-system software size

– No significant suggestions for additional size drivers




• Scale factor survey conclusions (so far)– Management scale factors:

• Most (80%) of the COSYSMO and COSOSIMO scale factors were thought, on average, to be of high or medium applicability

• None of the listed scale factors had an average rating of “not applicable

– Technical scale factors:• ALL of the proposed scale factors were thought, on average, to be of

either high or medium applicability

• No significant suggestions for additional scale factors• Questions to consider

– If COSYSMO is to be used to estimate LSI technical effort, are additional scale factors required?

– Or is there embedded overlap in the current set of scale factors?– Is there a better, minimal set of scale factors that are sufficient for

both COSYSMO and/or COSOSIMO?



Going Forward

• Continue to – Collect survey data– Identify sources of actual SoS LSI effort data

• Compare COSYSMO outputs with COSOSIMO outputs for selected SoS programs

• Topics for discussion at this week’s workshop– Impact of July 205 survey results on current COSOSIMO model

• LSI activities• Size drivers and scale factors

– Using COSYSMO to estimate LSI technical effort– Management model ideas

• Overview of some management model parameters• Sufficient for LSI management effort?

– SoS/FoS WBS ideas

For the most current information on COSOSIMO, seehttp://sunset.usc.edu/cse/pub/research/COSOSIMO/index.html

New Processes and Estimation Methods

for Acquiring 21st Century Software-Intensive Systems of

Systems ******************

Case Study

Barry Boehm [email protected]

Jo Ann Lane [email protected]

Winsor Brown [email protected]






Metropolitan Area Crisis Management System (MACMS)

Net - Centric SoS Net - Centric SoS



MACMS Overview• Objective: To integrate and share information between the various crisis

management organizations within a metropolitan area. • Integrated organizations include

– 911 system– Police and Sheriff’s Departments– Fire– Jail– Ambulances– City and county planning (provide information on roads, property boundaries, property

owners, etc.)– Hazardous materials management– Biological hazard containment– Bomb squad– Hospitals– Local news organizations (with live video feeds that can be used by people on the ground)– Community warning systems– Related State and Federal agencies (e.g., State Highway Patrol, State Wants and Warrants,

FBI, Border Patrol, Coast Guard)



Key Features

• Ability to coordinate responses to metropolitan area crises– Classify type of crisis– Alert appropriate organizations– Alert/evacuate public– Identify and manage needed resources

• Fire trucks• Airplanes• Helicopters• Robots/remotely controlled vehicles• Medical supplies/special treatment or isolation facilities

• Request and coordinate support from other agencies: state, Federal, or other regional areas

• Support crisis management activities in other regions



MACMS Issues and Risks

• Incompatible interfaces between existing systems• COTS products available to support interconnectivity, but have not

been used at this level (potential scalability issues)• Police and fire departments currently have on-going projects to

integrate the police, fire department, and 911 systems• Limited local budgets to modify other existing systems• Little or no modifications expected for related State and Federal

systems (potential impacts with interfaces to other metropolitan area systems)

• Federal funds available if system implemented within the next 5 years

• Both County Board of Supervisors and City Council need to approve plans and budgets

• Citizen privacy and security issues



Discussion Topics

• Who should be the LSI? County? City? Contractor? • Shared vision• Feature priorities• Candidate architecture(s) and increment definitions: What can

be defined as “independent projects”? How does this impact cost and schedule?

• Candidates for critical path• Risk management: What key risks should be addressed first?• Where to be agile? Where to be plan-driven?• Oversight and management: What to control with

vendors/suppliers? What to track (suppliers and collaborating organizations)?



SISOS Schedule Estimation:A Composite Approach

Customer,Users

LSI – Agile

LSI IPTs – Agile

Suppliers – Agile

Suppliers – PD – V&V

LSI – Integrators

RFP, SOW, Evaluations, Contracting

Effort/Staff

Proposals

Similar, withadded change

traffic fromusers…

Ass

ess

co

mpati

bili

ty,

short

-fa

lls

Rew

ork

LC

O

LC

APack

ages

at

all

levels

COSOSIMO-like

Assess sources of change;

Negotiate rebaselined

LCA2 package at all levelsCOSOSIMO

-like

Similar, withadded re-

baselineing risks and rework…

InceptionElaboration

Source SoSSelection

Architecting

Increment 1 Increments 2,… n

Develop to spec, V&VCORADMO

-like

Degree of Completene

ss

risks, rework

Proposal Feasibility

LCO LCA

LCA1

IOC1

Effort/staffat all levels

risks, rework

Risk-manage slow-

performer, completeness

risks, rework

Integrate

COSOSIMO-like

LCA2 shortfalls

risks, rework

Effort COSYSMO-like.

Schedule = Effort/Staff

Try to model ideal staff size

LCA2



SISOS Effort and Schedule Estimation Exercise

• Notes and assumptions:– To handle the unavailability of a calibrated COSOSIMO

model• Will use COSYSMO to estimate LSI technical effort• Will determine an additional % to address LSI management

effort

• Exercise– Using guidelines in tutorial, estimate Inception effort and

schedule– Using guidelines in tutorial, estimate Elaboration effort and

schedule– Outline approach for estimating Construction and Transition

effort and schedule



MACMS SISOS Characteristics

• Current size information– Key features documented in a requirements specification that contains approximately

(TBD) functional and performance requirements– To date, about (TBD) system interfaces have been identified, about half of which are

documented in some sort of Interface Requirements Specification– During the initial requirements specification process, about (TBD) top level use

cases/scenarios were developed• SISOS Approach

– Integrate existing legacy systems together using a net-centric architecture that includes wireless, mobile networks for mobile units and existing networks for fixed control center connectivity

– As part of this effort, the city and county planning and land use organizations would like to replace their location tracking systems with a new system that is based on city/county records and not the more general purpose map programs/ databases typically provide by Geographic Information System (GIS) vendors

– No other new system components planned for the early versions of this SoS– Build on existing connectivity

• Some sort of connectivity exists between– City police, sheriff’s, 911, and ambulance systems– Jail information system and state and Federal agencies

• Most other system components are relatively closed, independent systems



Notes from Interactive Exercise:Architecture (System of Interest)

View

Systemelement

System-of-interest

Systemelement

Systemelement

SystemelementSystem

Systemelement

Systemelement

Systemelement

System

Systemelement

Systemelement

Systemelement

System

Systemelement

Systemelement

SystemelementSystem

Systemelement

Systemelement

System

Systemelement

Systemelement

System

Systemelement

Systemelement System

Systemelement

Systemelement

Systemelement

Source: ISO/IEC 15288.Source: ISO/IEC 15288.

Goal: Describe MACMS System of Interest for estimation purposes…



Notes from Interactive Exercise:MACMS Architecture View



Notes from Interactive Exercise:

Assumptions




COSYSMO Parameters

Size drivers– # requirements: – # interfaces: – # algorithms: – # operational scenarios:

Scale factors– Requirements Understanding:– Architecture Understanding: – Level of Service Requirements: – Migration Complexity: – Technology Risk: – Documentation: – # and diversity of

installations/platforms: – # of recursive levels in the design: – Stakeholder team cohesion: – Personnel/team capability: – Personnel experience/continuity: – Process capability: – Multi-site coordination: – Tool support:



• COSYSMO Conceptual Phase Effort = 23%– 7% Inception; 16% Elaboration

• Inception Phase Effort = (total effort) * .07 = ____ PM• Inception Phase Schedule

– Months = 1.5 * cube root (inception effort) = ___ months– Average staff size = effort/months = ____ people


Schedule Estimation




Schedule Estimation (continued)• Sum of schedules for systems engineering, source selection, and post-selection

rebaselining• COSYSMO Elaboration effort = (total effort) * .16 = _____ PM• Systems engr. schedule = 1.5 * cube root (elab effort) = 19 months

– Average staff size = elab effort/months = ___ people

• Source selection schedule– Preparation in parallel: no added schedule– RFP finalization and publication: ___ month– Proposal responses: ___ months

• Including prototypes, architecture, Feasibility Rationale

– Parallel evaluation, finalist selection: ___ months– Finalist compatibility/feasibility, Q&A: ___ months– Contracting: ___month– Total schedule: ___ months

• Teambuilding, LCA rebaselining: ___ months• Total Elaboration Phase schedule: ___ months



Summary of Estimation Results

COSOSIMO October 2005 Workshop Jo Ann Lane [email protected] University of Southern California Center...

Documents

Transcript of COSOSIMO October 2005 Workshop Jo Ann Lane [email protected] University of Southern California Center...