Expanding Interpretive Structural Modeling

to Develop Materiel and Non-Materiel

Solution Requirements

NDIA 15th Annual Systems Engineering Conference

“Technology Maturity”

October 22-25, 2011

Justin M. Rettaliata, Thomas A. Mazzuchi, and Shahram Sarkani

Dissertation Topic

Department of Engineering Management and Systems Engineering

School of Engineering and Applied Science

The George Washington University

1176 G Street NW

Washington, DC 20052

Requirements Reform Initiatives

• According to Hanks, et al. (2005) in their book Reexamining Acquisition Reform: Are We There Yet?

– Of 63 reform initiatives evaluated from 1989 to 2002, 28 (or 44%) of these initiatives centered on requirements determination

– Proof that this has been and still is a very challenging part of the acquisition process

• Commercial Sourcing: FAR Part 12 Procurements – June 1995

• Elimination of Mil Specs and Mil Standards – Feb 1994

• Single Process Initiative (SPI) – June 1994

• Best Value Contracting – March 1996

• Multi-Year Contracting – Feb 1994

• Other Transaction Authority – Dec 1996

• Performance-Based Service Acquisition – Sep 1991

• Open Systems Approach – Nov 1994

• Advanced Concept Technology Demonstration (ACTD) – 1994

• Concurrent Developmental/Operational Testing – Nov 1994

• Contractor-Maintained Design Configuration – Mar 1994

• Rapid Prototyping for Software Development – Nov 1994

• Simulation-Based Acquisition – Mar 1996

• Streamlined ECP Review/Approval – Feb 1995

• Survivability/Lethality Below End-Item Level – Jun 1995

• Logistics Transformation – 1998

• Alpha Contracting – Oct 1997

• Improved Pre-Solicitation Phase Communication – Jan 1993

• Oral Presentations – 1999

• RFP Streamlining – Mar 1995

• Contractor Total system Performance Responsibility (TSPR) – 1998

• Cost as an Independent Variable (CAIV) – Dec 1995

• Evolutionary Acquisition – June 1996

• Integrated Product and Process Development (IPPD) – Feb 1994

• Joint Govn’t/Industry IPTs – OCT 1995

• Program Stability – Feb 1994

• Reduction in Total Ownership Cost (RTOC) – April 1998

• Modernization Through Spares – 1997

Hanks, C.H., Axelband, E.I., & Linsay, S., (2005). Reexamining Military Acquisition Reform: Are We There Yet?, Santa Monica, California, RAND Corporation

Recent GAO Reports

• GAO-08-294 (2008) – BEST PRACTICES: Increased Focus on Requirements and Oversight Needed to Improve DOD’s Acquisition Environment and Weapon System Quality

– “…DOD and its contractors often enter into development contracts before requirements have

been analyzed with disciplined systems engineering practices. (pg 4)”

• GAO-03-057 (2003) – BEST PRACTICES: Setting Requirements Differently Could Reduce Weapon Systems’ Total Ownership Costs

– “Requirements-setting by the war-fighting community focuses on system performance. DOD policy does not require inclusion of readiness or operating and support costs goals as key performance requirements equal in importance to other performance requirements. (pg 5)”

– “…85 percent of the operating and support costs of a weapon system will be determined as soon as requirements are set, while less than 10 percent of the life-cycle cost have been spent. (pg14)”

– “…there is no direct relationship between the requirement setters and the product developer. (pg11)”

Lack of detailed requirements analysis causes significant issues: schedule delays,

cost growth and even program cancellation

U.S. Government Accountability Office, (2008). Best Practices: Increased Focus on Requirements and Oversight Needed to Improve DOD’s Acquisition Environment and Weapon System Quality.

Retrieved June 10, 2012, from http://www.gao.gov/products/GAO-08-294

U.S. Government Accountability Office, (2003). Best Practices: Setting Requirements Differently Could Reduce Weapon Systems’ Total Ownership Coasts (Publication No.GAO-03-57).

Retrieved June 18, 2012, from http://www.gao.gov/products/GAO-03-57

Abstract of Research

Problem:

• The US Government does a poor job in properly writing requirements for their acquisition programs

• Poorly written requirements result in:

• Higher costs throughout the life cycle of program, constant schedule slips, or at the worst – program failure/cancellation

• Issues with Supportability, Testability, Ambiguity, and False Interpretations

• Lack of Guidance in DoD for structuring requirements for Materiel and Non-Materiel Solutions

• Requirements is the basis for a program, including technology readiness, design, costs, supportability, etc.

Research:

• MCDM methods provide decision makers with the ability to look into the future, and make the best decision based on past and present information and future predictions

• Adequately define key differences between a Materiel and Non-Materiel Solution

• This study will create a case study to demonstrate using a MCDM method and Davis’ 13 Attributes of a well written requirement to determine critical attributes for Materiel & Non Materiel solutions

Provide methods of properly writing/selecting requirements

Methods of Determining Requirements

Example of what’s taught in DAU PMT 257 Acquisition Class and Requirements 101:

• Affinity Diagrams

• Force Field Analysis

• Ishikawa Diagrams

DAU Curriculum does not adequately train how to adequately determine requirements

• Monte Carlo Simulations

• Pareto Analysis

Data Analysis for MCDM

Example MCDM Methods available

• Interpretive Structural Model (ISM)

• Robust Portfolio Modeling (RPM)

• Value Analysis/Value Engineering (VA/VE)

• Quality Function Deployment (QFD)

QFD House of Quality Template

House of Quality Result Example (Crow, 2002)

Value Study Process Flow Diagram

Thakkar, J., Deshmukh, S.G., Gupta, A.D., & Shankar, R., (2006). Development of a Balanced Scorecard: An Integrated Approach of Interpretive Structural Modeling (ISM) and Analytic Network

Process (ANP), International Journal of Productivity and Performance Management, Vol. 56, No. 1, 25-59

Cheah, C.Y.J. & Ting, S.K., (2005). Appraisal of Value Engineering in Construction in Southeast Asia, International Journal of Project management, Vol. 23, 151-158

Liesio, J., Mild, P., Salo, A.A., (2007). Preference Programming for Robust Portfolio Modeling and Project Selection, European Journal of Operational Research, Vol. 181, 1488–1505

Crow, K., (2002). QFD & Target Costing Case Study, DRM Associates, Retrieved Sept 30, 2012, from http://www.npd-solutions.com/qrtncasestudy.htm

Interpretive Structural Model

• Show the interrelationship of different criteria and their levels of importance.

• Aids in categorizing criteria depending on their driver power and dependence.

• Imposes order and direction on complex relationships among elements of system

• Provide a group learning process

• Identify Drivers and dependent issues while separating linkage and independent aspects

Structural Self-Interaction Matrix (SSIM)

Driver Power Dependence Matrix

Example ISM Digraph

Solicit Expert Judgment

Materiel Solution

Requirements

Non Materiel Solution

Requirements

System

Requirements

Solution Requirements

Utilize Interpretive Structural Modeling (ISM) (Thakkar et al.) to obtain expert judgment to develop attributes for Materiel and Non

Materiel Solutions

Davis’ 13 Attributes of a Well Written Requirement

• Correct – Represents something required of the system

• Unambiguous – can have only one interpretation

• Complete – All aspects are accounted for, and no gaps are left

• Verifiable – Finite way of determining if requirement meets intended purpose

• Consistent – Does not conflict or present inconsistency other requirements

• Understandable by customer – Explainable by non-SMEs

• Modifiable – Structure allows alterations to be made without compromising intent

• Traced – Origin is clear

• Traceable – Process can be followed back to origin

• Design Independent – Does not rely on a single architecture

• Annotated – Rank in order to determine importance to the end user

• Concise – Short and to the point

• Organized – Structured to be easily found

Davis, A.M., (1993). Software Requirements, Upper Saddle River, New Jersey, Pearson Prentice Hall, Inc.

Materiel and Non-Materiel Solutions

Per the Defense Acquisition University:

• Materiel Solution: Correction of a deficiency, satisfaction of a capability gap, or

incorporation of new technology that results in the development, acquisition, procurement,

or fielding of a new item, including ships, tanks, self-propelled weapons, aircraft, etc., and

related software, spares, repair parts, and support equipment, but excluding real property,

installations, and utilities, necessary to equip, operate, maintain, and support military activities

without disruption as to their application for administrative or combat purposes. In the case of

family of systems or systems of systems approaches, an individual materiel solution may not

fully satisfy a necessary capability gap on its own. (CJCSI 3170.01G)

• Non-Materiel Solution: Changes doctrine, organization, training, materiel, leadership

and education, personnel, facilities, or policy (including all human systems integration

domains) to satisfy identified functional capabilities. The materiel portion is restricted to

commercial or non-developmental items, which may be purchased commercially, or by

purchasing more systems from an existing materiel program.

Case Study Presentation

• Utilize Davis’ 13 Attributes to develop requirements for Materiel and Non-Materiel Solutions

• Solicit Expert Systems Engineers for their judgment on the applicability of Attributes to Materiel and Non-Materiel Solutions

Define Attributes for

Materiel/Non-Materiel

Solutions

Solicit Expert

Judgment Develop SSIM

Develop Reachability

Matrix

Driver Power

Dependence Matrix Develop Digraph Formulation of ISM –

System Requirements

Thakkar, J., Deshmukh, S.G., Gupta, A.D., & Shankar, R., (2006). Development of a Balanced Scorecard: An Integrated Approach of Interpretive Structural Modeling (ISM) and Analytic Network

Process (ANP), International Journal of Productivity and Performance Management, Vol. 56, No. 1, 25-59

Case Study Presentation

Determine relationships between Attributes

V – Relation from i to j, but not in both directions

A – relation from j to i, but not in both directions

X – relation from i to j and j to I

O – If relation between variables does no appear valid

Define Attributes for

Materiel/Non-Materiel

Solutions

Solicit Expert

Judgment Develop SSIM

Develop Reachability

Matrix

Driver Power

Dependence Matrix Develop Digraph Formulation of ISM –

System Requirements

Structural Self-Interaction Matrix (SSIM)

Thakkar, J., Deshmukh, S.G., Gupta, A.D., & Shankar, R., (2006). Development of a Balanced Scorecard: An Integrated Approach of Interpretive Structural Modeling (ISM) and Analytic Network

Process (ANP), International Journal of Productivity and Performance Management, Vol. 56, No. 1, 25-59

Case Study Presentation

• From SSIM, develop the Final Reachability Matrix

• Final Reachability Matrix enables the development of Driver Power Dependence Matrix

– Graphically determines which requirements are drivers, dependent, self sufficient and linked.

Define Attributes for

Materiel/Non-Materiel

Solutions

Solicit Expert

Judgment Develop SSIM

Develop Reachability

Matrix

Driver Power

Dependence Matrix Develop Digraph Formulation of ISM –

System Requirements

Driver Power Dependence Matrix Final Reachability Matrix

Thakkar, J., Deshmukh, S.G., Gupta, A.D., & Shankar, R., (2006). Development of a Balanced Scorecard: An Integrated Approach of Interpretive Structural Modeling (ISM) and Analytic Network

Process (ANP), International Journal of Productivity and Performance Management, Vol. 56, No. 1, 25-59

Case Study Presentation

• Determines the highest driving requirement/attribute

• Depicts how each requirement is linked in order of its driving power and dependence

• Will assist DoD in determining most important attributes to base requirements

Define Attributes for

Materiel/Non-Materiel

Solutions

Solicit Expert

Judgment Develop SSIM

Develop Reachability

Matrix

Driver Power

Dependence Matrix Develop Digraph Formulation of ISM –

System Requirements

Digraph and Formulation of ISM

Thakkar, J., Deshmukh, S.G., Gupta, A.D., & Shankar, R., (2006). Development of a Balanced Scorecard: An Integrated Approach of Interpretive Structural Modeling (ISM) and Analytic Network

Process (ANP), International Journal of Productivity and Performance Management, Vol. 56, No. 1, 25-59

Summary

• DoD does a poor job clearly writing/defining its requirements for acquisition programs, leading to issues with supportability, testability and false interpretations

• Initial research demonstrated critical differences in how DoD develops requirements and how Industry develops requirements, resulting in higher operational costs and supportability issues

• No standard methodology for developing requirements for Materiel and non Materiel solutions

• Goal is to research how government writes its requirement

• Propose a method of determining requirements by utilizing Interpretive Structural Modeling with Davis’ 13 Attributes in a case study to develop Attributes for Materiel and Non-Materiel Solutions to aid DoD in developing requirements