PROTOTYPING BENEFITS in Systems Acquisition Article 2 - 71-774... · Production Readiness Review;...

Image designed by Michael Krukowski

In 2007, John Young, then-Under Secretary of Defense for Acquisition, Technology and Logistics, mandated the use of “competitive prototyping” strategies in defense acquisition. Further, Department of Defense Instruction 5000.02 includes considerations for prototyping in the acquisition strategy. A 2007 memorandum circulated by Young lists five prototyping benefits, which are expected to “reduce technical risk, validate designs, validate cost estimates, evaluate manufacturing processes, and refine requirements.” However, a process to assess whether, and to what extent, a prototype will be or has been successful in achieving these benefits is not currently in use by the Department of Defense. Because cost increases and schedule extension downsides are inherent in prototyping, such an assessment is critical. This research proposes an approach for assessing the likelihood of achieving expected prototyping benefits based on identifying the factors yielding these benefits as well as their relative weights.

DOI: https://doi.org/10.22594/dau.17-774.24.04 Keywords: Program Success, Prototype, Prototyping, Defense Acquisition,Systems Engineering

ASSESSING THE LIKELIHOOD OF

ACHIEVINGPROTOTYPING BENEFITSin Systems Acquisition

Maroun Medlej, Steven M. F. Stuban, and Jason R. Dever

628 629Defense ARJ, October 2017, Vol. 24 No. 4 : 626-655 Defense ARJ, October 2017, Vol. 24 No. 4 : 626-655

Assessing the Likelihood of Achieving Prototyping Benefits http://www.dau.mil October 2017

In an era of budget stringency, the Department of Defense (DoD) multi-year acquisition process is too costly and lengthy for developing weapon systems (Coble, Royster, Glandon, Stewart, Pham, & Taylor, 2014). The inability to minimize technology risks before entering the system’s development phase contributed to a 13 percent cost growth in systems acquisition and a 17 percent schedule increase between 2007 and 2012 (Copeland, Holzer, Eveleigh, & Sarkani, 2015). Competitive prototyping (CP) was hailed as a solution to acquisition risks. It has been recommended

and encouraged in major DoD systems acquisition (Borowski, 2012). In CP, two or more teams compete to develop prototypes

during the acquisition phase, in which the competing pro-totypes are compared and the one that best addresses

issues, problems, or challenges is chosen (MITRE, 2016). By ensuring competition, risks are reduced

and more innovative solutions and better value can be attained.

In 2007, John Young, then-Under Secretary of Defense for Acquisition, Technology and

Logistics, mandated the use of “competitive prototyping” strategies in defense acquisi-

tion (Young, 2007). Further, Department of Defense Instruction 5000.02 includes

considerations for prototyping in the acquisition strateg y (DoD, 2017). Figure 1 shows the DoD Acquisition

Process phases and milestones.

In 2009, then-President Ba rack Oba ma approved the Weapon Systems Acquisition Reform Act (WSAR A). Section 203 of the

WSAR A required that each Major Defense Acquisition Program (MDAP) provide compet-

itive prototypes (WSARA, 2009). However, the WSARA allowed the Milestone Decision Authority to

waive the CP requirement if the cost of producing the pro-totypes exceeded the expected CP benefits in its systems engineering life cycle. Superseding the WSARA, the more recent National Defense Authorization Act (NDAA) for Fiscal Year 2016 no longer mandates CP, but continues to require the consideration of CP as a risk management

method in the acquisition strategy (NDAA, 2015).

FIGURE 1. THE DOD ACQUISITION PROCESS

Legend = Decision Point = Milestone Decision = Major Review

A B CMaterielSolutionAnalysis

MaterielDevelopmentDecision

MDD

Technology Maturation &Risk Reduction

CDD

RFP

PDR

Dev RFP ReleaseDecision Point

Engineering & ManufacturingDevelopment

CDR PDR

Production & Deployment

FRPLRIPFull RateProductionDecision

Operational Test &Evaluation (OT&E) IOC

Operation &Support

FOC

Pre-Systems Acquisitions Systems Acquisitions Sustainment

Dis

posa

l

Note. (Acquisition Process Overview, 2017). CDD = Capability Development Document; Dev = Developmental; FOC = Full Operational Capability; IOC = Initial Operational Capability; LRIP = Low-Rate Initial Production; PDR = Preliminary Design Review; PRR = Production Readiness Review; RFP = Request for Proposal.

Young’s memo lists five benefits a prototype is expected to provide (Table 1). These are the main drivers of CP and prototyping considerations in defense acquisition, except when costs exceed benefits. While it seems evident that prototyping is deemed critical to reducing overall program risk, the basis for this determination is far less clear. Although the DoD acquisition practice appears to have accepted as fact that a prototype can yield these benefits, not all agree. Those who demur note that adding a step inherently increases program cost, schedule, and time to delivery. As Coble et al. (2014) argue, while “the Technology Development System offers an effective way to dis-cover and reduce risk, the process will raise costs” (p. 146). And even if test results of delivered prototypes could demonstrate achievability, they might not be affordable (Pflanz, Yunker, Wehrli, & Edwards, 2012).

TABLE 1. FIVE DOD-STATED BENEFITS EXPECTED OF A DOD PROTOTYPE

B1 - Reduction in Technical Risk

B2 - Validation of Designs

B3 - Evaluation of Manufacturing Processes

B4 - Validation of Cost Estimates

B5 - Refining of Requirements

Accordingly, in the absence of a consistently replicable method to assess a prototype’s likelihood of achieving its benefits, the additional cost and schedule increases make it difficult to determine whether, and to what degree, a prototype has been or would be successful. Because achieving prototyping benefits is key to success, this article presents research results



on the factors that contribute to such benefits. Knowing the factors for achieving benefits provides additional levels of measurement and allows improved probability assessment for successful prototyping. This article and its accompanying research into prototypes as a strategy to mitigate risk includes a proof of feasibility and a plausibility test to assess a prototype’s likelihood of success—one that embodies DoD defense acquisition experts’ judgments. The proof of feasibility is described by developing a proto-typical solution based on solid scientific principles and motivated by a survey of selected experts in the domain. The technique is similar in concept to the U.S. Army, Navy, and Air Force Probability of Program Success (PoPS) approach shown in Figure 2. PoPS uses Level 1 factors and Level 2 metrics for each Level 1 factor (U.S. Air Force, 2007). The proposed method can also be customized or tailored to perform other assessments.

FIGURE 2. U.S. AIR FORCE: PROBABILITY OF PROGRAMSUCCESS (POPS) SUMMARY

PEO: Program NameACATXX

Program Success(100)

Pre-Milestone BDate of Review: Date PM: PM’s Name

ProgramRequirements (xx/25)

Program ParameterStatus (10)

Program ScopeEvolution (15)

ProgramResources (xx/16)

ProgramExecution (xx/24)

Budget (12)

Manning (2)

Contractor Health (2)

Cost/SchedulePerformance (2)

Contractor/DeveloperPerformance (2)

Fixed PricePerformance (2)

Contractor

Program RiskAssessment (6)

Sustainability RiskAssessment (3)

Testing Status (2)

Technical Maturity (7)

Software (Not used forPre-Ms B Evaluations)

Rebaselines: (X)Last Rebaseline: DATEProgram Life Cycle Phase: XXXXXXX

LEGENDSColors:G: On Track, No/Minor IssuesY: On Track, Significant IssuesR: O� Track, Major IssuesGray: Not Related/Not Applicable

Asterisk carried on metric to indicaterebaselined

Trends:Up Arrow: Situation Improving(number): Risk Score (based on 100 possible)Down Arrow: Situation Deteriorating

DoD Vision (7.5)

Program “Fit”Capability Vision (xx/15)

ProgramAdvocacy (xx/20)

Warfighter (6)

Congress (4)

OSD (2) or (3)

Joint Sta� (2)

HQ Air Force (2)

Industry (2) or (3)

International (2) or (0)

Air Force Vision (7.5)

Note. (U.S. Air Force, 2007, p. 60)

Research Competition can harness better value and might trigger additional

tradeoffs (Borowski, 2012), but the CP strategy “should focus on mitigating key technical risk areas” (Defense Acquisition University, 2017, Chap. 3). The focus of this research is the use of prototypes as a strategy to mitigate acquisition risk. The three objectives are: (a) identify and verify the factors required for prototyping benefits, (b) derive the weight of each factor, and (c) construct a PoPS-like approach to evaluate a prototype’s likelihood of success. The goal is to decompose the five prototyping benefits into the factors that are most important for achieving these benefits. Factors are verified based on expert judgments, then ranked according to their level—or weight—of contribution to the achievement of each prototyping benefit. The resulting model for assessing a prototype’s likelihood of success entails straightforward input and computation of defense acquisition professionals’ assessments of probable outcomes.



This research does not provide a true probabilistic assessment model, but does demonstrate an approach that indicates the likelihood of achieving prototyping benefits. The proposed approach is not offered as an author-itative assessment instrument, but rather as a flexible method to provide guidance in the assessment. In this article, the prototyping benefits stated in Young’s (2007) memorandum are referred to as the “DoD-stated benefits” or simply “prototyping benefits.”

Research LimitationsThe list of prototyping benefit factors compiled in this research is exten-

sive, but unlikely to be 100 percent complete because of the size, complexity, and uniqueness of defense acquisition programs. Available information is further restricted when dealing with classified programs. Also, defense acquisition is a specialized knowledge area, which limited the number of experts and the availability of real data for test-running the research model. Another limitation is the fact that survey participants represent only a small portion of the entire DoD acquisition workforce. Nonetheless, according to survey responses and comments, the selected factors appear to apply in most acquisition scenarios.

Prototyping in AcquisitionStudies by the RAND Corporation, the National Defense Research

Institute, and the Defense Acquisition University have attempted, by different means, to compare prototyping and nonprototyping programs and determine the number of successes for each—but results were either inferred, conjectured, or extrapolated and are, therefore, insufficient to establish a repeatable pattern. Also, no acquisition program has ever been delivered without CP, then repeated with CP to determine whether, and to what extent, CP improved outcomes. The cost alone of such duplica-tion would be prohibitive. As Drezner (1992) affirms, “One fundamental problem is that while we can examine prototyping programs, or compare the outcomes of prototyping and nonprototyping programs, the outcome

for the same program with and without prototyping can never be known” (p. 59). A literature review did not reveal any previous attempts or methods to assess the likelihood of a prototype’s success. This is largely due to the “ad hoc prototyping occurrences and disparate methodologies among the military branch acquisition constructs” (Coble et al., 2014, p. 158). Therefore, a formal discussion to assess prototyping benefits versus funds, time, and opportunity costs is nonexistent in the DoD. The methodology in this article is proposed to help with the discussion.

Prototype Benefit FactorsProviding a comprehensive assessment of, and indicators for, a proto-

type’s probability of success requires knowledge of the prototyping benefits and the factors required for their achievement. The key to comprehensive understanding of the factors critical to a project’s success is to examine the factors consistently used (Cooke-Davies, 2002). Yet it is difficult—and sometimes impossible—to quantify many project characteristics and success indicators (Wohlin & Andrews, 2001). Understanding roles and associations is equally important. The Office of the Deputy Assistant Secretary of Defense, Emerging Capability & Prototyping Office, discusses the prototyping roles and benefits relative to the acquisition phases and Technology Readiness Levels.

This research revealed several factors upon which prototyping benefits depend for overall risk reduction. Assessing the likelihood that such factors will be attained yields the probability that program risks will be reduced. To identify and document these factors, the literature and various acqui-sition-related documents were reviewed. Also, government records and reports were examined and scholars, researchers, and veteran defense acquisition professionals were interviewed. This research culls from a variety of disparate information sources to create an organized listing that directly relates a specific factor to a specific prototyping benefit.

Figure 3 shows the identified factors required to achieve each of the prototyping benefits. Each factor is given a designation—for example, B1-F1—according to its contribution to prototyping benefits. Benefits and factors were listed in a survey instrument, which was then given to defense acquisition professionals for verification. In addition, survey participants ranked the factors by importance using ordinal numbers. As shown in Figure 3, the Level 1 factors (top row) are DoD-stated benefits. Level 2 fac-tors are the benefits revealed in this research.



FIG

UR

E 3.

PR

OTO

TY

PIN

G B

EN

EFI

TS A

ND

CO

RR

ESP

ON

DIN

G F

AC

TOR

S

B1-R

educ

e Te

chni

cal R

iskB2

-Val

idat

e De

sign

B3-E

valu

ate

Man

ufac

turin

g Pr

oces

ses

B5-R

efin

e Re

quire

men

ts

Pro

toty

pe

Ben

efit

s

B1-

F1:

Co

ver

and

tes

t p

erfo

rman

ce r

equ

irem

ents

sp

ecif

ied

fo

r th

e en

d u

nit

B1-

F2:

Lik

ely

to m

eet

all

req

uir

emen

ts

B1-

F3

: Co

ntri

but

e to

id

enti

fyin

g a

nd r

eso

lvin

g

tech

nic

al u

nce

rtai

nty

B1-

F4

: Dem

ons

trat

e te

chno

log

ical

fea

sib

ility

B1-

F5

: Ad

vanc

e te

chno

log

ical

mat

urit

y

B1-

F6

:Pro

vid

e in

form

atio

n to

imp

rove

est

imat

es o

f co

st, s

ched

ule

, an

d

per

form

ance

B1-

F7:

Dis

cove

r te

chni

cal

unce

rtai

nty

no

t an

tici

pat

ed

by

des

ign

eng

inee

rs

B1-

F8

: Dis

cove

r te

chni

cal

unce

rtai

nty

no

t p

red

icte

d

by

des

ign

anal

yses

B1-

F9

: Dis

cove

r te

chni

cal

unce

rtai

nty

not

pre

dic

ted

b

y p

rio

r ex

per

ienc

e

B1-

F10

: Ad

dre

ss b

oth

the

“k

now

n un

kno

wn

s” a

nd t

he

“un

kno

wn

un

kno

wns

”

B1-

F11

:Po

ssib

ly y

ield

ing

u

nexp

ecte

d (

un

loo

ked

-fo

r)

ben

efit

s

B2

-F

1: F

ulfi

ll th

e d

efin

edu

ser

need

s un

der

sp

ecif

ied

o

per

atin

gco

ndit

ions

B2

-F

2: A

ble

to

val

idat

e th

e te

chno

log

y us

ed in

the

sy

stem

B2

-F

3: V

alid

ate

read

ines

s to

mo

ve in

to s

ubse

que

nt

pro

gra

m p

hase

s

B2

-F

4: V

alid

ate

read

ines

s to

m

ove

into

fo

rce

stru

ctu

re

B2

-F

5:G

ener

ate

info

rmat

ion

dur

ing

des

ign,

fa

bri

cati

on,

and

tes

ting

ac

tivi

ties

tha

t ca

n im

pro

ve

sub

seq

uent

dec

isio

ns

reg

ard

ing

co

st-

per

form

ance

tra

deo

ffs

B2

-F

6: G

ener

ate

info

rmat

ion

dur

ing

des

ign,

fa

bri

cati

on,

and

tes

ting

ac

tivi

ties

tha

t ca

n im

pro

ve

sub

seq

uent

dec

isio

ns

reg

ard

ing

so

urce

sel

ecti

on

B2

-F

7: T

he p

roto

typ

e w

ill

hel

p d

eter

min

e if

th

e sy

stem

sa

tisf

ies

its

op

erat

iona

l and

sy

stem

-lev

el r

equ

irem

ents

B2

-F

8: T

he p

roto

typ

e w

ill

hel

p d

eter

min

e if

th

e sy

stem

w

ill p

erfo

rm a

s sp

ecif

ied

B4-V

alid

ate

Cost

Es

timat

es

B4

-F

1: T

echn

ical

Ris

k R

educ

tio

n

B4

-F

2: M

ore

Mat

ure

T

echn

olo

gy

B4

-F

3: R

efin

emen

t o

f re

qui

rem

ents

bas

ed o

n d

emo

nstr

ated

fe

asib

ility

B4

-F

4: V

alid

atio

n o

f ke

y sy

stem

des

ign

el

emen

ts

B3

-F

3: T

oo

ling

B3

-F

4: M

ater

ial

use

and

han

dlin

g

B3

-F

5:

Pro

duc

tio

n p

roce

ss la

you

t

B3

-F

6: B

ring

ing

p

roce

sses

und

er

stat

isti

cal c

ont

rol

B6-O

ther

Be

nefit

s

B5

-F

1:

Fun

ctio

nal

B5

-F

2:P

erfo

rman

ce

B5

-F

3: S

yste

m

Tec

hnic

al

B5

-F

4:

Sp

ecif

icat

ions

B3

-F

2:D

emo

nstr

ate

pro

duc

tio

n p

roce

sses

B3

-F

1:

Dem

ons

trat

e an

d

valid

ate

man

ufac

turi

ng

pro

cess

es t

hat

rep

rese

nt

adva

nce

s b

eyo

nd t

he

curr

ent

cap

abili

ty

Do

D

Sta

ted

B

enef

its

=L

eve

l 1

Fac

tors

Ben

efit

s F

acto

rs

Der

ived

F

rom

T

his

R

esea

rch

=L

eve

l 2

Fac

tors

B4

-F

5: E

valu

atio

n o

f th

e co

st, s

ched

ule,

and

p

erfo

rman

ce o

f th

e p

rog

ram

, rel

ativ

e to

cu

rren

t m

etri

cs,

per

form

ance

re

qui

rem

ents

, and

b

asel

ine

par

amet

ers

Exp

lan

atio

n o

f V

aria

ble

s:

B1

= B

enef

it 1

B

5 =

Ben

efit

5

B1-

F1

= B

enef

it 1

, Fac

tor

1B

5-F

1 =

Ben

efit

5, F

acto

r 1

B1-

F11

= B

enef

it 1

, Fac

tor

11

B

5-F

4 =

Ben

efit

5, F

acto

r 4

B6

-F

1:

Red

ucti

on

in

Fie

ldin

g T

ime

B6

-F

2:

Enh

ance

d

Co

mp

etit

ion

B6

-F

3:

Imp

rove

d

Res

earc

h an

d

Dev

elo

pm

ent

(R&

D)

Eff

icie

ncy

B6

-F

1: H

edg

e ag

ains

t o

ther

ki

nds

of

unce

rtai

nty

bey

ond

te

chn

ical

un

cert

aint

y

The PoPS ApproachAcquisition programs vary, and contain several acquisition categories

and subcategories. Accordingly, the use of prototypes varies from one acqui-sition program to another. Also, each program requires its own strategy to describe key elements such as requirements, resources, testing, contracting approach, and open systems design (Brown, 2010). PoPS uses Level 1 factors and Level 2 metrics for each Level 1 factor. It follows the Work Breakdown Structure format for its calculation and reflects all of the Level 1 factors and Level 2 metrics that are applicable in a particular program. PoPS is flexible, and takes into account each program’s uniqueness and corresponding met-rics—for example, the “correct set of instructions to be used by a particular program depends on the current life cycle of the program” (U.S. Air Force, 2007, p. 8). As such, and considering that benefits vary depending upon the type of prototyping envisioned, a PoPS approach allows an assessment that takes into consideration the uniqueness of each program’s key elements. Accordingly, the metrics and scores can be designed based on different criteria for each program.

Similar to the problem this article addresses, PoPS is designed to address the absence of a “consistent, repeatable methodology to assess all program risks objectively in today’s acquisition environment” (U.S. Air Force, 2007, p. 7). The similarity between PoPS and this article’s methodology is in having Level 2 factors that in aggregate calculate the Level 1 factor assess-ment. The scoring and the aggregation of scores, however, are different: PoPS scoring includes colors to indicate a status driven by an assessment point. This article’s methodology includes only assessment-based scoring, but not a status. While PoPS uses a combination of internal and external factors as Level 1, the prototyping benefits are designated as Level 1 in this article. Similar to PoPS, each DoD-stated benefit is aligned with its factors as criteria for success, which are designated as Level 2 factors (Figure 3). To determine whether a prototype actually accomplishes Level 1 factors, Level 2 factors are used as a basis for predicting its success or failure.

The combination of decomposing benefits into factors, ranking the factors, and determining their weights provides a measure and tool to assess the likelihood of a prototype’s success in achieving the desired benefits.



SurveyA survey developed and piloted with assistance from acquisition and

systems engineering professionals elicited judgments that verified the benefits and factors. The survey contained 92 data fields. An open-ended comment area was also included for participants to provide rationales/observations. Participants were required to have at least 5 years of expe-rience in defense acquisition, but were not required to provide any data regarding career fields. The survey instrument was provided on a website or e-mailed as a PDF fillable form. Survey participants will remain anony-mous, and all potential respondents’ survey files were password-protected upon receipt.

The survey results were collected through February 2016. Participants were: (a) queried whether they agreed with each of the DoD-stated benefits and with the factors identified by the research; (b) asked to rank-order these benefits according to their importance; and (c) asked to rank-order the factors according to each factor’s contribution to the achievement of its corresponding benefit. Ordinal scales were used to rank the benefits and their factors. Eighty-one percent agreed with DoD-stated benefits, while 79 percent agreed with the research findings. Table 2 summarizes survey data and participant demographics.

TABLE 2. SURVEY SUMMARY RESEARCH DATA AND DEMOGRAPHICS

Number of Survey Participants 35

Average Years of Experience in Defense Acquisition 17.15

Percent Agreement with DoD-Stated Benefits 81%

Percent Agreement with these Research Findings 79%

AnalysisThe research findings and survey data are analyzed in two primary

areas: (a) verification of findings on prototyping benefits and their factors; and (b) comparison, ranking, and weighting of prototyping benefits and factors. The aim of the analysis is to understand how the results provide supportive evidence to justify the use of a PoPS-like approach to assess a prototype’s likelihood of success.

Rankings of Prototyping Benefits and Their FactorsBecause all variables are unlikely to have equivalent impacts, survey

participants were asked to rank-order the DoD-stated benefits and their fac-tors. Additionally, they noted agreement with, neutrality, or disagreement

with, the research findings. The benefits are ranked according to their importance. The factors are ranked according to each factor’s contribution to attaining prototyping benefits.

One of the main advantages of ranking weight methods is their simple reli-ance on ordinal information to describe criteria importance (Roszkowska, 2013). For this research, the purpose of the ranking is twofold: (a) to deter-mine which benefits and factors are most important for prototyping success, and (b) to determine, for each, the weight of its contribution to prototyping success, which is then used to calculate the overall likelihood of success. Since the use of a prototype is expected to reduce risks to cost, schedule, and scope, the weights provide flexibility when (and if) one risk is determined to be more significant than another in future programs. Consequently, the weights could change according to the importance of the specific risk in a particular program.

Table 3 shows the details of the experts’ levels of agreement with the DoD-stated benefits (Level 1 factors). Table 4 shows the details of the experts’ levels of agreement with survey findings regarding the factors that con-tribute to prototyping benefits (Level 2 factors). Unlike Level 1 factors, and based on recommendations from professionals who helped pilot the survey, only Agree, Neutral, and Disagree were offered as responses for Level 2 factors.

TABLE 3. SCORES OF DOD-STATED PROTOTYPE BENEFITSBY SURVEY PARTICIPANTS

DoD-StatedPrototype Benefits

Agree Somewhat Agree Neutral Somewhat

Disagree Disagree

Ans % Ans % Ans % Ans % Ans %

B1 - Reduction in Technical Risk 28 80% 4 11% 0 0% 2 6% 0 0%

B2 - Validation of Designs 20 57% 15 43% 1 3% 0 0% 0 0%

B3 - Evaluation of Manufacturing Processes 9 26% 12 34% 6 17% 4 11% 4 11%

B4 - Validation of Cost Estimates 8 23% 14 40% 7 20% 3 9% 2 6%

B5 - Refining of Requirements 19 54% 11 31% 1 3% 4 11% 0 0%

B6 - Others 20 57% 10 29% 5 14% 0 0% 1 3%

Total Score 104 50% 66 31% 20 10% 13 6% 7 3%

Note. Ans = Answers.



TABLE 4. SCORES OF PROTOTYPE BENEFIT FACTORS BYSURVEY PARTICIPANTS

1- Reduction in Technical Risk Factors Agree Neutral Disagree

B1-F1 The prototype covers and tests performance requirements specified for the end unit 31 2 2

B1-F2 The prototype is likely to meet all requirements 11 7 17

B1-F3 The prototype will contribute to identifying and resolving technical uncertainty 33 2 0

B1-F4 The prototype will demonstrate technological feasibility 33 2 0

B1-F5 The prototype will advance technological maturity 28 6 1

B1-F6 The prototype will provide information to improve estimates of cost, schedule, and performance 29 5 2

B1-F7 The prototype will discover technical uncertainty not anticipated by design engineers 34 1 0

B1-F8 The prototype will discover technical uncertainty not predicted by design analyses 32 3 0

B1-F9 The prototype will discover technical uncertainty not predicted by prior experience 31 4 0

B1-F10 The prototype will address both the “known unknowns” and the “unknown unknowns” 20 8 6

B1-F11 The prototype might possibly yield unexpected (unlooked-for) benefits 34 1 0

TOTAL 316 41 28AVERAGE SCORE 82% 11% 7%

2- Validation of Designs Factors Agree Neutral Disagree

B2-F1 The prototype fulfills the defined user needs under specified operating conditions 23 10 3

B2-F2 The prototype is able to validate the technology used in the system 33 2 0

B2-F3 The prototype can validate readiness to move into subsequent program phases 35 0 0

B2-F4 The prototype can validate readiness to move into force structure 17 13 5

B2-F5The prototype can generate information during design, fabrication, and testing activities that can improve subsequent decisions regarding cost performance trade-offs

33 1 0

B2-F6The prototype can generate information during design, fabrication, and testing activities that can improve subsequent decisions regarding source selection

32 3 0

B2-F7 The prototype will help determine if the system satisfies its operational and system-level requirements 23 10 2

B2-F8 The prototype will help determine if the system will perform as specified 25 8 2


3- Evaluation of Manufacturing Processes Factors Agree Neutral Disagree

B3-F1 Demonstrate and validate manufacturing processes that represent advances beyond the current capability 27 5 3

B3-F2 Demonstrate production processes 23 6 6

B3-F3 Tooling 20 13 2

B3-F4 Enhancement of material use and handling 22 10 3

B3-F5 Production process layout 20 11 4

B3-F6 Bringing processes under statistical control 17 11 7


4- Validation of Cost Estimates Factors Agree Neutral Disagree

B4-F1 Competitive Prototyping could improve the quality of cost estimates if reductions in technical risk are achieved 30 4 1

B4-F2 Competitive Prototyping could improve the quality of cost estimates if it can contribute to more mature technology 28 7 0

B4-F3 Refinement of requirements based on demonstrated feasibility 32 3 0

B4-F4 Validation of key system design elements 33 1 1

B4-F5Evaluation of the cost, schedule, and performance of the program, relative to current metrics, performance requirements, and baseline parameters

27 6 2


5- Refining of Requirements Factors Agree Neutral Disagree

B5-F1 Functional requirements (task/action/activity that provide an operational capability or an operational requirement) 32 2 1

B5-F2 Performance requirements (quantity, accuracy, coverage, timeliness, and readiness) 33 2 0

B5-F3 System technical requirements (allocated and derived requirements) 31 2 2

B5-F4 Specifications (material, dimensions, quality of work) 28 5 2


Prototype Benefits Factors Summary Table Agree Neutral Disagree

Total Answers 940 176 74

Score 79% 15% 6%



Methodology to Determine RankingsTo determine the final ranking for each Level 1 and Level 2 factor, a

basic scoring method using simple math, as described herein, was used. Also, Bradley-Terry (BT), a logistic pairwise comparison model, was used. Two methods were used to compare the results and verify that the derived rankings are independent of the method. Both methods yield the same rankings.

Basic Scoring The basic scoring method counts the number of times a factor is given a

certain rank, where each rank is scored according to the number of factors. For example, in the case in which a participant is ranking four factors, when a factor is ranked No. 1, it gets 4 points; when ranked No. 2, it gets 3 points; when ranked No. 3, it gets 2 points; and when ranked No. 4, it gets 1 point. After tallying the results, the factor with the highest scores is ranked No. 1, the factor with the next best scores is ranked No. 2, etc. It works as follows:

1. Count how many times each factor is ranked #1, #2, #3, ..., #n.

2. Populate the scoring matrix according to the data from the surveys, where Xij is the number of times a factor i was voted j; i=1, …, n; j=1, …, n.

3. Calculate the score, Si, for each factor:

Si = (Xi1 )(n) + (Xi2)(n – 1) + (Xi3)(n – 2) + ... + (Xin )[n – (n – 1)] (1)

4. Compute the sum, S, which is the total for all Si:

S =n

Si (2)∑ i=1

5. Determine the weight Wi of each factor according to its score percentage out of the overall score:

Wi = Si (3)S

6. Determine the rank ri of the ith factor, where r1 is the factor with the highest score percentage, and rn is the factor with the lowest score percentage:

1 • j • n Sum(i) Weight(i) Rank(i)

(4)

1 X11 X1j X1n S1 W1 R1j

•i Xi1 Xij Xin Si Wi Rij

•n Xn1 Xnj Xnn Sn Wn Rnj

S

Bradley-Terry Scoring A BT model (Bradley & Terry, 1952) is used to compare the ranking

of Level 1 and Level 2 factors obtained from the simple scoring method to rankings obtained using the BT model. A Dictionary of Statistics (Upton & Cook, 2008) describes the BT model as capable of yielding the probability that one treatment will be preferable to another. The comparison is also useful for validating these rankings, and according to Rettaliata, Mazzuchi, and Sarkani (2014), “for determining which of the subjects being compared are more probable to be picked” (p. 12).

Introduced as a paired comparison method, the BT model estimates the rankings or preferences of surveyors, assessors, and evaluators, and is valuable for identifying preferences in pairwise comparisons. The BT model computes and analyzes the judgments of survey participants to determine the presence of circular triads and a coefficient of concordance. It establishes, as Hunter (2004) states, “the probabilities of the possible outcomes when individuals are judged against one another in pairs” (p. 384). Rettaliata et al. (2014) point out that “an important aspect of the BT model is that by following a logarithmic approach, it allows the user to determine the consistency of the experts’ preference in their decision between com-parisons” (p. 12).



Calculation of a pairwise comparison using a BT derivation, which is com-plex, has been extensively addressed in the literature (Agresti, 2002; Bradley & Terry, 1952; Cooke, 1991; Davidson, 1970; Hunter, 2004). However, since this is not the focus of our research, this article does not expound upon the actual derivation; instead, UNIBALANCE (Macutkiewicz & Cooke, 2006), a commercially available software, is used to compute and validate the fac-tors-ranking program. The factor ranked higher was given a binary value of 1, and the factor ranked lower was given a binary value of 0. An example can be seen in Figure 4, in which factor 2 was ranked higher than all other factors by participant n and received a 1 horizontally relative to other fac-tors. A shaded area corresponds to the same factor where no comparison is possible. Survey data were entered into UNIBALANCE for all Level 1 and Level 2 factors. Table 5 shows BT results for a DoD-stated prototype’s benefits (Level 1 factors) as computed by UNIBALANCE. A higher score means a higher level of contribution to prototype success.

FIGURE 4. SAMPLE UNIBALANCE BINARY INPUT

Factor 6 ranked lower than Factor 4

Factor 3 ranked lower than Factors

1 , 2, 4, 5, and 6

1 2 3 4 5 6

1 0 1 0 1 1

2 1 1 1 1 1

3 0 0 0 0 0

4 1 0 1 1 1

5 0 0 1 0 1

6 0 0 1 0 0

Factor 2 ranked higher than Factors

1, 3, 4, 5, and 6

FIGURE 4. SAMPLE UNIBALANCE BINARY INPUT

TABLE 5. UNIBALANCE BRADLEY-TERRY MODEL RESULTSFOR LEVEL 1 FACTORS

DoD-Stated Prototype Benefits UNIBALANCE Scores

B1 - Reduction in Technical Risk 56.73%

B2 - Validation of Designs 22.89%

B3 - Evaluation of Manufacturing Processes 2.69%

B4 - Validation of Cost Estimates 3.18%

B5 - Refining of Requirements 12.61%

B6 - Others 1.90%

Weights of Factors Relative to RankingsFrom the survey rankings’ results, the weight of each prototyping ben-

efit and factor is calculated based on the corresponding rank using existing weighting techniques. Weighting each benefit and factor further improves the value and usefulness. The combination of decomposing benefits into factors, ranking the factors, and determining their weights provides a mea-sure and tool to assess the likelihood of a prototype’s success in achieving the desired benefits. There are two fundamental methods for deriving the weight of a criterion’s importance: direct and indirect explication (Zeleny & Cochrane, 1973). In direct explication, weights are determined before data collection and referred to as a priori weights (Kao, 2010). In indirect explication, weights are assigned based on the data. Because weights were not elicited from survey participants, the indirect explication method of weighting by ranking is chosen due to its simplicity and because its explan-atory power remains high when there are a relatively small number of criteria. Roszkowska (2013) states: “Generally, the ranking method of weight determination involves two steps: ranking the criteria according to their importance, and weighting the criteria from their ranks using one of the rank-order weighting formula” (p. 15). Rankings were in ascending order: The most important factor was given rank 1, the second most important factor given rank 2, and the least important factor given rank n. To avoid assigning arbitrary weights and to compare and verify weighting results, several weighting methods are used. The weights are derived once using each of these meth-ods without seeking to identify which method is better than the other in calculating the weights (since that is not the main objective of this research).

Simple WeightingXij is the number of times the ith factor was voted j; i=1, …, n; j=1, …, n;

Si = (Xi1)(n) + (Xi2)(n – 1)+(Xi3)(n – 2) + ... + (Xin)[n – (n – 1)] (5)

S =n

Si (6)∑ i=1

Wti = Si = weight of the ith factor (7)S



Rank Sum

Wti = K – ri + 1

= weight for the ith factor (8)∑ Kj=1 K – ri + 1

ri is the rank of the ith factor

K is the total number of factor

Rank Exponent

Wti = (K – ri + 1)z

= weight for the ith factor (9)∑ Kj=1 (K – ri + 1)z

ri is the rank of the ith factors

K is the total number of factors

z is an undefined measure of the dispersion in the weights

Rank Reciprocal

Wti = 1/ri

= weight for the ith factor (10)∑ Kj=1 1/rj

ri is the rank of the ith factors

K is the total number of factors

Rank Order Centroid

Wti = 1K

K

1rj

= weight for the ith factor (11)∑ j=1

Wt1 = 12

1 + + + ... /K13

1K( ) (12)

WtK= 0 + 0 + 0 + ... /K1K( ) (13)

K is the total number of factors.

Table 6 provides a summary of the Level 1 factors’ rankings and weights according to the experts surveyed. Results indicate that all weighting methods, including BT, yielded the same rankings, with only slight varia-tions in weights.

Table 7 lists the results for Level 2 factors using the same methodology as in Table 6. Similar to Table 6, Table 7 shows that all weighting methods, including BT, yielded the same rankings, with only slight variations in weights.

TABLE 6. PROTOTYPE BENEFITS RANKINGS AND WEIGHTS

Prototype Benefits

SC Weight of Contribution toPrototype Success by Method Rankings Bradley-

Terry

SiBasic

WeightingRank Sum

Rank Reciprocal

Rank Order

Centroid

Rank Exponent

Z=1

Weights AVG Si

AVG SC Rank

B1 - Reduction in Technical Risk

189 25.71% 28.57% 40.82% 40.83% 28.57% 32.90% 1 1 56.73% 1

B2 - Validation of Designs 162 22.04% 23.81% 20.41% 24.17% 23.81% 22.85% 2 2 22.89% 2

B3 - Evaluation of Manufacturing Processes

83 11.29% 9.52% 8.16% 6.11% 9.52% 8.92% 5 5 2.69% 5

B4 - Validation of Cost Estimates

89 12.11% 14.29% 10.20% 10.28% 14.29% 12.23% 4 4 3.18% 4

B5 - Refining of Requirements

141 19.18% 19.05% 13.61% 15.83% 19.05% 17.34% 3 3 12.61% 3

B6 - Others 71 9.66% 4.76% 6.80% 2.78% 4.76% 5.75% 6 6 1.90% 6

Note. AVG = average; SC = scores



TAB

LE 7

. PR

OTO

TY

PE

BE

NE

FIT

FAC

TOR

S: R

AN

KIN

GS

AN

D W

EIG

HTS

B1

- R

educ

tio

n in

Tec

hnic

al R

isk

Fact

ors

Sco

reS

i

Lev

el 2

W

eig

ht

Fac

tor

Ran

kin

gB

T

Sco

reB

T

Ran

kin

gO

vera

ll W

eig

ht

B1-

F1

Th

e p

roto

typ

e co

vers

an

d t

ests

pe

rfo

rman

ce r

eq

uir

em

en

ts s

pe

cifi

ed

fo

r th

e e

nd

un

it2

59

11.3

9%

312

.40

%3

3.7

5%

B1-

F2

Th

e p

roto

typ

e is

like

ly t

o m

eet

all

req

uir

em

en

ts10

94

.80

%11

1.9

5%

111.

58%

B1-

F3

Th

e p

roto

typ

e w

ill c

on

trib

ute

to

ide

nti

fyin

g a

nd

res

olv

ing

te

chn

ical

un

cert

ain

ty3

03

13.3

3%

12

3.5

4%

14

.39

%

B1-

F4

Th

e p

roto

typ

e w

ill d

em

on

stra

te t

ech

no

log

ical

fe

asib

ility

30

113

.24

%2

22

.81%

24

.36

%

B1-

F5

Th

e p

roto

typ

e w

ill a

dva

nce

te

chn

olo

gic

al m

atu

rity

23

810

.47

%4

9.6

9%

43

.45%

B1-

F6

Th

e p

roto

typ

e w

ill p

rovi

de

info

rmat

ion

to im

pro

ve e

stim

ates

of

cost

, sch

ed

ule

, an

d

pe

rfo

rman

ce16

47.

22

%8

4.0

8%

82

.37%

B1-

F7

Th

e p

roto

typ

e w

ill d

isco

ver

tech

nic

al u

nce

rtai

nty

no

t an

tici

pat

ed

by

des

ign

en

gin

ee

rs2

36

10.3

8%

59

.16

%5

3.4

2%

B1-

F8

Th

e p

roto

typ

e w

ill d

isco

ver

tech

nic

al u

nce

rtai

nty

no

t p

red

icte

d b

y d

esig

n an

alys

es2

09

9.1

9%

66

.44

%6

3.0

3%

B1-

F9

Th

e p

roto

typ

e w

ill d

isco

ver

tech

nic

al u

nce

rtai

nty

no

t p

red

icte

d b

y p

rio

r ex

pe

rie

nce

184

8.1

0%

74

.76

%7

2.6

6%

B1-

F10

Th

e p

roto

typ

e w

ill a

dd

ress

bo

th t

he

“kn

ow

n u

nkn

ow

ns”

an

d t

he

“un

kno

wn

un

kno

wn

s”13

65

.98

%9

2.6

7%

91.

97%

B1-

F11

Th

e p

roto

typ

e m

igh

t p

oss

ibly

yie

ld u

nex

pe

cte

d (

un

loo

ked

-fo

r) b

en

efi

ts13

45

.90

%10

2.5

0%

101.

94

%

B2

- V

alid

atio

n o

f D

esig

ns F

acto

rsS

core

Si

Lev

el 2

W

eig

ht

Fac

tor

Ran

kin

gB

T S

core

BT

R

anki

ng

Ove

rall

Wei

gh

t

B2-

F1

Th

e p

roto

typ

e fu

lfills

th

e d

efi

ne

d u

ser

ne

ed

s u

nd

er

spe

cifi

ed

op

era

tin

g c

on

dit

ion

s15

412

.32

%5

10.1

6%

52

.81%

B2-

F2

Th

e p

roto

typ

e is

ab

le t

o v

alid

ate

the

tech

no

log

y u

sed

in t

he

syst

em

216

17.2

8%

127

.30

%1

3.9

5%

B2-

F3

Th

e p

roto

typ

e ca

n va

lidat

e re

adin

ess

to m

ove

into

su

bse

qu

en

t p

rog

ram

ph

ases

183

14.6

4%

315

.47

%3

3.3

4%

B2-

F4

Th

e p

roto

typ

e ca

n va

lidat

e re

adin

ess

to m

ove

into

fo

rce

stru

ctu

re9

17.

28

%8

3.5

3%

81.

66

%

B2-

F5

Th

e p

roto

typ

e ca

n g

en

era

te in

form

atio

n d

uri

ng

des

ign

, fab

rica

tio

n, a

nd

tes

tin

g a

ctiv

itie

s th

at c

an im

pro

ve s

ub

seq

ue

nt

de

cisi

on

s re

gar

din

g c

ost

pe

rfo

rman

ce t

rad

e-o

ffs

198

15.8

4%

219

.83

%2

3.6

2%

B2-

F6

Th

e p

roto

typ

e ca

n g

en

era

te in

form

atio

n d

uri

ng

des

ign

, fab

rica

tio

n, a

nd

tes

tin

g a

ctiv

itie

s th

at c

an im

pro

ve s

ub

seq

ue

nt

de

cisi

on

s re

gar

din

g s

ou

rce

sele

ctio

n16

513

.20

%4

11.6

1%4

3.0

2%

B2-

F7

Th

e p

roto

typ

e w

ill h

elp

det

erm

ine

if t

he

syst

em

sat

isfi

es it

s o

pe

rati

on

al a

nd

syst

em

-lev

el r

eq

uir

em

en

ts11

28

.96

%7

5.1

8%

72

.05%

B2-

F8

Th

e p

roto

typ

e w

ill h

elp

det

erm

ine

if t

he

syst

em

will

pe

rfo

rm a

s sp

eci

fie

d13

110

.48

%6

6.9

3%

62

.39

%

B3

- E

valu

atio

n o

f M

anuf

actu

ring

Pro

cess

es F

acto

rsS

core

Si

Lev

el 2

W

eig

ht

Fac

tor

Ran

kin

gB

T S

core

BT

R

anki

ng

Ove

rall

Wei

gh

t

B3

-F1

De

mo

nst

rate

an

d v

alid

ate

man

ufa

ctu

rin

g p

roce

sses

th

at r

ep

rese

nt

adva

nce

s b

eyo

nd

th

e cu

rre

nt

cap

abili

ty15

83

0.3

3%

13

5.9

4%

12

.71%

B3

-F2

De

mo

nst

rate

pro

du

ctio

n p

roce

sses

107

20

.54

%2

25

.13

%2

1.8

3%

B3

-F3

Too

ling

7013

.44

%5

10.5

6%

51.

20%

B3

-F4

En

han

cem

en

t o

f m

ate

rial

use

an

d h

and

ling

85

16.3

1%3

13.0

9%

31.

46

%

B3

-F5

Pro

du

ctio

n p

roce

ss la

you

t74

14.2

0%

411

.51%

41.

27%

B3

-F6

Bri

ng

ing

pro

cess

es u

nd

er

stat

isti

cal c

on

tro

l27

5.1

8%

63

.76

%6

0.4

6%

B4

- V

alid

atio

n o

f C

ost

Est

imat

es F

acto

rsS

core

Si

Lev

el 2

W

eig

ht

Fac

tor

Ran

kin

gB

T S

core

BT

R

anki

ng

Ove

rall

Wei

gh

t

B4

-F1

Co

mp

etit

ive

Pro

toty

pin

g c

ou

ld im

pro

ve t

he

qu

alit

y o

f co

st e

stim

ates

if r

ed

uct

ion

s in

te

chn

ical

ris

k ar

e ac

hie

ved

114

21.

84

%3

22

.98

%3

2.6

7%

B4

-F2

Co

mp

etit

ive

Pro

toty

pin

g c

ou

ld im

pro

ve t

he

qu

alit

y o

f co

st e

stim

ates

if it

can

co

ntr

ibu

te

to m

ore

mat

ure

te

chn

olo

gy

98

18.7

7%

415

.55

%4

2.3

0%

B4

-F3

Re

fin

em

en

t o

f re

qu

ire

me

nts

bas

ed

on

de

mo

nst

rate

d f

eas

ibili

ty12

624

.14

%1

31.

05

%1

2.9

5%

B4

-F4

Val

idat

ion

of

key

syst

em

des

ign

ele

me

nts

116

22

.22

%2

23

.56

%2

2.7

2%

B4

-F5

Eva

luat

ion

of

the

cost

, sch

ed

ule

, an

d p

erf

orm

ance

of

the

pro

gra

m, r

ela

tive

to

cu

rre

nt

met

rics

, pe

rfo

rman

ce r

eq

uir

em

en

ts, a

nd

bas

elin

e p

aram

ete

rs6

813

.03

%5

6.8

6%

51.

59%

B5

- R

efini

ng o

f R

equi

rem

ents

Fac

tors

Sco

reS

i

Lev

el 2

W

eig

ht

Fac

tor

Ran

kin

gB

T S

core

BT

R

anki

ng

Ove

rall

Wei

gh

t

B5

-F1

Fu

nct

ion

al r

eq

uir

em

en

ts (

task

/act

ion

/act

ivit

y th

at p

rovi

de

an o

pe

rati

on

al c

apab

ility

or

an o

pe

rati

on

al r

eq

uir

em

en

t)10

33

0.1

2%

13

7.8

6%

15

.22%

B5

-F2

Pe

rfo

rman

ce r

eq

uir

em

en

ts (

qu

anti

ty, a

ccu

racy

, co

vera

ge

, tim

elin

ess,

an

d r

ead

ines

s)9

72

8.3

6%

22

8.3

6%

24

.92%

B5

-F3

Sys

tem

te

chn

ical

re

qu

ire

me

nts

(al

loca

ted

an

d d

eri

ved

re

qu

ire

me

nts

)9

12

6.6

1%3

26

.63

%3

4.6

1%

B5

-F4

Sp

eci

fica

tio

ns

(mat

eri

al, d

ime

nsi

on

s, q

ual

ity

of

wo

rk)

51

14.9

1%4

7.14

%4

2.5

9%



Model ImplementationAfter survey participants had verified the research choices, data were

divided into Level 1 and Level 2 factors, as shown in Figure 3. Raw data—that is, ordinal ranks from the survey instrument—were computed using equations (1), (2), (3), and (4) to provide rankings for Level 1 factors, as shown in Table 6, and Level 2 factors, as shown in Table 7. Once the rank-ings had been finalized, the weights for each Level 1 and Level 2 factor were computed using equations (5) through (13) for the corresponding weighting methods. Weights from all elicitation techniques were averaged to provide the final weight for each Level 1 factor, as shown in Table 6, and each Level 2factor, as shown in Table 7. Similar to PoPS, Level 1 and Level 2 weights were arranged together (Figure 5)

FIGURE 5. LEVEL 1 AND LEVEL 2 FACTORS: WEIGHTS SIMILAR TO POPS

B132.90%

B222.85%

B412.23%

B517.34%

% Prototype Success

B1-F111.39%

B1-F24.80%

B1-F313.33%

B1-F413.24%

B1-F510.47%

B1-F67.22%

B1-F710.38%

B1-F89.19%

B1-F98.10%

B1-F105.98%

B1-F115.90%

B38.92%

Others5.75%

B2-F1 12.32%

B2-F2 17.28%

B2-F3 14.64%

B2-F4 7.28%

B2-F5 15.84%

B2-F6 13.20%

B2-F7 8.96%

B2-F8 10.48%

B3-F1 30.33%

B3-F2 20.54%

B3-F3 13.44%

B3-F4 16.31%

B3-F5 14.20%

B4-F1 21.84%

B4-F2 18.77%

B4-F3 24.14%

B4-F4 22.22%

B4-F5 13.03%

B3-F6 5.18%

B5-F1 30.12%

B5-F2 28.36%

B5-F3 26.61%

B5-F4 14.91%

Level 1 Factors Weight of Individual Contribution to Prototype Success

Level 2 Factors Weight of Individual Contribution to Level 1 Factor

B6-F1 25%

B6-F2 25%

B6-F3 25%

B6-F4 25%

The computed weights of Level 2 factors shown in Figure 5 (also in Table 7) represent the weight of each factor’s contribution to its corresponding Level 1 factor. Level 2 factors for other benefits in the last column were assigned

equal weights. When each Level 2 factor’s weight is computed according to its direct contribution to a prototype’s success rather than to its corre-sponding benefit, as in Figure 6, defense acquisition program managers can directly measure the prototype’s likelihood of success.

FIGURE 6. WEIGHTS OF LEVEL 1 AND LEVEL 2 FACTORS:INDIVIDUAL CONTRIBUTIONS TO PROTOTYPE SUCCESS

32.90% 22.85% 12.23% 17.34%

% Prototype Success

3.75%

1.58%

4.39%

4.36%

3.45%

2.37%

3.42%

3.03%

2.66%

1.97%

1.94%

8.92% 5.75%

2.81%

3.95%

3.34%

1.66%

3.62%

3.02%

2.05%

2.39%

2.71%

1.83%

1.2%

1.46%

1.27%

2.67%

2.3%

2.95%

2.72%

1.59%

0.46%

5.22%

4.92%

4.61%

2.59%



1.44%

1.44%

1.44%

1.44%

Let WL2ij= the weight of the jth Level 2 factor that corresponds to the ith Level 1 factor Bi, and WL1i= the weight of Bi. The Level 2 factor weight that directly corresponds to prototype success, WPij, is computed as follows:

WPij= (WL2ij)(WL1i) (14)

The computed Level 2 factor weights in Figure 5 were used to test compu-tation of the likelihood of a prototype’s success based on assessment, at the individual level, of each Level 2 factor. For the test, only four assessment levels are used: Low, Average, Good, and Excellent. Each corresponds to the likelihood that a Level 2 factor will be achieved and, therefore, its individual contribution to prototype success. This assessment can be done subjec-tively by defense acquisition program managers, based on their knowledge of prototype specifications or any other key information available to them.



Each assessment level is matched with a percentage, as follows: Low = 25 percent; Average = 50 percent; Good = 75 percent; and Excellent = 100 percent. The model was tested at various assessment levels for each Level 2 factor. In the test, each expected or assessed value of a Level 2 factor is entered in the designated entry field after being assessed or determined by the program manager. Of course, the prototype’s likelihood of success computed by the program would change according to changes made to assessment levels. For illustrative purposes and to present the results, 25 percent assessment is purposely used in one of the research tests for all Level 2 factors being achieved (Figure 7), which corresponds to a Low assessment level. Results show a 25 percent success likelihood, which is the mathematically expected result. Figure 7 is a Java program developed for the research as a sample implementation tool. The weights are built-in and do not require computation, but can be modified to reflect specific program needs. Factors or benefits can be added or removed. New weights can be determined by following the steps outlined previously in this article.

FIGURE 7. LIKELIHOOD OF PROTOTYPE SUCCESS:MODEL TEST RESULTS AT 25% LEVEL

Compute

B1(32.90%)8.23

B2(22.85%)5.71

B3(8.92%)2.23

B4(12.23%)3.06

B5(17.34%)4.34

B6(5.75%)1.44

Likelihood ofPrototype Success

25.01

B1-F1 (3.75%)25

B1-F2 (1.58%)25

B1-F3 (4.39%)25

B1-F4 (4.36%)25

B1-F5 (3.45%)25

B1-F6 (2.37%)25

B1-F7(3.42%)25

B1-F8 (3.03%)25

B1-F9 (2.66%)25

B1-F10 (1.97%)25

B1-F11 (1.94%)25

B2-F1 (2.81%)25

B2-F2 (3.95%)25

B2-F3 (3.34%)25

B2-F4 (1.66%)25

B2-F5 (3.62%)25

B2-F6 (3.02%)25

B2-F7(2.05%)25

B2-F8 (2.39%)25

B3-F1 (2.71%)25

B3-F2 (1.83%)25

B3-F3 (1.20%)25

B3-F4 (1.46%)25

B3-F5 (1.27%)25

B3-F6 (0.46%)25

B4-F1 (2.67%)25

B4-F2 (2.30%)25

B4-F3 (2.95%)25

B4-F4 (2.72%)25

B4-F5 (1.59%)25

B5-F1 (5.22%)25

B5-F2 (4.92%)25

B5-F3 (4.61%)25

B5-F4 (2.59%)25

B6-F1 (1.44%)25

B6-F2 (1.44%)25

B6-F3 (1.44%)25

B6-F4 (1.44%)25

Conclusions When asked about prototyping benefits, as enumerated by the DoD,

81 percent of survey participants agreed on their value. Seventy nine per-cent of the participants also agreed with the research findings. The results present an opportunity to move a step forward in assessing a prototype’s likelihood of success. They offer defense acquisition program managers a flexible approach for assessing success before, during, and after the pro-totyping phase, and can be customized for each prototyping situation. Program managers have the option of adding or removing Level 1 factors or changing the contribution weight based on their expertise and priorities. The same can be done for Level 2 factors. Also, based on survey feedback, the results of the research highlight the relative importance of each factor, and thus can assist program managers in concentrating prototyping effort and assessments on the most important Level 2 factors. Additionally—and sim-ilar to the way Level 1 factors were decomposed into Level 2 factors—Level 2 factors can each be decomposed further into Level 3 factors, for example, when a more granular or precise measurement is needed.

The research focused on exhibiting a repeatable and consistent approach for assessing the likelihood of a prototype’s success to improve the quality of decision making in defense acquisition. The analysis of results provides supportive evidence for a feasibility demonstration and a prototypical solu-tion based on robust scientific principles. Equally important, the proposed approach can be refined and used to assess prototype benefits’ factors identified in the future.

Recommendations for Future ResearchIn an environment where requirements often change, prototyping is

about building options to help achieve capability goals. Future investigation should explore how and to what degree prototyping improves the return on investment and benefits systems engineering. It is also recommended to expand the research to factors that contribute to overall acquisition pro-gram success, including the down-select process. Additionaly, the use and implementation of real probabilistic techniques, such as Bayesian Networks or Fuzzy Theory, should be explored. Finally, future studies should extend the survey size and range to achieve more statistically significant results, and perhaps delineate respondents’ different fields.



ReferencesAcquisition Process Overview. (2017). In AcqNotes [Online DoD acquisition knowl-

edge source]. Retrieved from http://www.acqnotes.com/acqnote/acquisitions/acquisition-process-overview

Agresti, A. (2002). Wiley series in probability and statistics. Analysis of Ordinal Cat-egorical Data, Second Edition, 397, 405. https://doi.org/10.1002/0471249688.scard

Borowski, S. M. (2012). Competitive prototyping in the Department of Defense: Sug-gestions for a better approach. Retrieved from http://oai.dtic.mil/oai/oai?ver-b=getRecord&metadataPrefix=html&identifier=ADA584059

Bradley, R. A., & Terry , M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324–345. https://doi.org/10.2307/2334029

Brown, B. (2010). Introduction to defense acquisition management. Retrieved from http://www.dtic.mil/get-tr-doc/pdf?AD=ADA606328

Coble, M., Royster, J., Glandon, D., Stewart, J., Pham, P., & Taylor, B. (2014).Improving the prototyping process in Department of Defense acquisition.Retrieved from http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=htm-l&identifier=ADA608062

Cooke, R. M. (1991). Experts in uncertainty: Opinion and subjective probability in science. Oxford, UK: Oxford University Press on Demand.

Cooke-Davies, T. (2002). The ‘‘real’’ success factors on projects. International Journal of Project Management, 20(3), 185–190. https://doi.org/10.1016/s0263-7863(01)00067-9

Copeland, E. J., Holzer, T. H., Eveleigh, T. J., & Sarkani, S. (2015). The effects of system prototype demonstrations on weapon systems. Defense Acquisition Research Journal, 22(1), 106–135. Retrieved from http://dau.dodlive.mil/2015/01/02/the-effects-of-system-prototype-demonstrations-on-weapon-systems/

Davidson, R. R. (1970). On extending the Bradley-Terry model to accommodate ties in paired comparison experiments. Journal of the American Statistical Association, 65(329), 317–328. https://doi.org/10.1080/01621459.1970.10481082

Defense Acquisition University. (2017). Defense acquisition guidebook. Retrieved from https://www.dau.mil/tools/dag

Department of Defense. (2017). Operation of the defense acquisition system (DoDI 5000.02). Washington, DC: Office of the Under Secretary of Defense for Acquisition, Technology and Logistics.

Drezner, J. A. (1992). The nature and role of prototyping in weapon system development. (Report No. RAND/R-4161-ACQ). Retrieved from http://www.rand.org/pubs/reports/R4161.html

Hunter, D. R. (2004). MM Algorithms for generalized Bradley-Terry models. Annals of Statistics, 32(1), 384–406. https://doi.org/10.1214/aos/1079120141

Kao, C. (2010). Weight determination for consistently ranking alternatives in multiple criteria decision analysis. Applied Mathematical Modelling, 34(7), 1779–1787. https://doi.org/10.1016/j.apm.2009.09.022

Macutkiewicz, M., & Cooke, R. M. (2006). UNIBALANCE user’s manual. Delft, Netherlands: Technische Universiteit Delft.

MITRE. (2016). Systems engineering guide—competitive prototyping. Retrieved from https://www.mitre.org/publications/systems-engineering-guide/acquisition-systems-engineering/contractor-evaluation/competitive-prototyping

National Defense Authorization Act for Fiscal Year 2016, Pub. L. No., 114–92, §1356, 129 Stat. 726 (2015). Retrieved from https://www.congress.gov/114/plaws/publ92/PLAW-114publ92.pdf

Pflanz, M., Yunker, C., Wehrli, F. N., & Edwards, D. (2012, October). Applying early systems engineering: Injecting knowledge into the capability development process. Defense Acquisition Research Journal, 19(4), 422–443. Retrieved from http://dau.dodlive.mil/2012/10/01/applying-early-systems-engineering-injecting-knowledge-into-the-capability-development-process/

Rettaliata, J. M., Mazzuchi, T. A., & Sarkani, S. (2014). Identifying requirement attributes for materiel and non-materiel solution sets utilizing discrete choice models. Information Knowledge Systems Management, 12(3, 4), 233–254. https://doi.org/10.3233/IKS-140223

Roszkowska, E. (2013). Rank ordering criteria weighting methods – A comparative overview (Research & Theses). Optimum.Studia Ekonomiczne. University of Bialystok, Warsaw, Poland. https://doi.org/10.15290/ose.2013.05.65.02

U.S. Air Force. (2007). Probability of program success (PoPS) spreadsheet operations guide. Retrieved from https://acc.dau.mil/CommunityBrowser.aspx?id=245260&lang=en-US

Upton, G., & Cook, I. (2008). A dictionary of statistics. Oxford, UK: Oxford University Press. https://doi.org/10.1093/acref/9780199679188.001.0001

Weapon Systems Acquisition Reform Act of 2009, 10 U.S.C., Pub. L. 111-23 §454, 123 Stat. 1704 (2009). Retrieved from https://www.gpo.gov/fdsys/pkg/PLAW-111publ23/pdf/PLAW-111publ23.pdf

Wohlin, C., & Andrews, A. A. (2001). Assessing project success using subjec-tive evaluation factors. Software Quality Journal, 9(1), 43–70. https://doi.org/10.1023/a:1016673203332

Young, J. (2007). Prototyping and competition [Memorandum]. Retrieved from https://acc.dau.mil/adl/en-US/170517/file/30581/20070921%20Prototyping%20and%20Competition%20ATL.pdf

Zeleny, M., & Cochrane, J. L. (Eds.). (1973). Multiple criteria decision making. Columbia, SC: University of South Carolina Press.



Author Biographies

Dr. Maroun Medlej is an assistant professor of Finance at The George Washington University School of Business and a program management consultant supporting the Pension Benefits Guaranty Corporation government agency. He is certified in Project Management, Information Technology Infrastructure Library, and Agile. He has supported various government agencies including the Department of Defense. Dr. Medlej holds a Master of Science in Finance degree and a PhD in Systems Engineering from The George Washington University.

(E-mail address: [email protected])

Dr. Steven M. F. Stuban is deputy director of the National Geospatial-Intelligence Agency’s Facility Program Office. He is Defense Acquisition Workforce Improvement Act Level III certified in Program Management and Program Systems Engineer. Dr. Stuban holds a BS in Engineering from the U.S. Militar y Academy, an MS in Engineering Management from the University of Missouri-Rolla, and both an MS and PhD in S y s t em s E n g i ne er i n g f r om T he G e or ge Washington University.


Dr. Jason R. Dever works as a systems engineer supporting the National Reconnaissance Office. He has supported numerous positions across the systems engineering life cycle, including requirements, design, development, deployment, and operations and maintenance. Dr. Dever received his BS in Electrical Engineering from Vi rg i n ia Poly tech n ic Instit ute a nd St ate University, an MS in Engineering Management from The George Washington University, and a PhD in Systems Engineering from The George Washington University. His teaching interests are project management, systems engineering, and quality control.


PROTOTYPING BENEFITS in Systems Acquisition Article 2 - 71-774... · Production Readiness Review;...

Documents

Transcript of PROTOTYPING BENEFITS in Systems Acquisition Article 2 - 71-774... · Production Readiness Review;...