SponsorSponsorProblemProblem

AssessAssessRiskRisk

SolutionSolutionStrategyStrategy

Measures ofMeasures ofMerit (Merit (MoMMoM))

Human &Human &OrganisationalOrganisational

IssuesIssues

ScenariosScenarios

MethodsMethods& Tools& Tools

DataDataProductsProducts

3

6

4

5 7

10

8

911

ProblemProblemFormulationFormulation

Prepare forSuccess

NATO Code of Best Practice (COBP)for C2 Assessment

Measures of MeritChapter 5

Corinne WallsheinAir Force Studies and Analyses Agency (AFSAA)

2

CO B

P Measures of Merit

SponsorSponsorProblemProblem

AssessAssessRiskRisk

ProblemProblemFormulationFormulation

SolutionSolutionStrategyStrategy

Measures ofMeasures ofMerit (Merit (MoMMoM))

Human &Human &OrganizationalOrganizational

IssuesIssues

ScenariosScenarios

ModelsMethods& Tools& Tools

DataDataProductsProducts

3

6

4

5 7

10

8

911

Prepare forSuccess

3

CO B

P Agenda

• Linkages• Key Considerations• Definitions, Hierarchy, and Characterizations• Key Properties• Uncertainties• Examples• Summary

4

CO B

P Linkages

• Measures of Merit (MOMs) should directly link to established assessment objectives– Identify selected MoMs, their thresholds and standards, their

means of collection, their relationship to assumptions and assessment processes, and their imposed constraints

• Review reliability and validity for the selected MoMs– Determine level of confidence for each MoM

• Establish and measure control variables to correlate MoMs in a spectrum of multiple scenarios

5

CO B

P Key Considerations

• A multi-faceted approach is recommended– A multi-phased approach may add to insights– No single measure or methodology exists to satisfactorily assess

the overall effectiveness of C2

• Expect and control for variations in measurements related to the human element

• Analysis of uncertainties and measurements of central tendency and dispersion are significant for C2 issues

6

CO B

P Definitions

• DP - Dimensional Parameters– Properties or characteristics in physical entities

• MoP - Measures of Performance– Measures attributes of internal system behavior

• MoCE - Measures of C2 Effectiveness– Measures impact of C2 systems

• MoFE - Measures of Force Effectiveness– Measures force accomplishment of mission objectives

• MoPE – Measures of Policy Effectiveness– Measures impacts in policy environment

7

CO B

P Hierarchy

C2 Subsystems

MoPElement

DP

C2 Systems

MoCE

Force

MoFE

Environment

MoPE

8

CO B

P Characterizations

MoM Focus Scenario Effort Required

Number Impact Compre-hension

Generaliz-ability

MoPE Outcome Dependent High Few High Policy Low MoFE Mission MoCE C3I MoP Systems DP Process Independent Low Many Limited Technical High

9

CO B

P Key Properties

• Reliability– Accuracy of measurement– Variance of repeated

measurements of same phenomenon

– Must be known or estimated to discriminate between real effects and measurement effects

• Validity– Internal: Causal relationship

between variables– Construct: Measure

objective and only objective– Statistical conclusion:

Results are robust with sufficient sensitivity

– External: Extent to which results could be generalized

– Expert: Degree accepted by experts in the field

10

CO B

P Uncertainties

• Input parameters– Uncertainties in model input values– Relationship of uncertainties between scenario and model

inputs

• Assessment– Uncertainties in assessment methodologies– Uncertainties in model, structural uncertainty

• Output sensitivities– Uncertainties in model outcomes– Sensitivities of outcomes to inputs

11

CO B

P Examples

• Time based– Time to perform a task or process– Rate of performing tasks or processes– Time to react to events

• Accuracy based– Precision of performance– Reliability of performance– Completeness– Error rates– Quality of decisions (e.g. number of times decision changed)

12

CO B

P Human Behaviour

• Human performance– Psycho-physiological (stress, fatigue, hunger) – Ergonomic/external factors limiting performance– Individual psychology (risk propensity, decision style)

• Social interactions among individuals and groups– Interactive processes (shared awareness)– Background (cultural, educational, religious)

• Assessment Possibilities– Team competence – Team experience

13

CO B

P Normality Indicators in OOTW

• Assessment of changes occurring in civilian populace

• Characterize elements of the civil environment

– Data collected on regular basis– Inexperienced personnel– Limited resources

• Data calibrated against baselines– Relative measures

• Extrapolation across space and time

• Political & operational constraints• Effect of military presence• Shifts in emphasis and thresholds

Criterion ExamplesPolitical Elections, political participationEconomic Unemployment, interest rates, and market basketsSocial Number of students in schools, number of refugeesTechnological Telephone system availabilityLegal Judicial system functioningEnvironmental Roads, water supply, power supplyCultural Sports events, concerts

14

CO B

P Case Study MoMs

• Dimensional Parameters– Communications: Connectivity, Bandwidth– Collaborative Information Technology: Sophistication, Latency, and Security– ADP Support to Personnel: Quality, Flexibility

• Measures of C2 Performance– Ability to perform CIMIC tasks, functions (e.g.,time, number of people)

• Measures of C2 Effectiveness– Situational awareness– Synchronization of effort

• Measures of Force Effectiveness– Military: ability to create and sustain a secure environment

• Measures of Policy Effectiveness– Overall: Progress in transitioning from a failed to a stable state

• UNHCR: ability to relocate disposed refugees• IOs: ability to conduct fair elections

15

CO B

P Summary

• Linkage of DP - MoP - MoCE - MoFE - MoPE• Interpretation of MoMs• Environmental components and their effects• MoM reliability and validity• Characterization of uncertainties• Human-in-the-loop• Resources available versus cost and convenience

16

CO B

P

Questions?