Post on 25-Dec-2015
UNCLASSIFIED
Value Focused Metrics
Professor D.F. Davis
David.Davis@africom.mil
ddavis@gmu.edu
Presentation to 28 ISMOR
1 September 2011
UNCLASSIFIED 2
Metrics modeling: Using the metrics framework as the base for analysis.
All assessment/metrics/evaluations normally consist of a set of measurements that are aggregated to some overall meaning. This aggregation technique is what I’m referring to as a metrics model.
Definitions
UNCLASSIFIED 3
Most Frequent
Top down decomposition aggregated by Multi-Attribute Utilities
Most Important
Next Important Next Important Next Important
And so on….
1)(0,1,)()(1
xuwwherexuwxU ii
n
iii
UNCLASSIFIED 4
If and only if the decompositions are collectively exhaustive and mutually exclusive – cover every thing, doesn’t leave anything out, and are independent in definition.
Most common error is that they are nowhere near independent, and are often highly dependent. Of course, knowing that you have included all necessary elements is always risky.
But, they are easy to use and all you have to do is determine the weights.
However
UNCLASSIFIED 6
Value Focused Metrics
Use decomposition for only the highest level, fundamental or strategic elements (Fundamental Objectives Hierarchy or FOH) – be careful and don’t expect much.
Use a Means Ends Objectives Network (MEON) to represent the intermediate, dependent objectives and exploit the relationships. This is difficult because it requires that you graph your understanding of the inherent theory of change.
Identify, for some or all of the objectives, metrics that would indicate the level of achievement of that objective.
UNCLASSIFIED 7
Value Focused Metrics - 2
Objectives should be written as something you wish to achieve. If they can be caused to happen directly, they are tasks or actions, not objectives.
Therefore actions are something that you identify and execute with the hope that they will assist in the achievement of the objectives.
The MEON represents how you believe the objectives support one another and propagate the results of actions up to the achievement of the fundamental objectives.
UNCLASSIFIED 8
Fundamental Objective Hierarchy
Means Ends Objective Network
Controls, Actions,Programs, Projects
The System
ProcessMetrics
OutcomeMetrics
PerformanceMetrics
UNCLASSIFIED 9
The Somalia Game
A discovery game was held with multiple agencies of the US Government to look at the impact on US Africa Command’s End States and Objectives of various shocks or scenarios.
Impacts on objectives and actions to be taken were collected from the teams. Impacts were represented by specific expectations on metrics for individual objectives. Prevention and response actions were modeled by applying these actions to one or more objectives and developing likelihood expectations based on player input.
The actual game was held as For Official Use Only and is not currently fully releasable. However, the gist of the process can be seen in the following models and tables where the actual objective is represented by its coding identifier.
UNCLASSIFIED 10
Numbers
The MEON requires that each local model, an objective and it’s contributing (parent) objectives, have a conditional probability model. These models were developed in the Joint Planning Team for Somalia at the Command. Several sessions augmented by modeler subject mater expertise resulted in a fully described model for the FOH, MEON and the Metrics.
During the game, the players identified which objectives their actions would effect and to what degree and directions (negative, positive) that action would impact the objective. This information along with modeler input was used to provide the local models for prevention and response turns.
The model results were not used in the game, but were part of the post game analysis. The Control Group, or White Cell, led all in-game discussions.
UNCLASSIFIED 11
A Model Base Model
Fundamental Objectives Hierarchy
Means Ends Objectives
Network
Developed over six months and three continents: Africa – Nairobi and Djibouti, Europe – Stuttgart, US – Washington
Shows how the overall end states are achieved through a network of intermediate objectives.
UNCLASSIFIED 12
Base Model with Metrics
Concurrent to the development of the Base Model, each objective was addressed by asking the question: How would we know that the objective was achieved?
During the game, the impacts assessed by the player teams were reflected in values of the metrics.
UNCLASSIFIED 13
Base Model with Actions
During the game, there were two turns that addressed needed actions: Prevention and Response. These actions were identified by which objectives they impacted and with what magnitude and direction.
UNCLASSIFIED 14
Base Scenario 1a Impact Scenario 1b Impact Scenario 1 Prevention Scenario 1 Response
Strategic 0.488 0.314 0.399 0.802 0.938
F1 0.473 0.342 0.453 0.836 0.938
F2 0.524 0.248 0.273 0.721 0.939
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1.000
End State Achievment
Strategic
F1
F2
UNCLASSIFIED 15
M1_1 M1_2 M1_3 M1_4 M1_5 M1_6 M2_1 M2_2 M2_3
Base 0.200 0.200 0.067 0.129 0.159 0.173 0.577 0.368 0.381
Scenario 1 - Diplomacy 0.010 0.973 0.042 0.006 0.0002 0.131 0.373 0.014 0.026
Scenario 1 - Others 0.760 0.973 0.040 0.006 0.0002 0.190 0.379 0.014 0.041
Scenario 1 Prevention 0.751 0.963 0.212 0.683 0.673 0.911 0.684 0.831 0.540
Scenario 1 Response 0.984 0.976 0.937 0.985 0.728 0.923 0.896 0.952 0.902
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1.000
Intermediate Objective Achievment
Base
Scenario 1 - Diplomacy
Scenario 1 - Others
Scenario 1 Prevention
Scenario 1 Response
UNCLASSIFIED 16
BaseScenario 1 -Diplomacy Scenario 1 - Others
Scenario 1 Prevention Scenario 1 Response
M1_5 0.1593 0.0002 0.0002 0.6730 0.7277
0.0000
0.1000
0.2000
0.3000
0.4000
0.5000
0.6000
0.7000
0.8000
0.9000
1.0000
Critical Objective