Applying the Churchman/AckoffValue Estimation

Procedure to Spatial ModelingSusan L. Ose

MGIS Capstone PresentationPenn State University - World Campus

17 June 2008

GIS Models - A Refresher

• GIS models provide– Decision Support– Prediction– Cost assessment, etc.

• Weighting values– Emphasize/De-emphasize impact of

inputs– Model adjusted via tweaking weights

• How does an analyst determine weights?

Figure source: Joseph K. Berry, University of Denver

Estimating Weights - Gathering Expert Opinions via Delphi Process

?? CompileResults

Experts

Questionnaire

Discussion

• Anonymous Questionnaires

• Statistical aggregation

• Controlled feedback

• Avoids groupthink

• Consensus determines number of rounds

Questionnaire Format

• The ideal questionnaire:– Guides the user through the weight estimation

process– Focuses on expert knowledge - not statistics– Emphasizes relative not numeric values

• Software module– Stores and compiles results– Allows easy distribution to expert group– Can be used in online meeting

Churchman/Ackoff Procedure

• Focuses on relative value of layers – Compares layer against combination of other layers– Uses recursive procedures– Breaks down larger groups of layers for easier

assessment

• Recommended for group decision making• Project adapted method to browser-based

software module

Churchman/Ackoff Procedure Step 1: Rank Layers

Layers are dragged and dropped into desired order

Churchman/Ackoff Procedure Step 2: Assign Initial Weights

Layers divided subdivided into equal groups of no more than four.

Control layer randomly chosen, assigned value of 50, added to each group

Expert inputs initial estimate of weight

Group 1

Group 2

Churchman/Ackoff Procedure Step 3: Judge Importance

Expert chooses one of the three conditions and clicks on it.

Algorithm adjusts values accordingly.

Churchman/Ackoff Procedure :Results Screen

Procedure Test Case

• Potential for rain-fed agriculture in Liberia taking into account cost to market

• Expert panel consisted of MDA Federal, Inc. employees experienced in modeling

• Used two Delphi rounds – Compiled statistics – Ran model using average, high, and low values

Model Diagram

First Round Results• Reference layers received lowest scores• Proximity to water and land layers received highest

scores• Model output using average score and 8 highest

weighted layers consistent with results obtained using different methodology

• Discussion focused on experience with procedure• Panel agreed on top 7 layers to include in final model

Second Round Results• Results reflected post-first round discussion

– Access to water considered key, water layers grouped near top of ranking

– Land cover shows potential land to be transformed to agriculture, thus higher score than previous

• Higher variance in scores than in first round– Probably due to score distribution among less layers – First round lower variance may not have happened if

expert could discard layers

• Model more definitive around water features

Round Comparison

Model Results - Average

2nd round1st round

Does the method work?

• Model result viable • Considering one vs. many values more difficult than

considering one vs. one (pairwise method)– Deliberately designed that way to challenge one's opinions

– May frustrate participants - "weighting fatigue"

• Random groupings confused some participants– Difficult to recall original ranking, weight

– However randomness focuses judgment on subgroup

• Can assist individual in examining own conclusions

What happens when someone joins the group

later?• One expert did not attend discussions, used tool with

minimal guidance• First round results changed slightly• Average Z-score highest of the group

Second Round with Additional Person

• Pronounced difference in results• Greater difference in layer order in mid to lower layers• Average Z-score still highest of the group• Demonstrates importance of group discussion

 Z-ScoresNew

expert Expert 1 Expert 2 Expert 3 Expert 4 Expert 5

First Round 1.21 0.45 0.66 0.67 0.65 0.88

Second Round 1.20 0.46 0.83 0.88 0.55 0.77

Further Development/Study

• Improve the software tool– Rework the math algorithm that calculates the weights– Offer opportunity to discard layers– Provide comparison of start/finish results– Provide progress bar

• Application of the method– Categorical values within layers need to be rated first – Orientation meetings important

• Conduct an experiment using both this method and a pairwise approach and compare results

Acknowledgements

• Dr. Gregory Koeln, President, MDA Federal Inc. for sponsoring this project

• Michael Schreiber, Webmaster, MDA Federal Inc. for transforming the procedure from paper to software

• Dr. Todd Bacastow and Dr. Douglas Miller, Penn State World Campus, for their advice and guidance

• Dr. Douglas Way, MDA Federal Inc. for his invaluable guidance

• David Cunningham, Dr. Anna Oldak, Dr. François Smith, and Dr. Andrew Ralowicz, MDA Federal Inc. for their expertise

• Gregory A. Ose, for his unfailing support