2006 GoldSim Conference June 2006 Neptune and Company, Inc. Decision Analysis in GoldSim Paul Black...

June 20062006 GoldSim Conference

Neptune and Company, Inc.

Decision Analysis in GoldSimDecision Analysis in GoldSim

Paul BlackJohn Tauxe

Ralph PeronaTom Stockton

http://www.neptuneandco.com


Presentation OutlinePresentation Outline

• Decision Analysis Basics• Background

• Some greek!

• Example context

• Simple example in GoldSim

• Add some uncertainty

• Smoky Site

• NTS low-level waste sites


Decision Analysis OverviewDecision Analysis Overview

• “Formalized common sense”

• A set of tools for structuring and analyzing complex decision problems

• An approach for making logical, reproducible, and defensible decisions in the face of:• Technical complexity

• Uncertainty

• Costs and value judgments

• Multiple, competing objectives


Decision Analysis or….Decision Analysis or….

• Multi-Attribute Utility Theory (MAUT)

• Cost/benefit analysis

• Multi-Criteria Decision Analysis - MCDA

• Probabilistic modeling

• Deterministic analysis does not support decision analysis in the face of uncertainty, which is really how all decisions are made.


Decision Analysis OverviewDecision Analysis Overview

• In the long run, you will be better off if you choose the alternative (decision option) that gives you the best expected outcome, given what you know or believe about future events

• Expectation implies uncertainty• Maximize Expected Utility

• Minimize Expected Losses


Decision Problems have 3 Decision Problems have 3 Basic ComponentsBasic Components

Alternative 1

Alternative 2

Alternative 3

Alternative 4

What you can do

Decision options & reward structure

Low

Nominal

High

What you know

Uncertainties

What you want

Objectives

MaximizeFinancialReturns

Minimize Health & Safety

Impacts

MaximizeCustomer

Satisfaction


Decision Analysis – “Greek”Decision Analysis – “Greek”

• Event or outcome space, W, with events, w

• Probability distribution, P(w)

• Decision space, D, with decision options, d

• Utility functions U(w, d), or Loss functions, L(w, d). Note L(w, d) = - U(w, d)

• Expected Utility = U(w,d)*P(w)

• Objective is to find the decision option for which Expected Utility (EU) is Maximized


Roots of Decision AnalysisRoots of Decision Analysis

• Decision Analysis established as an applied discipline and a field of research in the late 1960’s

• Howard Raiffa (Harvard)• emphasis on decision analysis as a method with

real world applications

• Ron Howard (Stanford)• emphasis on influence diagrams and economic

analyses in the face of uncertainty


Roots of Decision AnalysisRoots of Decision Analysis

• Bayesian probability theory (Bayes, mid-1700s)

• Utility theory (von Neumann & Morgenstern, 1947)

• Bayesian Statistical Decision Theory (de Finetti, 1930s, Savage, 1954, DeGroot, 1970)

• Behavioral Science (von Winterfeldt and Edwards, 1986)

• Policy Analysis (Morgan and Henrion, 1990)


Common Application AreasCommon Application Areas

• Oil and gas industry

• Risk analysis (business decision risk)

• Pharmaceutical and biotechnology industries

• Public sector applications• Department of Defense

• Environmental – moving in this direction


Environmental EvolutionEnvironmental Evolution

• Compliance with deterministic models• Difficult to overcome inertia and intransigence in

the industry (old dogs - new tricks)

• Difficult to overcome established regulations

• Changes at the top-level take a long time to trickle down through Regions and States

• Strong evidence of an evolutionary change• OMB, EPA SAB, EPA CREM, NUREG, SRA

• Impact of changes in education system


Simple ExampleSimple Example

• Planning our wedding on Red Lodge Mountain

• Options for the ceremony include:• Indoors in the ski chalet, d1

• On the covered deck, d2, and,

• On the mountain, d3

• The decision is impacted by:• The chance of rain, Prob(rain), and

• The likely temperature, Prob(temperature)

• Rain and temperature are the w’s.



• If it is a nice day out, we would prefer to be outside on the mountain (e.g., dry and hot)

• If it pours and it is cold, we would prefer to be inside

• If it rains, but it is still warm, we will use the covered deck area.

• The decision needs to be made far ahead of time based on our best understanding of weather in Montana in mid-August.


Simple Example - ProbabilitiesSimple Example - Probabilities

• To keep things simple we set up:• P(rain) = 0.2

• P(dry) = 1 – P(rain) = 0.8

• 3 categories for temperature• Prob(cold) = 0.2

• Prob(warm) = 0.5

• Prob(hot) = 0.3

• The probability space has 6 possible events


Simple Example – Utilities ($)Simple Example – Utilities ($)Indoors Cold Warm Hot

Rain 5,000 5,000 5,000

Dry 4,000 4,000 4,000

Deck Cold Warm Hot

Rain 2,000 2,000 3,000

Dry 5,000 9,000 6,000

Outdoors Cold Warm Hot

Rain (base) 0 0 0

Dry 6,000 10,000 7,000


Simple Example - CalculationsSimple Example - Calculations

• E(U|Indoors) = P(rain&cold)*U(Indoors|rain&cold)+ P(rain&warm)*U(Indoors|rain&warm)+P(rain&hot)*U(Indoors|rain&hot)+P(dry&cold)*U(Indoors|dry&cold)+ P(dry&warm)*U(Indoors|dry&warm)+P(dry&hot)*U(Indoors|dry&hot)

• I.e., sum the probability of each event multiplied by the utility of the decision option given each event.



• E(U|Indoors) = $4,200

• E(U|Deck) = $6,300

• E(U|Outdoors) = $6,600

• Choose the Outdoors option (the decision option that provides the greatest benefit)

• Depends on the input probabilities for rain and temperature

• Example in GoldSim


Simple Example – Expansion?Simple Example – Expansion?

• No uncertainty built in – no simulation needed – no global sensitivity analysis possible• Uncertainty in the P(rain)?

• Continuous temperature (probability distribution)

• Value of further information?• Should we continue to track the weather forecast to

get better information?

• How much would it be worth if we could control the weather (just for that day)?


Simple Example – Expansion?Simple Example – Expansion?

• P(rain) follows a Beta distribution

• Temperature follows a Normal Distribution

• GoldSim model

• Is this a better model?

• Yes, because uncertainty is included• Allows global SA

• Allows value of information to be assessed

• Don’t make the decision until you have enough information to make it.


Expanded Example – Results?Expanded Example – Results?

• E(U|Indoors) (4497, 4500, 4510)

• E(U|Deck) (5790, 5850, 5900)

• E(U|Outdoors) (3930, 3990, 4050)

• Choose to hold our wedding on the Deck

• No overlap of ranges, so decision can probably be made without collecting more information

• Uncertainty could be introduced into utilities.


Decision Analysis in Decision Analysis in PracticePractice

• Ten Commandments (Morgan and Henrion)

• Model building• transparency, traceability, reproducibility, peer

review – “simple as possible but no simpler”

• Sensitivity Analysis (Model Evaluation)

• Value of Information (do we need more data?)

• Data collection

• (Bayesian) Updating

• Iterate until no value in collecting more


Decision Analysis in GoldSimDecision Analysis in GoldSim

• Decision Analysis math is basically simple, so it can be done in GoldSim.

• GoldSim interface allows decision models to be developed using the 10 commandments.

• However, GoldSim interface does not:• Build pure influence diagrams directly (formality issues)

• Does not handle statistical modeling easily (units issue)

• Does not handle Bayesian updating (need MCMC)

• Does not do global SA (needs to be done externally)


When to use DAWhen to use DA

• When decisions are not trivial• Obvious outcome (e.g., drive to work or not)

• Trivial topic (e.g., which egg to boil)

• I.e., when decisions are hard or difficult, e.g.,• Which house to buy (perhaps)

• Business decisions (re-organization (e.g., GTG), mergers, acquisitions, contracts)

• Political decisions (war, health care, immigration)

• Environment (brownfields, DOE’s ALARA, NEPA)


So, What is Decision Analysis?So, What is Decision Analysis?

• An overall approach for making decisions that is:• Rational• Logical• Reproducible (transparent and traceable)• Defensible

• In the face of:• technical complexity• uncertainty, and• multiple, possibly competing, objectives.

• A set of tools for structuring and analyzing complex decision problems


Applying Decision AnalysisApplying Decision Analysis

• Identify objectives, decision options, and events that define the decision analysis

• Clearly communicate judgments about utilities (costs and value judgments), uncertainty (probabilities), and risks (EU) in an unambiguous way

• Actively involve stakeholders, customers or users of the decision model at all stages of the decision analysis process (instead of only at later stages, which is more typical)


More Complex ExampleMore Complex Example

• The Smoky Site at the Nevada Test Site

• Multi-disciplinary work, involved a technical team

• Model development – assumptions, structure, quantification – all with complete documentation

• Top-down GoldSim model focused on the Decision Analysis needs (as simple as possible but no simpler)

• Results

• Decision outcome


Smoky Site BackgroundSmoky Site Background

• The Smoky Site at the NTS was used to conduct safety shots and atmospheric tests of nuclear devices.

• Long term maintenance of power lines that cross the Smoky Site are of concern for worker safety.

• Current contamination levels require investigation under human health and environmental protection (DOE O 5400.5) and occupational exposure (10 CFR 835).


Smoky Site Decision FrameworkSmoky Site Decision Framework

Remediation Depth and Area

Aerial Survey Concentrations

Member of Public

Exposure

Site Access Control

Transport Control

Access Control Costs

Transport Control Costs

Residual Concentrations

Cleanup Costs

Price-Anderson

Fines

Occupational Exposure

Member of Public

Exposure

ALARA Dose Costs

Population Dose

Dose Threshold

Total Cost

Decision Option

Individual Dose


ScrapeHCA [ $99.4M ]ScrapeWash+MoveLine [ $83M ]

ScrapeWash+MoveLine+PostHCA [ $82.8M ]ScrapeWash+MoveLine+FenceHCA [ $82.6M ]

ScrapeWash+ScrapeLine [ $82.6M ]ScrapeWash+ScrapeLine+FenceHCA [ $82.2M ]

ScrapeWash+ScrapeLine+PostHCA [ $82M ]ScrapeWash [ $81.7M ]

ScrapeWash+PostHCA [ $81.5M ]ScrapeWash+Causeway [ $81.3M ]

ScrapeWash+FenceHCA [ $81.3M ]ScrapeWash+Causeway+PostHCA [ $81.2M ]ScrapeWash+Causeway+FenceHCA [ $81M ]

ScrapeHCAlessGZ [ $76.2M ]ScrapeHCAlessGZ+PostGZ [ $75.9M ]

ScrapeHCAlessGZ+FenceGZ [ $75.8M ]ScrapeHCAintoGZ [ $20.5M ]

ScrapeHCAintoGZ+PostGZ [ $20.2M ]ScrapeHCAintoGZ+FenceGZ [ $20.1M ]

MoveLine+Basin [ $7.1M ]MoveLine+Basin+PostHCA [ $6.8M ]

ScrapeLine+Basin+PostHCA [ $6.6M ]ScrapeLine+Basin [ $6.6M ]

MoveLine+Basin+FenceHCA [ $6.5M ]ScrapeLine+Basin+FenceHCA [ $6.4M ]ScrapeLine+Basin+Brdgs+Post [ $6.3M ]

ScrapeLine+Basin+Bridge [ $6.3M ]MoveLine [ $6.1M ]

ScrapeLine+Basin+Brdgs+Fence [ $6M ]Basin [ $5.8M ]

MoveLine+PostHCA [ $5.8M ]ScrapeLine+PostHCA [ $5.7M ]

ScrapeLine [ $5.6M ]MoveLine+FenceHCA [ $5.5M ]

Basin+PostHCA [ $5.5M ]ScrapeLine+Bridges+PostHCA [ $5.4M ]

ScrapeLine+FenceHCA [ $5.3M ]ScrapeLine+Bridges [ $5.3M ]

Basin+FenceHCA [ $5.2M ]ScrapeLine+Bridges+FenceHCA [ $5.1M ]

NCA [ $4.8M ]ScrapeLine+C+B+PostHCA [ $4.7M ]

ScrapeLine+Causeway+Basins [ $4.6M ]PostHCA [ $4.5M ]

ScrapeLine+C+B+FenceHCA [ $4.4M ]FenceHCA [ $4.2M ]

Causeway+Basin+PostHCA [ $3.9M ]Causeway+Basin [ $3.8M ]

ScrapeLine+Causeway+PostHCA [ $3.7M ]ScrapeLine+Causeway [ $3.7M ]

Causeway+Basin+FenceHCA [ $3.6M ]ScrapeLine+Causeway+FenceHCA [ $3.4M ]

Causeway+PostHCA [ $2.9M ]Causeway [ $2.9M ]

Causeway+FenceHCA [ $2.6M ]

1 2 5 10 20 50 100Cost (M$)

Decision Options • CostDecision Options • Cost


0 20 40 60 80 100

ScrapeHCAScrapeWash+MoveLine

ScrapeWash+MoveLine+PostHCAScrapeWash+MoveLine+FenceHCA

ScrapeWash+ScrapeLineScrapeWash+ScrapeLine+FenceHCA

ScrapeWash+ScrapeLine+PostHCAScrapeWash

ScrapeWash+PostHCAScrapeWash+Causeway

ScrapeWash+FenceHCAScrapeWash+Causeway+PostHCA

ScrapeWash+Causeway+FenceHCAScrapeHCAlessGZ

ScrapeHCAlessGZ+PostGZScrapeHCAlessGZ+FenceGZ

ScrapeHCAintoGZScrapeHCAintoGZ+PostGZ

ScrapeHCAintoGZ+FenceGZMoveLine+Basin

MoveLine+Basin+PostHCAScrapeLine+Basin+PostHCA

ScrapeLine+BasinMoveLine+Basin+FenceHCA

ScrapeLine+Basin+FenceHCAScrapeLine+Basin+Brdgs+Post

ScrapeLine+Basin+BridgeMoveLine

ScrapeLine+Basin+Brdgs+FenceBasin

MoveLine+PostHCAScrapeLine+PostHCA

ScrapeLineMoveLine+FenceHCA

Basin+PostHCAScrapeLine+Bridges+PostHCA

ScrapeLine+FenceHCAScrapeLine+Bridges

Basin+FenceHCAScrapeLine+Bridges+FenceHCA

NCAScrapeLine+C+B+PostHCA

ScrapeLine+Causeway+BasinsPostHCA

ScrapeLine+C+B+FenceHCAFenceHCA

Causeway+Basin+PostHCACauseway+Basin

ScrapeLine+Causeway+PostHCAScrapeLine+Causeway

Causeway+Basin+FenceHCAScrapeLine+Causeway+FenceHCA

Causeway+PostHCACauseway

Causeway+FenceHCA

Costs (M$)

Transport.ControlSite.Access.ControlCleanupPrice.Anderson.FinesALARA.Dose

Median Cost BreakdownMedian Cost Breakdown


Temporal Cost BreakdownTemporal Cost Breakdown

Year

Co

st (

M$

)

0 10 20 30 40 50

0

2

4

6

NoCleanup

BasinBasin+PostBasin+FenceNoControlPostFenceCauseway+Basin+PostCauseway+BasinCauseway+Basin+FenceCauseway+PostCausewayCauseway+Fence


The completed causeway


Smoky Site Concluding NotesSmoky Site Concluding Notes

• Credibility gained through peer review and thorough documentation (transparency, traceability)

• Top-down GoldSim model focused on the Decision Analysis needs (as simple as possible but no simpler)• E.g., no fate and transport model – from a Decision

Analysis perspective it was not needed

• Iteratively refined problem statement and model• E.g., 5400.5 vs. 835.1 issues

• Explicitly defined decision options, costs and uncertainties


An Example from the Nevada Test SiteAn Example from the Nevada Test Site

Area 5 Radioactive Waste Management Site • Photo courtesy NNSA/NSO


Even More Complex ExampleEven More Complex Example

• Low-level Radioactive Waste Management Sites (RWMS) at the Nevada Test Site

• Decision Objectives:

• Optimize future disposal

• Optimize closure design

• Long term management

• Optimize monitoring program with stopping rules


DOE Performance AssessmentsDOE Performance Assessments• Establish “reasonable expectation” that performance

objectives are not exceeded (e.g. DOE M 435.1), in order to authorize waste disposal.

• Compliance-based decision making

• PAs are traditionally deterministic and “conservative”, yet there are inherent uncertainties in assumptions, parameter values, and in the models themselves.

• ALARA (as low as reasonably achievable – 5400.5) offers a regulatory path to performing decision analysis instead.


Costs and Value JudgmentsCosts and Value Judgments• Disposal costs (material costs, depth of

disposal, containerization)

• Closure costs (material costs, cover thickness, institutional controls)

• Management costs (maintenance, institutional controls)

• ALARA costs (related to receptor population doses)

• Other values (ecological, stakeholder concerns)


PAs and UncertaintyPAs and UncertaintySources of uncertainty in PA modeling

include

• conceptual model assumptions and exposure scenarios,

• analytical and numerical models and their assumptions, and

• model input parameters in space and time (variability and knowledge uncertainty).


Transport Model UncertaintiesTransport Model Uncertainties

• The conceptual model of transport at the Area 5 Radioactive Waste Management Site at the Nevada Test Site includes:

• upward flux of water driven by high evapotranspiration potentials,

• diffusion in liquid and gaseous phases (radon),

• biotic transport of contamination and materials in the near surface, and

• resuspension and air dispersion.


GoldSim at the NTSGoldSim at the NTS


Important Results to dateImportant Results to date• GoldSim provides a forum for thorough

documentation (transparency, traceability)

• Explicit about decision options, costs and uncertainties

• Perform sensitivity analysis on each iteration of the model

• Iterate each time we collect data/information

• DOE peer review has resulted in a fast track acceptance of the PA results.


Areas that need workAreas that need work• Bayesian updating as more data are collected

• Accommodating statistical/regression models (empirical vs. mechanistic – units issue)

• Correlation structures (also a statistical issue)

• Model simplification (using sensitivity analysis, and Kalman filtering) could aid decision analysis

• Explaining the importance of distribution averaging to match the spatial/temporal scale of the model/problem


RWMS Concluding NotesRWMS Concluding Notes

• A very large model driven by:• Historical modeling

• Compliance-based regulation

• Perception of the need for a “complete” model creates challenges for model simplification

• ALARA will help us refine the model because of the decision analysis context

• We have been fought every step of the way, overcoming each obstacle in turn.

• It has been critical to have an advocate at DOE.

2006 GoldSim Conference June 2006 Neptune and Company, Inc. Decision Analysis in GoldSim Paul Black...

Documents

Transcript of 2006 GoldSim Conference June 2006 Neptune and Company, Inc. Decision Analysis in GoldSim Paul Black...