Parameter Estimation: A Glimpse of Backing Out, Sensitivity Analysis ... · Sensitivity in Initial...
Transcript of Parameter Estimation: A Glimpse of Backing Out, Sensitivity Analysis ... · Sensitivity in Initial...
Parameter Estimation: A Glimpse of Backing Out, Sensitivity
Analysis & Meta-Analysis
Nathaniel Osgood
CMPT 858
Sources for Parameter Estimates
• Surveillance data
• Controlled trials
• Outbreak data
• Clinical reports data
• Intervention outcomes studies
• Calibration to historic data
• Expert judgement
• Systematic reviews
Content
redacted
for
copyright
compliance
Anderson & May
Introduction of Parameter Estimates
Non Obese
General
Population
Undx Prediabetic
Popn
Obese General
Population
Becoming Obese
Dx Prediabetic Popn
DevelopingDiabetesBeing Born Non
Obese Being Born At
Risk
Annual Likelihood of
Becoming Obese
Annual Likelihood of
Becoming Diabetic
Diagnosis of
prediabetics
undx uncomplicated
dying other causes
dx uncomplicated
dying otehr causes
Annualized ProbabilityDensity of prediabetic
recongnition
Non-Obese
Mortality
Annual Mortality Rate for
non obese population
Annualized MortalityRate for obese
population
<Annual Not at
Risk Births>
Annual Likelihood ofNon-Diabetes Mortality forAsymptomatic Population
<Annual at Risk
Births>
Obese Mortality
Dx Prediabetics
Recovering
Undx Prediabetics
Recovering
Annual Likelihood ofUndx Prediabetic
Recovery
Annual Likelihood of Dx
Prediabetic Recovery
<Annual Likelihood ofNon-Diabetes Mortality forAsymptomatic Population>
Hassmiller, K. M. (2006). The Impact of Smoking on Population-Level Tuberculosis Outcomes. Unpublished PhD, University of Michigan, Ann Arbor, Ann Arbor.
Preparation for Pooling
Hassmiller, K. M. (2006). The Impact of Smoking on Population-Level Tuberculosis Outcomes. Unpublished PhD, University of Michigan, Ann Arbor, Ann Arbor.
Hassmiller, K. M. (2006). The Impact of Smoking on Population-Level Tuberculosis Outcomes. Unpublished PhD, University of Michigan, Ann Arbor, Ann Arbor.
Example of Other Pooling
Hassmiller, K. M. (2006). The Impact of Smoking on Population-Level Tuberculosis Outcomes. Unpublished PhD, University of Michigan, Ann Arbor, Ann Arbor.
Pooled Results
Hassmiller, K. M. (2006). The Impact of Smoking on Population-Level Tuberculosis Outcomes. Unpublished PhD, University of Michigan, Ann Arbor, Ann Arbor.
Forest Plot
Hassmiller, K. M. (2006). The Impact of Smoking on Population-Level Tuberculosis Outcomes. Unpublished PhD, University of Michigan, Ann Arbor, Ann Arbor.
Forest Plot 2
Hassmiller, K. M. (2006). The Impact of Smoking on Population-Level Tuberculosis Outcomes. Unpublished PhD, University of Michigan, Ann Arbor, Ann Arbor.
Sensitivity Analyses
• Same relative or absolute uncertainty in different parameters may have hugely different effect on outcomes or decisions
• Help identify parameters that strongly affect – Key model results
– Choice between policies
• We place more emphasis in parameter estimation into parameters exhibiting high sensitivity
Types of Sensitivity Analyses
• Variables involved – One-way
– Multi-way
• Type of component being varied – Parameter sensitivity
analysis: Parameter values
– Structural sensitivity analysis: Examine effects of Model structure on results
• Type of variation – Single alternative values
– Monte Carlo analyses: Draws from probability distributions (many types of variations)
• Frequency of variation – Static (parameter retains
value all through simulation)
– Ongoing change: Stochastic process • Accomplished via Monte-Carlo
analyses
Example Spider Diagram
Content
redacted
for
copyright
compliance
• Each axis represents a % change in a particular parameter – This proportional change
is identical for the different parameters
• The distance assumed by the curve along that axis represents the magnitude of response to that change – Note that these
sensitivities will depend on the state of system!
http://www.niwotridge.com/images/BLOGImages/SpiderDiagram.jpg
Systematic Examination of Policies
08-13
Female14-18 19-24 25-34 35-44 45-54 55-64 65+ 08-13
Male14-18
19-24 25-3435-44 45-54
55-6465+
Cessation
Relapse
Uptake
0
50000
100000
150000
200000
250000
300000
350000
Cessation
Relapse
Uptake
Apply Different Uptake Rates for Children of Mothers who Smoked During Pregnancy? No linkage Time 2050
Sum of Total QALYs
InterventionGender Intervention Age Category
Smoking Status of Intervention Target
Tengs, T., Osgood, N., & Lin, T. (2001). Public health impact of changes in smoking behavior: results from the Tobacco Policy Model. Medical Care, 39(10), 1131-1141.
Sensitivity in Initial Value
• Frequently we don’t know the exact state of the system at a certain point in time
• A very useful type of sensitivity analysis is to vary the initial value of model stocks
• In Vensim, this can be accomplished by
– Indicating a parameter name within the “initial value” area for a stock
– Varying the parameter value
Imposing a Probability Distribution Monte Carlo Analysis
• We feed in probability distributions to reflect our uncertainty about one or more parameters
• The model is run many, many times (realizations)
– For each realization, the model uses a different draw from those probability distribution
• What emerges is resulting probability distribution for model outputs
Example Resulting Distribution
Empirical Fractiles
Impact on cost of uncertainty regarding mortality and medical costs
50% 60% 70% 80% 90% 95% 97% 99%
Incremental Costs
6 B
4.5 B
3 B
1.5 B
0 2001 2014 2026 2039 2051
Time (Year)
Static Uncertainty
Multi-Way Sensitivity Analyses
• When examining the results of changing multiple variables, need to consider how multiple variables vary together
• If this covariation reflects dependence on some underlying factor, may be able to simulate uncertainty in underlying factor
Performing Monte Carlo Sensitivity Analyses in Vensim
• Need to specify three things
– The parameters to vary
– How to vary those parameters
– Which model variables to save away
How & What Parameters to Vary
Model Values to Save Away
Monte Carlo Analyses
Sensitivity Results (Prevalence)
Sensitivity Results (Fraction of Susceptibles)
Stochastic Processes • Examples of things stochastically approximated
– Stock market
– Rainfall
– Oil prices
– Economic growth
• What considered “stochastic” will depend on the scope of the model
– Detailed model: Individual behaviour, transmission, etc.
– A meteorological model may not consider rainfall stochastic
Stochastic Processes
Baseline50% 60% 70% 80% 90% 95% 98% 100%
Average Variable Cost per Cubic Meter
0.6
0.45
0.3
0.15
00 1457 2914 4371 5828
Time (Day)
Dealing with Data Gradients • Often we don’t have reliable information on some
parameters, but do have other data – Some parameters may not be observable, but some
closely related observable data is available
– Sometimes the data doesn’t have the detailed breakdown needed to specifically address one parameter • Available data could specify sum of a bunch of flows or stocks
• Available data could specify some function of several quantities in the model (e.g. prevalence)
• Some parameters may implicitly capture a large set of factors not explicitly represented in model
• There are two big ways of dealing with this: manually “backing out”, and automated calibration
“Backing Out”
• Sometimes we can manually take several aggregate pieces of data, and use them to collectively figure out what more detailed data might be
• Frequently this process involves imposing some (sometimes quite strong) assumptions – Combining data from different epidemiological
contexts (national data used for provincial study)
– Equilibrium assumptions (e.g. assumes stock is in equilibrium. Cf deriving prevalence from incidence)
– Independence of factors (e.g. two different risk factors convey independent risks)
Example
• Suppose we seek to find out the sex-specific prevalence of diabetes in some population
• Suppose we know from published sources – The breakdown of the population by sex (cM, cF)
– The population-wide prevalence of diabetes (pT)
– The prevalence rate ratio of diabetes in women when compared to men (rrF)
• We can “back out” the sex-specific prevalence from these aggregate data (pF, pM)
• Here we can do this “backing out” without imposing assumptions
Backing Out
# male diabetics + # female diabetics = # diabetics
(pM* cM) + (pF* cF) = pT*(cM+cF)
• Further, we know that pF / pM =rrF=> pF = pM * rrF
• Thus
(pM* cM) + ((pM * rrF)* cF) = pT*(cM+cF)
pM*(cM + rrF* cF) = pT*(cM+cF)
• Thus
– pM = pT*(cM+cF) / (cM + rrF* cF)
– pF = pM * rrF = rrF * pT*(cM+cF) / (cM + rrF* cF)
Disadvantages of “Backing Out”
• Backing out often involves questionable assumptions (independence, equilibrium, etc.)
• Sometimes a model is complex, with several related known pieces
– Even thought we may know a lot of pieces of information, it would be extremely complex (or involve too many assumptions) to try to back out several pieces simultaneously
Another Example: Joint & Marginal Prevalence
Rural Urban
Male pMR pMU pM
Female pFR pMU pF
pR pU
Perhaps we know •The count of people in each { Sex, Geographic } category •The marginal prevalences (pR, pU , pM , pF)
We need at least one more constraint
•One possibility: assume pMR / pMU = pR/ pU We can then derive the prevalences in each { Sex, Geographic } category
Example Tying Together Meta-analysis & Calibration
Hassmiller, K. M. (2006). The Impact of Smoking on Population-Level Tuberculosis Outcomes. Unpublished PhD, University of Michigan, Ann Arbor, Ann Arbor.