Introduction to Computer Experiments€¦ · Introduction Example Conclusions Computer Experiments...

IntroductionExample

Conclusions

Introduction to Computer Experiments

Thomas Santner

Department of StatisticsThe Ohio State University

Columbus, Ohio

January 12, 2012

TJ Santner Introduction to Computer Experiments

IntroductionExample

Conclusions

Outline

Introduction

Example

Conclusions


IntroductionExample

Conclusions

Overview

This talk will describe• The use of computer simulators as experimental tools.• To study the output of a complex computer simulator, arapidly-computable emulator of the output of the simulator isordinarily used. Such an emulator, sometimes called a metamodel,• A method for assessing their prediction uncertainty.• Emulators are the basis for the construction of (1) criteria-basedexperimental designs, (2) for calibration methodology, and (3) asmotivation for extensions of the model that allow prediction whenthe inputs are both quantitative and qualitative.


IntroductionExample

Conclusions

Experiments

◮ Physical Experiments◮ Gold standard for establishing cause and effect relationships◮ Mainstay of Agriculture, Engineering, Medicine◮ Phy. Exps. Motivated Many Methodological

Developments

1. Reduce measurement errors More precise estimates oftreatment differences can be made when measurement errorsare smallera. Use instruments that maintain reproducibility overtechnicians, measurement conditionsb. Use genetically similar laboratory animalsc. Use common interviewer (sample surveys)


IntroductionExample

Conclusions

Experiments

2. Block (group) experiment units to otherwise similarexperiment units that are alike except for the treatmentsapplied (to account for recognized nuisance factors)

2.1 Paired comparison studies in taste tasting2.2 Twin studies

3. Randomization over time or space (to account forunrecognized nuisance factors)

4. Correct choice of sample size (the most elementary form ofexperimental design)

5. Use Stochastic Models that Reflect Experimental Conditions(that reflect different types of “measurement errors”, lack ofrandomization, nature of blocking variables)


IntroductionExample

Conclusions

Experiments

◮ Physical Experiments

◮ Stochastic Simulation Experiments Complex physicalsystem each of whose parts behave in a stochastic manner butwhose ensemble behavior is not understood analytically.Heavily used in Industrial Engineering and OperationsResearch–e.g., compare job shop set ups


IntroductionExample

Conclusions

Experiments


◮ Stochastic Simulation Experiments

◮ Computer Experiments Use a computer simulator to relateinputs/outputs rather than a physical experiment. In use forat least 15-20 years; most methodological developments in thepast 10 years


IntroductionExample

Conclusions

Experiments


◮ Simulation Experiments

◮ Computer Experiments

◮ Combinations of the above particularly ComputerExperiments + Physical Experiments


IntroductionExample

Conclusions

Computer Experiments

◮ Sometimes it is not feasible to perform a physical experiment

1. Too expensive to study directly (too many input variables,physical process is technically too difficult, . . . )

2. Ethical considerations

◮ When physical experiments are not possible, it may still befeasible to conduct a computer experiment


IntroductionExample

Conclusions


◮ IF the physical process relating the inputs to the response(s)

a. Can be described by a mathematical model relating theoutput, y(x), to the inputs x,

b. Numerical methods exist for solving the mathematical model,c. The numerical methods can be implemented with computer

code (in reasonable time!)

THEN one can run the computer code to produce a“response” y(x) at any input x, i.e., one can conduct acomputer experiment

◮ Mathematical models often coupled systems of PDEs

◮ Numerical methods FE, CFD algorithms

◮ Running Times seconds to months


IntroductionExample

Conclusions

Mathematical Models

• Consider dropping an object from a height h (≡ the input) andmeasuring the time until it hits the ground (≡ y(h)).


IntroductionExample

Conclusions

Mathematical Models

• Newton Physics Model for y c(h) Let s(t) denote the heightof the object at time t. Then y c(h) = τ where τ is the solution ofs(τ) = 0 and s(t) satisfies (the equation of motion)

s2(t)

dt2= −g

subject to the boundary conditions s(0) = h and s(t)dt

∣

∣

∣

t=0= 0.

• Newton Physics Model with Drag y c(h) = τ where τ is thesolution of s(τ) = 0 and s(t) satisfies

s2(t)

dt2= −g + c ×

s(t)

dt

subject to s(0) = h and s(t)dt

∣

∣

∣

t=0= 0 (Note: s(τ, c) = 0 ??)


IntroductionExample

Conclusions


The output of a computer simulator relating x and y(x) can beviewed as a black-box process

x −→ Simulator −→ y(x)

(The computer code is a proxy for the physical process.)


IntroductionExample

Conclusions

Inputs to a Computer Experiments

• Types of Inputs x = (xd , xe , xc , xt)

◮ xd ≡ engineering design (manufacturing, treatment,control) variables


IntroductionExample

Conclusions




◮ xe ≡ noise (field, environmental) input variables


IntroductionExample

Conclusions





◮ xc ≡ calibration (model) variables – if observational orexperimental data are available in addition to code outputs,then xc inputs are those code inputs whose values in theobserved data are unknown (eg, friction, rates of metabolism,rate of expansion of the galaxy)


IntroductionExample

Conclusions






◮ xt ≡ tuning parameters are present only in the computercode–they are used to make the bias in the computer outputas small as possible.


IntroductionExample

Conclusions






◮ xt ≡ tuning parameters are present only in the computercode–they are used to make the bias in the computer outputas small as possible.

(Usually only some of the xd ,xe ,xc ,xt types are present inapplications)


IntroductionExample

Conclusions

Output of a Computer Experiments

The output of a Computer Experiment has the following features• y(x) is deterministic


IntroductionExample

Conclusions


The output of a Computer Experiment has the following features• y(x) is deterministic• y(x) may be biased for the physical relationship that it issupposed to describe (inclomplete physics, numerical issues)


IntroductionExample

Conclusions


The output of a Computer Experiment has the following features• y(x) is deterministic• y(x) may be biased for the physical relationship that it issupposed to describe (inclomplete physics, numerical issues)• Traditional DoE principles are irrelevant –no nuisance orunrecognized factors.


IntroductionExample

Conclusions


The output of a Computer Experiment has the following features• y(x) is deterministic• y(x) may be biased for the physical relationship that it issupposed to describe (inclomplete physics, numerical issues)• Traditional DoE principles are irrelevant –no nuisance orunrecognized factors.• Sometimes output from an associated physical experiment is alsoavailable, but sometimes


IntroductionExample

Conclusions


The output of a Computer Experiment has the following features• y(x) is deterministic• y(x) may be biased for the physical relationship that it issupposed to describe (inclomplete physics, numerical issues)• Traditional DoE principles are irrelevant –no nuisance orunrecognized factors.• Sometimes output from an associated physical experiment is alsoavailable, but sometimes

1. Physical experiments are available only for components ofthe ensemble process, eg, code that emulates an auto crashtest.

2. Experiments that only approximate reality are available,e.g., Instron-Stanmore knee simulator

3. Only observational data are available, e.g., In Cosmology –only SDSS data


IntroductionExample

Conclusions


• In practice

◮ Real-valued y(x)


IntroductionExample

Conclusions


• In practice


◮ Multivariate (y1(x), . . . , yk(x))


IntroductionExample

Conclusions


• In practice



◮ Functional (t, y(t, x))


IntroductionExample

Conclusions


• In practice




• Target Field Conditions Xe ∼ πe(·) may be given or can besolicited


IntroductionExample

Conclusions


• In practice




• Target Field Conditions Xe ∼ πe(·) may be given or can besolicited• Prior Information Regarding calibration parametersXc ∼ πc(·) may be known


IntroductionExample

Conclusions


• Our interest in settings where

1. Few computer runs are possible - codes are complex e.g.,fine-grid FEA codes

2. High-dimensional input x


IntroductionExample

Conclusions

Conceptualizing Experimental Output

• Contrasting output from a physical experiment (orobservation study) with output from a computer experiment• Output obtained from a physical experiment is a noisymeasurement of the true input-output relationship, i.e.,

yP(x) = µT (x) + ǫ(x)

1. x −→ µT (x) ≡ true input-output relationship

2. {ǫ(x)}x ≡ measurement error (often modeled as i.i.d N(0, σ2ǫ)

a.k.a. white noise)


IntroductionExample

Conclusions

Conceptual Thoughts Computer Experiments

• Output from a computer experiment is a possibly biaseddescription of the true input-output relationship (inadequatephysics, biology, . . . )

y c(x) = µT (x) + δ(x)

where

1. δ(x) ≡ computer model bias

2. µT (x) is the true input-output relationship


IntroductionExample

Conclusions

A Classification of Problems

• Interpolation/Prediction Given output of computer code at aset of training inputs,

(xt1, yc(xt1)), . . . (x

tm, y

c(xtm))

predict y c(·) at a new input x0. An extended version of thisobjective is to predict µT (x) based on training data from thecomputer simulator and an associated physical experiment.


IntroductionExample

Conclusions


• Interpolation/Prediction Given output of computer code at aset of training inputs,

(xt1, yc(xt1)), . . . (x

tm, y

c(xtm))

predict y c(·) at a new input x0. An extended version of thisobjective is to predict µT (x) based on training data from thecomputer simulator and an associated physical experiment.• Assess Prediction Accuracy Using the data from both aphysical experimental data and a (calibrated) computerexperiment, give uncertainty bounds for the predicted value ofy c(x) or µT (x) of an associated physical system.


IntroductionExample

Conclusions


• Experimental design Determine a set of inputs at which tocarry out the sequence of code runs. (a “good” design of aphysical or computer experiment depends on the scientificobjective of the research)

◮ Exploratory Designs (geometric “space-filling”)

◮ Designs that yield good overall prediction

◮ Designs to find optimal inputs (find xoptd

≡ argmin y(x))


IntroductionExample

Conclusions


• Uncertainty/Output Analysis Determine the distribution ofthe random variable y c(xd ,Xe), i.e., determine the variability inthe performance measure y c(·) for design xd when applied to thepopulation defined by the distribution of Xe , eg., patient specificvariables (patient weight or bone material properties) or surgeonspecific variables (measuring surgical skill)


IntroductionExample

Conclusions


• Uncertainty/Output Analysis Determine the distribution ofthe random variable y c(xd ,Xe), i.e., determine the variability inthe performance measure y c(·) for design xd when applied to thepopulation defined by the distribution of Xe , eg., patient specificvariables (patient weight or bone material properties) or surgeonspecific variables (measuring surgical skill)• Calibration Given outputs from a computer simulatory c(xd , xc) where the information about xc is described by the(prior) distribution πc(xc) and also from a physical experimentyp(xd ), refine the prior to a posterior distribution for xc .


IntroductionExample

Conclusions


• Sensitivity Analysis Determine the important (unimportant)input variables, i.e., determine those xi of x = (x1, . . . , xd ) thaty c(x) (or µT (x)) is most (least) sensitive to changes in?

Philosophy Inputs that have relatively little effect on the outputcan be set to some nominal value; additional investigation can berestricted to determining how the output depends on the activeinputs• Set Tuning Parameters for the computer code (FEA−−mesh density, discretization of continuous functional inputs,solution tolerances, . . . )


IntroductionExample

Conclusions

Applications of Computer Simulators

• Policy Planning –the Wonderland model (41 inputs) describesglobal economic and environmental scenarios. Wonderland has 41inputs detailing population growth, economic activity in developedand undeveloped areas, etc and an output which is a weightedmeasure of human development that takes into account

◮ Net output per capita (output minus environ control costs)◮ Death rates◮ Annual flow of pollutants◮ “Carrying capacity”


IntroductionExample

Conclusions


• Industry

1. Design of VLSI circuits

2. Design engines and other automobile components (Fang, Li,and Sudjianto, 2005)

3. Determine optimum operating conditions for a compressionmolding process

4. Design of jet engines, helicopter rotor blades


IntroductionExample

Conclusions


• Industry

1. Design of VLSI circuits

2. Design engines and other automobile components (Fang, Li,and Sudjianto, 2005)

3. Determine optimum operating conditions for a compressionmolding process

4. Design of jet engines, helicopter rotor blades

• Environmental Science NIST codes for the temporalevolution of contained and wild fires


IntroductionExample

Conclusions


• Cosmology determination of cosmological and computer modelparameters.1. Habib, Heitmann, Higdon, Nakhleh, and Williams (2006)Cosmic Calibration: Constraints from the Matter Power Spectrumand the Cosmic Microwave Background, LANL Technical Report,LA-UR-07-00562. Heitmann, Higdon, Nakhleh, and Habib (2006) CosmicCalibration, LANL Technical Report, LA-UR-06-2320


IntroductionExample

Conclusions

Outline

Introduction

Example

Conclusions


IntroductionExample

Conclusions

An Example

In his Cornell PhD thesis, KevinOng conducted an uncertaintyanalysis of the effects of Engi-neering Cup design, Sur-gical, Patient variableson the Stability of UncementedAcetabular Components


IntroductionExample

Conclusions

An Example


IntroductionExample

Conclusions

Example-Inputs


IntroductionExample

Conclusions

Example-Outputs

• Three related outputs related to the amount of material that willeventually accumulate behind the acetabular cup

1. Total contact surface area

2. Rim contact surface area

3. Change in the bone-implant gap volume.


IntroductionExample

Conclusions

Example-Data


IntroductionExample

Conclusions

Example-Sensitivity Analysis


IntroductionExample

Conclusions

Outline

Introduction

Example

Conclusions


IntroductionExample

Conclusions

Conclusions

• Bottom Line Many of the Problems 1-7 have “natural”solutions obtained by approximating y(x), by a fast (i.e., linear inthe training data) predictor, a metamodel• Statisticians use a Bayesian approach to produce fast predictorsfor “smooth” computer codes, based on a stationary Gaussianstochastic process model (or more complex mode). Theposterior distribution of the process gives both a predictor anderror estimate due to model uncertainty.• In addtion to predictions of the code at “new” locations,stationary Gaussian stochastic processes can be used to produceexperimental designs for criteria-based objectives, and to performtuning and calibration.


IntroductionExample

Conclusions

Questions?


Introduction to Computer Experiments€¦ · Introduction Example Conclusions Computer Experiments...

Documents

Transcript of Introduction to Computer Experiments€¦ · Introduction Example Conclusions Computer Experiments...