CHAPTER 17CHAPTER 17 OOPTIMALPTIMAL DDESIGNESIGN FORFOR EEXPERIMENTALXPERIMENTAL IINPUTSNPUTS

•Organization of chapter in ISSO*–Background

•Motivation•Finite sample and asymptotic (continuous) designs•Precision matrix and D-optimality

–Linear models•Connections to D-optimality•Key equivalence theorem

–Response surface methods

–Nonlinear models

*Note: Appendix to these slides is brief discussion of factorial design (not in ISSO)

Slides for Introduction to Stochastic Search and Optimization (ISSO) by J. C. Spall

17-2

Optimal Design in SimulationOptimal Design in Simulation• Two roles for experimental design in simulation

– Building approximation to existing large-scale simulation via “metamodel”

– Building simulation model itself• Metamodels are “curve fits” that approximate simulation input/output

– Usual form is low-order polynomial in the inputs; linear in parameters – LinearLinear design theory useful

• Building simulation model– Typically need nonlinearnonlinear design theory

• Some terminology distinctions:– “FactorsFactors” (statistics term) “InputsInputs” (modeling and simulation terms)– “LevelsLevels” “ValuesValues”– “TreatmentsTreatments” “RunsRuns”

17-3

Unique Advantages of Design in SimulationUnique Advantages of Design in Simulation

• Simulation experiments may be considered special case of general experiments

• Some unique benefits occur due to simulation structure• Can control factors not generally controllable (e.g., arrival rates into

network)• Direct repeatability due to deterministic nature of random number

generators– Variance reduction (CRNs, etc.) may be helpful

• Not necessary to randomize runs to avoid systematic variation due to inherent conditions– E.g., randomization in run order and input levels in biological experiment to

reduce effects of change in ambient humidity in laboratory– In simulation, systematic effects can be eliminated since analyst controls

nature

17-4

Design of Computer Experiments in Design of Computer Experiments in StatisticsStatistics

• There exists significant activity among statisticians for experimental design based on computer experiments– T. J. Santner et al. (2003), The Design and Analysis of Computer Experiments,

Springer-Verlag– J. Sacks et al (1989), “Design and Analysis of Computer Experiments (with

discussion),” Statistical Science, 409–435 – Etc.

• Above statistical work differs from experimental design with Monte Carlo simulations– Above work assumes deterministic function evaluations via computer (e.g.,

solution to complicated ODE)• One implication of deterministic function evaluations: no need to replicate

experiments for given set of inputs• Contrasts with Monte Carlo, where replication provides variance reduction

17-5

General Optimal Design Formulation General Optimal Design Formulation (Simulation or Non-Simulation)(Simulation or Non-Simulation)

• Assume modelz = h(,x) + v ,

where x is an input we are trying to pick optimally

• Experimental design consists of N specific input values x = i and proportions (weights) to these input values wi :

• Finite-sampleFinite-sample design allocates n N available measurements exactly; asymptotic (continuous)asymptotic (continuous) design allocates based on n

1 2

1 2

N

Nw w w

17-6

DD-Optimal Criterion-Optimal Criterion

• Picking optimal design requires criterion for optimization

• Most popular criterion is D-optimal measure

• Let M(,) denote the “precision matrix” for an estimate of based on a design – M(,) is inverse of covariance matrix for estimate

and/or

– M(,) is Fisher information matrix for estimate

• D-optimal solution is

( , )

arg max det M

17-7

Equivalence TheoremEquivalence Theorem

• Consider linear model

• Prediction based on parameter estimate and “future” measurement vector hT is

• Kiefer-Wolfowitz equivalence theorem states:

D-optimal solution for determining to be used in forming is is the same the same that minimizes the maximum variance of predictor

• Useful in practical determination of optimal

, Tk k kz v k n =1,2,...,h

n

ˆˆ Tnz = h

nz

17-8

Variance Function as it Depends on Variance Function as it Depends on Input: Optimal Input: Optimal AsymptoticAsymptotic Design for Design for

Example 17.6 in Example 17.6 in ISSOISSO

17-9

Orthogonal DesignsOrthogonal Designs

• With linear models, usually more than one solution is D-optimal

• Orthogonality is means of reducing number of solutions

• Orthogonality also introduces desirable secondary properties– Separates effects of input factors (avoids “aliasing”)

– Makes estimates for elements of uncorrelated

• Orthogonal designs are not generally D-optimal; D-optimal designs are not generally orthogonal– However, somesome designs are both

• Classical factorial (“cubic”) designs are orthogonal (and often D-optimal)

17-10

Example Orthogonal Designs, Example Orthogonal Designs, rr = 2 Factors = 2 Factors

xk2

xk1

Cube (2r design)

xk2

xk1

Star (2r design)

17-11

Example Orthogonal Designs, Example Orthogonal Designs, rr = 3 Factors = 3 Factors

Star (2r design)

xk1

xk2

xk3

Cube (2r design)

xk2

xk1

xk3

17-12

Response Surface Methodology (RSM)Response Surface Methodology (RSM)

• Suppose want to determine inputs x that minimize the mean response z of some process (E(z))– There are also other (nonoptimization) uses for RSM

• RSM can be used to build local models with the aim of finding the optimal x– Based on building a sequence of local models as one

moves through factor (x) space• Each response surface is typically a simple regression

polynomial• Experimental design can be used to determine input

values for building response surfaces

17-13

Steps of RSM for Optimizing Steps of RSM for Optimizing xxStep 0 (Initialization)Step 0 (Initialization) Initial guess at optimal value of x. Step 1 (Collect data) Step 1 (Collect data) Collect responses z from several x values in neighborhood of current estimate of best x value (can use experimental design).Step 2 (Fit model) Step 2 (Fit model) From the x, z pairs in step 1, fit regression model in region around current best estimate of optimal x. Step 3 (Identify steepest descent path) Step 3 (Identify steepest descent path) Based on response surface in step 2, estimate path of steepest descent in factor space.Step 4 (Follow steepest descent path) Step 4 (Follow steepest descent path) Perform series of experiments at x values along path of steepest descent until no additional improvement in z response is obtained. This x value represents new estimate of best vector of factor levels. Step 5 (Stop or return)Step 5 (Stop or return) Go to step 1 and repeat process until final best factor level is obtained.

17-14

Conceptual Illustration of RSM for Two Conceptual Illustration of RSM for Two Variables in Variables in xx; Shows More Refined ; Shows More Refined Experimental Design Near SolutionExperimental Design Near Solution

Adapted from: Montgomery (2005), Design and Analysisof Experiments, Fig. 11-3

17-15

Nonlinear DesignNonlinear Design

• Assume model

z = h(,x) + v ,where enters nonlinearly and x is r-dimensional input vector

• D-optimality remains dominant measure– Maximization of determinant of Fisher information matrix

(from Chapter 13 of ISSO: Fn(, X) is Fisher information matrix based on n inputs in n×r matrix X)

• Fundamental distinction from linear case is that D-optimal criterion depends on

• Leads to conundrum:

Choosing X to best estimate , yet need to know to determine X

17-16

Strategies for Coping with Strategies for Coping with Dependence on Dependence on

• Assume nominal value of and develop an optimal design based on this fixed value

• Sequential design strategy based on an iterated design and model fitting process.

• Bayesian strategy where a prior distribution is assigned to , reflecting uncertainty in the knowledge of the true value of

17-17

Sequential Approach for Parameter Sequential Approach for Parameter Estimation and Optimal DesignEstimation and Optimal Design

  Step 0 (Initialization)Step 0 (Initialization) Make initial guess at , Allocate n0

measurements to initial design. Set k = 0 and n = 0.• Step 1 (Step 1 (DD-optimal maximization)-optimal maximization) Given Xn , choose the nk inputs in

X = to maximize

• Step 2 (Update Step 2 (Update estimate) estimate) Collect nk measurements based on

inputs from step 1. Use measurements to update from to • Step 3 (Stop or return)Step 3 (Stop or return) Stop if the value of in step 2 is

satisfactory. Else return to step 1 with the new k set to the former k + 1 and the new n set to the former n + nk (updated Xn now includes

inputs from step 1).

kn n nn n

ˆ ˆdet , ,( ) ( )[ ] .F X F XknX

0ˆ .

nˆ .

kn n+

17-18

Comments on Sequential DesignComments on Sequential Design

• Note two optimization problems being solved: one for , one for

• Determine next nk input values (step 1) conditioned on

current value of – Each step analogous to nonlinear design with fixed

(nominal) value of • “Full sequential” mode (nk = 1) updates based on

each new inputouput pair (xk , zk)

• Can use stochastic approximation to update :

where

| 1 1 1ˆ ˆ ˆ ,n n n n n n na zY x

| 21 1 1 1

12, ( , )( )n n n n nz z hY x x

17-19

Bayesian Design StrategyBayesian Design Strategy

• Assume prior distribution (density) for , p(), reflecting uncertainty in the knowledge of the true value of .

• There exist multiple versions of D-optimal criterion

• One possible D-optimal criterion:

• Above criterion related to Shannon information

• While log transform makes no difference with fixed , it doesdoes affect integral-based solution

• To simplify integral, may be useful to choose discrete prior p()

n nE p dlogdet , logdet , ( )( ) ( )F X F X

17-20

Appendix to Slides for Chapter 17: Appendix to Slides for Chapter 17: Factorial Design (not in Factorial Design (not in ISSOISSO))

• Classical experimental design deals with linear models

• Factorial designFactorial design is most popular classical method– All r inputs (“factors”) changed at one time

• Factorial design provides two key advantages over one-at-a-time changes:

1. Greater efficiency in extracting information from a given number of experiments2. Ability to determine if there are interaction effects

• Standard method is 2r factorial; “2” comes about by looking at each input at two levels: low () and high (+)

– E.g., if r = 3, then have 23 = 8 input combinations: ( ), (+ ), ( + ), ( +),

(+ + ), (+ +), ( + +), (+ + +)

17-21

Appendix to Slides (cont’d): Appendix to Slides (cont’d): Factorial Design with 3 InputsFactorial Design with 3 Inputs

• Consider r = 3 linear model

zk = 0 + 1xk1 + 2xk2 + 3xk3 + 4xk1xk2 + 5xk1xk3 +

6xk2xk3 + 7xk1xk2xk3 + noise,

where = [0, 1,…, 7]T represents vector of (unknown) parameters

and xki represents ith term in input vector xk

• 23 factorial design allows for efficient estimation of allall parameters in

• In contrast, one-at-a-time provides no information for estimating 4 to 7

• However, 23 factorial design must be augmented in some way if wish to add quadratic (e.g., ) or other higher-order polynomial terms to model

21kx

17-22

Appendix to Slides (cont’d): Appendix to Slides (cont’d): Illustration of Interaction with 2 InputsIllustration of Interaction with 2 Inputs

• Example responses for r = 2: no interactionno interaction and interactioninteraction between input variables

• Left plot (no interaction) shows that change in zk with change in xk2 does notdoes not depend on xk1; right plot (interaction) shows change in zk doesdoes depend on xk1

No interactionNo interactionzk

( +)

xk2

( )

(+ +)

(+ )Xk1= low

Xk1= high

InteractionInteractionzk

( +)

xk2

( )(+ +)

(+ )

Xk1= low

Xk1= high