sensit (1)

8/2/2019 sensit (1)

http://slidepdf.com/reader/full/sensit-1 1/125

Sensitivity Analysis:a Validation andVerification Tool

Terry BahillSystems and Industrial EngineeringUniversity of Arizona

Tucson, AZ [email protected] ©, 1993-2009 BahillThis file is located athttp://www.sie.arizona.edu/sysengr/slides/

8/2/2019 sensit (1)


4/12/2012 2

ReferencesSmith, E. D., Szidarovszky, F., Karnavas, W. J. and

Bahill, A. T., Sensitivity analysis, a powerful systemvalidation technique, The Open Cybernetics and Systemics

Journal,

http://www.bentham.org/open/tocsj/openaccess2.htm,

2: 39-56, 2008, doi: 10.2174/1874110X00802010039

W. J. Karnavas, P. Sanchez and A. T. Bahill, Sensitivity

analyses of continuous and discrete systems in the timeand frequency domains, IEEE Trans. Syst. Man.

Cybernetics, SMC-23(2), 488-501, 1993.

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 3

You should perform asensitivity analysis anytime you

create a model write a set of requirements

design a system

make a decision

do a tradeoff study

originate a risk analysis

want to discover the cost drivers

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 4

In a sensitivity analysis change

the values of

inputs

parameters

architectural features measure changes in

outputs

performance indices

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 5

A sensitivity analysis can be used to validate a model,

warn of unrealistic model behavior, point out important assumptions,

help formulate model structure, simplify a model,

suggest new experiments, guide future data collection efforts, suggest accuracy for calculating parameters,

adjust numerical values of parameters,

choose an operating point, allocate resources,

detect critical criteria, suggest manufacturing tolerances,

identify cost drivers.© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 6

History: the earliest sensitivity analyses The genetics studies on the pea by Gregor

Mendel, 1865. The statistics studies on the Irish hops crops by

Gosset (reported under the pseudonymStudent), ca 1890.

© 2009 Bahill

8/2/2019 sensit (1)

http://slidepdf.com/reader/full/sensit-1 7/1254/12/2012 7

Classes of sensitivity functions Analytic

for well defined systems usually partial derivatives

Empirical

show sensitivity to parameters

observe system changes whenparameters are changed

works for an unmodeled system

© 2009 Bahill

8/2/2019 sensit (1)


Types of sensitivity functions

Analytic

absolute

relative

semirelative

Empirical direct observation

sinusoidal variation of parameters

design of experiments

Excel, using what-if analysis

© 2009 Bahill

8/2/2019 sensit (1)


The absolute-sensitivity function

The absolute-sensitivity of the

function F to variations in theparameter is

It should be evaluated at the

normal operating point

(NOP).

NOP

F F

S

© 2009 Bahill

8/2/2019 sensit (1)


Examples Absolute-sensitivity functions are used

to calculate changes in the output due to changesin the inputs or system parameters

to see when a parameter has its greatest effect

in adaptive control systems

© 2009 Bahill

8/2/2019 sensit (1)


A process model example

x and y are inputs and A to F model system

parameters. The output z is love potion number 9.The normal operating point is

What is the easiest way to increasethe quantity of z?

This sounds like a problem for absolute-sensitivityfunctions.

2 2 z Ax By Cxy Dx Ey F

0 0 0 0

0 0 0 0

( , ) (1,1), 1, 2,

3, 5, 7, 8,

x y A B

C D E F

0 2. z

© 2009 Bahill

8/2/2019 sensit (1)


Absolute-sensitivity functions*

4/12/2012 12 © 2009 Bahill

0 0 0 0 0NOP

0 0 0 0 0

NOP

2 0,

2 0.

z

x

z

y

z

S A x C y D x

zS B y C x E

y

2

0NOP

2

0

NOP

0 0

NOP

0

NOP

0

NOP

NOP

1,

1,

1,

1,

1,

1,

z

A

z

B

zC

z

D

z

E

z

F

z

S x A

zS y

B

zS x yC

zS x

D

zS y

E

zS

F

8/2/2019 sensit (1)


What about interactions? Change two parameters at the same time.

Interactions can be bigger than the first-order effects.

Non-steroidal, anti-inflammatory drugs (NSAID) such

as Ibupropren and Aleve have dangerous interactionswith angiotensin converting enzymes, which effectthe kidneys and lower blood pressure. No

pharmacists would allow you to take both.

My mother once cleaned the toilet with sodium

hypochlorite (Clorox bleach) and ammonia. Itproduced chorine gas.

Alcohol and barbiturates are much more dangerousif mixed.

© 2009 Bahill

8/2/2019 sensit (1)


Cooperation The performance of a system could be greater than

the sum of its subsystems (cooperation). Alone, neither a human nor a knife can slice bread.

Together a blind person and a Seeing Eye dog dobetter than either alone.

A pair of chopsticks performs more than twice aswell as an individual chopstick.

Two lions chasing a Thompkins Gazelle are more

than twice as likely to catch it, than a single lion.

4/12/2012 © 2009 Bahill14

8/2/2019 sensit (1)


4/12/2012 15

Interactions in the process control model Change two parameters at the same time.

Mixed partial derivatives can be bigger than first-order partial derivatives.

Of the 64 possible second-partial derivatives, onlythe following are nonzero.

© 2009 Bahill

8/2/2019 sensit (1)


Interactions*

4/12/2012 16 © 2009 Bahill

2

2

2

NOP

2

0

NOP

2

02

NOP

2

02

NOP

1,

3,

2 2,

2 4.

z

y E

z

x y

z

x

z

y

zS

y E

zS C

x y z

S A x

zS B

y

2

0

NOP

2

0

NOP

2

NOP

2

0

NOP

2

0

NOP

2 2,

1,

1,

2 2,

1,

z

x A

z

x C

z

x D

z

y B

z

y C

z

S x x A

zS y

x C

zS

x D

z

S y y B

zS x

y C

8/2/2019 sensit (1)

http://slidepdf.com/reader/full/sensit-1 17/1254/12/2012 © 2009 Bahill17

Table 1. Effects of Individual and Combined Parameter Changes

for some second-order interaction terms with delta of 1.0, where

0 x x x etc.

FunctionsNormal

values

Valuesincreased

by one

unit

New z

values z

Total

change

in z

( , ) f x A A=1

x=1

A=2

x=27 5 5

0( , ) f x A A=1 A=2 3 1 2 z 0( , ) f x A x=1 x=2 3 1

0 0( , ) f x A A=1

x=12 0

( , ) f y B B=2

y=1

B=3

y=28 6 6

0( , ) f y B B=2 B=3 3 13 z

0( , ) f y B y=1 y=2 4 2

0 0( , ) f y B B=2

y=12 0

The purpose of

this slide is to

show the affectsof interactions,

without using

mathematics.

We will now usethese data to

estimate the value

of one of the

mixed-partial

derivatives

8/2/2019 sensit (1)


Estimate themixed-second-partial derivative

4/12/2012 © 2009 Bahill18

2

0 0 0 0 0 0( , ) ( , ) ( , ) ( , ) ( , ) f f f f f

2 7 3 3 23

1

z

x A

This is the wrong answer.Analytically we found that the correct value is 2.

8/2/2019 sensit (1)


A smaller step size, 0.01

4/12/2012 © 2009 Bahill19

Use the following general equation from Smith, Szidarovszky, Karnavas and Bahill [2008].

0 0 0 0 0 0 0 0

2 2

0 0 0 0

( , ) ( , ) ( , )( ) ( , )( )

1( , )( ) 2 ( , )( )( ) ( , )( )

2!

x y

xx xy yy

f x y f x y f x y x x f x y y y

f x x f x x y y f y y

Converting to find the value if we change x and A yields

0 0 0 0 0 0 0 0

2 2

0 0 0 0

( , ) ( , ) ( , )( ) ( , )( )1

( , )( ) 2 ( , )( )( ) ( , )( )2!

x A

xx xA AA

f x A f x A f x A x x f x A A A

f x x f x x A A f A A

Now, using the symbols that we used in our absolute sensitivity functions, we can write

2 2

0 0 0 0

2 2 2

( , ) ( , )

1 ( , ) 2 ( , ) ( , )2!

z z

x A

z z z

x A x A

z S x A S x A

S S S

Inserting numbers we get

1

0 *0.01 1* 0.01 2 *0.0001 2 *2 *0.0001 0.01 30 02

z

This delta z will now be put into row 2 column 5 of the following table.

z

8/2/2019 sensit (1)


4/12/2012 © 2009 Bahill20

Table 2. Effects of Individual and Combined Parameter Changes for

some second-order interaction terms with delta of 0.01, where

0 x x x etc.

FunctionsNormal

values

Valuesincreased

by delta

New z

values z

Total change

in z

( , ) f x A A=1

x=1

A=1.01

x=1.012.0103 0.0103 0.0103

0( , ) f x A A=1 A=1.01 2.0100 0.01000.0101 z

0( , ) f x A x=1 x=1.01 2.0001 0.0001

0 0( , ) f x A A=1

x=12.0000 0

( , ) f y B B=2

y=1

B=2.01

y=2.012.0104 0.0104 0.0104

0( , ) f y B B=2 B=2.01 2.0100 0.01000.0102 z

0( , ) f y B y=1 y=2.01 2.0002 0.0002

0 0( , ) f y B B=2

y=12.0000 0

8/2/2019 sensit (1)


4/12/2012 © 2009 Bahill21

Estimate the mixed-second-partial derivative, according to this formula from Smith,

Szidarovszky, Karnavas and Bahill [2008].

2

0 0 0 0 0 0( , ) ( , ) ( , ) ( , ) ( , ) f f f f f

Table 3. Values to be used in estimating the second

partial derivative

TermsParameter values with a

0.01 step size

Function

values

( , ) f A =1.01

x=1.01

2.0103

0( , ) f A =1.00

x =1.012.0100

0( , ) f A =1.01

x =1.002.0001

0 0( , ) f A =1.00

x =1.002.0000

2 2.0103 2.0100 2.0001 2.0000 0.00022

0.01*0.01 0.0001

z

x A

This is the same value that we computed analytically.

8/2/2019 sensit (1)


4/12/2012 © 2009 Bahill22

Table 4. Effects of Individual and Combined Parameter Changes for some third-

order interaction terms, delta = 0.001

FunctionsNormal

values

Values

increased

by delta

New z

values z Total change in z

( , ) f x A A=1

x=1

A=1.001

x=1.0022.0108 0.0108 0.0108

0( , ) f x A A=1 A=1.001 2.0100 0.0100

0.0102 z 0( , ) f x A x=1 x=1.001 2.0001 0.0001

0( , ) f x A x=1 x=1.001 2.0001 0.0001

0 0( , ) f x A A=1 x=1

2.0000 0

( , ) f y B B=2

y=1

B=2.001

y=1.0022.0112 0.0112 0.0112

0( , ) f y B B=2 B=2.001 2.0100 0.0100

0.0104 z 0( , ) f y B y=1 y=1.001 2.0002 0.0002

0( , ) f y B y=1 y=1.001 2.0002 0.0002

0 0( , ) f y B B=2

y=12.0000 0

8/2/2019 sensit (1)


4/12/2012 23

Third-order partial derivatives Once again the interaction affect is larger than the

sum of the individual changes. But at least the third-order terms are smaller than

the first and second-order terms.

Three of the third-order partial derivatives aregreater than zero.

All of the fourth-order partial derivatives are zero.

© 2009 Bahill

8/2/2019 sensit (1)


Minisummary The purposes of this section were

to show the bad affects of too large of a step size

to show how to calculate derivatives analytically

and to estimate derivatives numerically

to show that interactions are important

to show how to consider third- and forth-orderderivatives.

4/12/2012 © 2009 Bahill24

8/2/2019 sensit (1)


4/12/2012 25

A pendulum clock 1I have a grandfather clock in Tucson that I would

like to move to a cabin up on Mount Lemmon.But I’ve been told that the changes in

temperature and altitude will make it inaccurate.Which will be the bigger culprit?

The period of oscillation of a pendulum is

A one-meter pendulum has a two-second

period. If the temperature changes by T , thenthe length becomes

Use the absolute-sensitivity function of P withrespect to T to calculate how many seconds perday the clock will gain.

glP / 2

0 (1 )T l l k T

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 26

A pendulum clock 2The absolute-sensitivity function is

The coefficient of expansion of a brass rod is

At the normal operating point T=0 and l=l 0, so

Mt. Lemmon is 2000 meters higher than Tucson and temperaturechanges 5ºC per 1000 m. So T=-10ºC.

Therefore, the change in period is

The pendulum will gain 8.6 seconds per day*.

NOP NOP

2 /

(1

P T

T

T

l g l k

T g l k T S

50 0 / 2 10 sec/ C

P

T T k glS

5

2 10 /°CT k

42 10 secP

T P S T

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 27

A pendulum clock 3However, the gravitational acceleration constant depends on

altitude (H)

Therefore the period becomes

and the absolute-sensitivity of P with respect to H is

For this equation the normal operating point is sea level, soH=0 and g 0=9.78. So,

6

0 3 10 , where is in meters.g g H H

0

2

H

l

P g k H

3

0NOP

P H H

H

k l

Sg k H

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 28

A pendulum clock 4

Going from Tucson to Mount Lemmon H=2000, so

The clock loses 26 seconds per day.

Although changes due to temperature and altitude are in the

opposite direction, they do not cancel each other out, becausechanges due to altitude are bigger.

0 7

3

0

3 10 s/m H P

H

k l

Sg

46 10 secP H P S H

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 29

Single pole with time delay 1

Use an absolute-sensitivity function to find when the

parameter K has the greatest effect on the step

response of the system. The step response is

( )

( ) ( ) 1

sY s Ke

M s R s s

( )

1

s

sr

KesY

s s

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 30

Single pole with time delay 2The absolute-sensitivity function of the step-response

with respect to K is

which transforms into

K has its greatest effect when the response reachessteady-state.

1τ)(

0

θ0

ss

esY

s

K S sr

0 0( ) / ( ) 1sr

t

K

yt eS

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 31


Analytic

absolute

relative

semirelative





© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 32

The relative-sensitivity function The relative-sensitivity of the function F to

variations in the parameter is

Relative-sensitivity functions are used tocompare parameters.

% change in

% change in

F F F F S

0

0NOP

F F

F S

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 33

Process control modelWhat is the easiest way (smallest percent change in an

operating point parameter) to increase the quantity of z that is being produced? Now this problem seemsappropriate for relative-sensitivity functions.

© 2009 Bahill

8/2/2019 sensit (1)


Relative-sensitivity functions

4/12/2012 34 © 2009 Bahill

0 0

0

NOP 0 0

0 0

NOP 0 0

0 00 0 0 0 0

NOP 0 0

0 00 0 0 0 0

0 0NOP

3.5,

4,

2 0,

2 0.

z

E

z

F

z

x

z

y

z E E

S y E z z

z F F S

F z z

z x x

S A x C y D x z z

z y yS B y C x E

y z z

20 0

0NOP 0 0

20 00

NOP 0 0

0 00 0

NOP 0 0

0 00

NOP 0 0

0.5,

1,

1.5,

2.5,

z

A

z

B

zC

z

D

z A A

S x A z z

z B BS y

B z z

z C C S x yC z z

z D DS x

D z z

8/2/2019 sensit (1)


4/12/2012 35

F is most important Therefore, we should increase F if we wish to

increase z. What about interactions? Could we do better by

changing two parameters at the same time?

© 2009 Bahill

8/2/2019 sensit (1)


Interactions*

4/12/2012 36 © 2009 Bahill

2 2 2

0 0 0 0

2 2 2

0 0NOP

2

0 0 0 0 0

2 2

0 0NOP

2

0 0

2

0NOP

2 2

0 0 0 0

2 2

0 0NOP

2

0 0 0 0 0

2 20 0NOP

2 2*1 *10.5,

2

0.75,

1.25,

21.0,

0.75,

z

x A

z

x C

z

x D

z

y B

z

y C

z x A x AS

x A z z

z x C x y C S

x C z z

z x DS

x D z

z y B y BS

y B z z

z y C x y C S

y C z z

2

2

2

0 0

2

0NOP

2

0 0 0 0 0

2 2

0 0NOP

2 2 2

0 0 0

2 2 2

0 0NOP

2 2 2

0 0 0

2 2 2

0 0NOP

1.75,

0.75,

20.5,

21.0.

z

y E

z

x y

z

x

z

y

z y E S

y E z

z x y x y C S

x y z z

z x x AS

x z z z y y B

S y z z

8/2/2019 sensit (1)


4/12/2012 37

Mini-summary Using absolute-sensitivity functions, the second- and

third-order terms, e. g.

were the most important, but using relative-

sensitivity functions, F was the most importantparameter.

The absolute-sensitivity functions show the mostimportant parameters for a fixed size change in theparameters

The relative-sensitivity functions show the mostimportant parameters for a certain percent change in

the parameters.

3

2and

z

x A

2

2

z

y

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 38

Relatively the most important

0 0

NOP 0 04

z

F

F F zS F z z

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 39

The operating point* In the process control example, if the operating point

is changed from (1, 1) to (10, 10), then the output z becomes most sensitive (relatively) to the input y .

At the operating point (1, 1), the output is not

sensitive to the inputs, which means we could

twiddle with the inputs forever and not be able tocontrol the output. Therefore, this is not a desirable

operating point.*

© 2009 Bahill

8/2/2019 sensit (1)


More examples of relative sensitivity functions

4/12/2012 © 2009 Bahill40

8/2/2019 sensit (1)


4/12/2012 41

Double pole with time-delay

Which of the parameters is

most important?

2

( )

( ) ( ) (τ 1)

sY s Ke

M s R s s

0

0NOP

1 M

K M K

M K S

0

0

0

θ NOP

θθ

θ

M M s

M S

0

0

0

NOP 0

2ττ τ 1

M s M

M sS

© 2009 Bahill

8/2/2019 sensit (1)


4/12/201242

Frequency domain output

8/2/2019 sensit (1)


4/12/2012 43

Results For low frequencies, K is biggest

For mid-frequencies, is biggest

For high frequencies, is biggest

© 2009 Bahill

A i l l d l t

8/2/2019 sensit (1)


A simple closed loop system

4/12/2012 44 © 2009 Bahill

Transfer function( )

( )

( )

Y s K M s

R s s K

Time-domain step-response

( ) 1Kt

sr y t e

Time-domain relative-sensitivity function0

0 0

0 0

1 1

sr

K t y Kt

K K t K t NoP

K K teS te

e e

Frequency-domain step-response

( )( )

sr

K Y s

s s K

Frequency-domain relative-sensitivity function

0

ˆ ( )sr Y

K

sS ss K

Take the inverse Laplace transform0

0ˆ ( )sr Y K t

K S t K e

where is the unit impulse.

8/2/2019 sensit (1)


4/12/2012 45 © 2009 Bahill

Time domain output

8/2/2019 sensit (1)


4/12/2012 46

An expert systemSensitivity functions need not be functions of

time or frequency.If premise1 = true (CF 1)

and premise2 = true (CF 2)

or premise3 = true (CF 3

)

then conclusion = true CF 4.

The certainty of the rule uses the minimum of

the certainties of the AND clauses.

© 2009 Bahill

Certainty factor domain output

8/2/2019 sensit (1)


4/12/2012 47 © 2009 Bahill

If CF 1 < CF 2, then the final certainty factorbecomes

3 41 4 1 4

100100 100 10,000 f

CF CF CFCF CFCF

CF

The relative-sensitivity functions are

1

3 4 104

0NOP

110,000 100

f CF

CF f

CF CF CF CF S

CF

20

f CF

CF S

3

301 4 4

0NOP

1 10,000 100

f CF

CF

f

CF CFCF CF S CF

4

3 1 3 4 401

0NOP

2

100 100 1,000,000

f CF

CF

f

CF CFCF CF CF CF S

CF

8/2/2019 sensit (1)


4/12/2012 48 © 2009 Bahill

Assume 1 3 4 280 and 81CF CF CF CF

Then 0 87 f CF and

10.26

f CF

CF S

20

f CF

CF S

30.26

f CF

CF S

40.53

f CF

CF S

Changes in CF 4 are twice as importantas changes in CF 1 or CF 3 and increases in CF 2 have no effect.

8/2/2019 sensit (1)


4/12/2012 49

Limitations of therelative-sensitivity function

L x t y t Lx t Ly t

0

NOP 0

( ) ( ) ( )

( )

f t f t f t f t S

f t

0

NOP 0

( ) ( ) ( )ˆ( )

F s F s F s F sS

F s

Because

ˆ f t F s

S S we have two functions

© 2009 Bahill

8/2/2019 sensit (1)


4/12/201250

Disadvantages of relative-sensitivity functions

different in time and frequency domains cannot use Laplace transforms to get time-

domain solution

division by zero problem

0

0NOP

F F

F S

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 51


Analytic

absolute relative

semirelative





© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 52

Double pole with time delay

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 53

The semirelative-sensitivity function The semirelative-sensitivity of the function F to

variations in the parameter is

0NOP

F F S

© 2009 Bahill

8/2/2019 sensit (1)


Tradeoff study

4/12/2012 54 © 2009 Bahill

A Generic Tradeoff Study

CriteriaWeight of

ImportanceAlternative-1 Alternative-2

Criterion-1 Wt1 S11 S12 Criterion-2 Wt2 S21 S22 AlternativeRating

Sum1 Sum2

A Numeric Example of a Tradeoff StudyAlternatives

CriteriaWeight of

ImportanceUmpire’s

AssistantSeeing

Eye DogAccuracy 0.75 0.67 0.33

Silence ofSignaling

0.25 0.83 0.17

Sum of weighttimes score

0.71The

winner0.29

1 1 11 2 21 2 1 12 2 22

andSum Wt S Wt S Sum Wt S Wt S

8/2/2019 sensit (1)


4/12/2012 55

Which parameters could changethe recommendations?

Use this performance index

Compute the semirelative-sensitivity functions.

1 1 2

1 11 2 21 1 12 2 22 0.420

PI Sum Sum

Wt S Wt S Wt S Wt S

© 2009 Bahill

8/2/2019 sensit (1)


Semirelative-sensitivity functions*

4/12/2012 56 © 2009 Bahill

1

2

11

21

12

22

11 12 1

21 22 2

1 11

2 21

1 12

2 22

0.26

0.16

0.50

0.21

-0.25

-0.04

F

Wt

F

Wt

F

S

F

S

F S

F

S

S S S Wt

S S S Wt

S Wt S

S Wt S

S Wt S

S Wt S

8/2/2019 sensit (1)


A sensitivity matrix

4/12/2012 57 © 2009 Bahill

Analytic semirelative-sensitivity function values for PI 1, the

difference of the alternative ratings

Alternatives

CriteriaWeight of

Importance

Umpire’s

Assistant

Seeing Eye

Dog

Accuracy 1

1

PI

Wt S = 0.26 1

11

PI

SS = 0.50 1

12

PI

SS = -0.25

Silence of

Signaling1

2

PI

Wt S = 0.16 1

21

PI

SS = 0.21 1

22

PI

SS = -0.04

A nice way todisplay the sensitivities

8/2/2019 sensit (1)


4/12/2012 58

What about interactions?The semirelative-sensitivity function of PI1 for the

interaction of Wt1 and S11 is

which is as big as the first-order terms.

1 11 0 0 0 0

2

1 11 1 11

1 11 NOP

0.5025F

Wt S

F S Wt S Wt S

Wt S

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 59

Interactions for PI1

So interactions are important.

Semirelative Sensitivity Values Showing Interaction Effects

Function Normalvalues

Valuesincreasedby 10%

New Fvalues

F Total changein z

1

F

Wt S 1Wt =0.75 1Wt =0.82 0.446 0.026

11

F

SS 11

S =0.67 11S =0.74 0.470 0.0500.076F

1 11

F

Wt SS

1Wt =0.75

11S =0.67

1Wt =0.82

11S =0.74

0.501 0.081 0.081

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 60

A new performance index A problem with performance index PI1 is that if s11=s12,

then the sensitivity with respect to Wt1 becomes equalto zero.

Mathematically this is correct, but logically it is wrong.

Another problem is that the sensitivity with respect to

Wt1 does not depend on scores for the nonwinningalternatives and we do want the sensitivities to dependon the other parameters.

The following performance index solves both of these

problems.

3

1 1

1 n m

i ij

i j

PI Wt S

m

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 61

Semirelative sensitivity functions

3 3

1NOP

1i

m

PI Wt i i ij

ji

PI S Wt Wt SWt m

3

ij

i ijPI S

Wt SSm

© 2009 Bahill

8/2/2019 sensit (1)


Sensitivity matrix for PI3

Table VI. Analytic semirelative-sensitivity function values

for PI 3, the sum of all weight times scoresAlternatives

CriteriaWeight of

Importance

Umpire’s

Assistant

Seeing Eye

Dog

Accuracy of

the call

3

1

0.38PI

Wt S 3

11 0.25

PI

SS 3

12

0.12PI

S

S

Silence of

Signaling3

20.13PI

Wt S 3

210.10PI

SS 3

220.02PI

SS

4/12/2012 © 2009 Bahill62

8/2/2019 sensit (1)


4/12/2012 63

It is not difficultAlthough these equations may look formidable, they

are easy to compute with a spreadsheet. For example

is merely the sum of the weight times scores in column

k and this is already in the spreadsheet. Furthermore,because

and the rest of the second order sensitivities are zero,Table VI is complete: it has all of the sensitivities in it.

1

n

k kj

k

Wt S

3 3

ij i ij

PI PI

S Wt SS S

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 64

What about interactions?

Yes, We do have to worry about interactions,because this is bigger than most of the first order

sensitivities.

3

1 11 0 0

2

3 1 111 11

1 11 NOP0.25

PI

Wt S

PI Wt S

S Wt SWt S m

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 65

Interaction Sensitivity MatrixTable VII. Analytic semirelative-sensitivity function values for

the interactions of PI 3

Alternatives

CriteriaWeight of

Importance

Umpire’sAssistant

Seeing Eye

Dog

Accuracy of

the call3

1 110.25

PI

Wt SS 3

1 120.12

PI

Wt SS

Silence of

Signaling3

2 210.10PI

Wt SS 3

2 220.02

PI

Wt SS

These cells contain the same numerical values as Table VI.

© 2009 Bahill

Tradeoff studies are hierarchical

8/2/2019 sensit (1)


Tradeoff studies are hierarchical

4/12/2012 66 © 2009 Bahill

The structure of a hierarchical tradeoff study

Criteria

Normalized

Criteria

Weights

Subcriteria

Normalized

Subcriteria

Weights

Scores for

Alternative-1

Scores for

Alternative-2

Performance (1)CW

Subcriteria-1 (1)

1Wt (1)

11S (1)

12S

Subcriteria-2 (1)

2Wt (1)

21S (1)

22S

Subcriteria-3 (1)

3Wt (1)

31S (1)

32S

Subcriteria-4 (1)

4Wt (1)

41S (1)

42S

Cost(2)

CW Subcriteria-1 (2 )

1Wt (2 )

11S (2 )

12S

Subcriteria-2 (1)

2Wt (2 )

21S (2 )

22S

Schedule (3)CW

Subcriteria-1 (1)

1Wt (3)

11S (3)

12S

Subcriteria-2 (3)

2Wt (3)

21S (3)

22S

Risk (4)CW

Subcriteria-1 (4 )

1Wt (4 )

11S (4 )

12S

Subcriteria-2 (4 )

2Wt (4 )

21S

(4 )

22S

Subcriteria-3 (4 )

3Wt (4 )

31S (4 )

32S

Alternative

Ratings1Sum 2Sum

8/2/2019 sensit (1)


4/12/2012 67

A new performance index, PI5Because most of tradeoff studies are hierarchical, in the

Spin Coach and the PopUp Coach I used thisperformance index

( )( ) ( ) ( )

5

1 1 1

1 n lk ml l l

i ij

l i j

PI CW Wt S

m

5( )

( )( ) ( ) ( )

1 1

1l

n l mPI l l l

i ijCW i j

S CW Wt Sm

5 ( ) ( ) ( )1ij

PI l l l

S i ijS CW Wt S

m

© 2009 Bahill

Single pole with time delay (when)

8/2/2019 sensit (1)


4/12/2012 68

Single pole with time delay (when)Transfer function

( )( )

( ) 1

sY s Ke

M s

R s s

Step-response

( )( 1)

s

sr

KeY s

s s

Semirelative-sensitivity functions0

0

0( 1)sr

sY

K

K eS

s s

0

0 0

0( 1)sr

sY K e

Ss

0

0 0

2

0( 1)sr

sY K e

Ss

Which (for 0t ) transform to0 0( )

0( ) (1 )sr y t

K S t K e

0 0( )

0 0( )sr y t S t K e

0 0( )

0 0 0( ) ( )sr y t S t K t e

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 69 © 2009 Bahill

h d hi h

8/2/2019 sensit (1)


4/12/2012 70

What does this teach us? If the model does not match the physical system in

the early part of the step response, then adjust thetime-delay of the model.

For steady state …

In the middle …

© 2009 Bahill

T f i i i f i

8/2/2019 sensit (1)


4/12/2012 71


Analytic

absolute relative

semirelative

We have just examined the analytic sensitivity

functions. We are now ready to look at theempirical sensitivity functions.

Empirical

direct observation




© 2009 Bahill

E i i l i i i f i

8/2/2019 sensit (1)


4/12/2012 72

Empirical sensitivity functions1

The method of direct observation

can be preformed on real-world systems

models of those systems

simulations of those models

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 73 © 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 74 © 2009 Bahill

RealSystem

Model ofReal

System

ComputerSimulationof Model

Modelers ModelersGood

ModelersGood

Modelers

M a t h e m a t i c i a n

s E x p e r i m e n t a l i s t s

E ti ti d i ti

8/2/2019 sensit (1)


4/12/2012 75

Estimating derivatives

are small, then the second term on the right can beneglected.

0If (x-x ) and ( ) f

© 2009 Bahill

T d ff t d l

8/2/2019 sensit (1)


4/12/2012 76

Tradeoff study exampleFor a +10% parameter change, the semirelative-

sensitivity function is

This is very easy to compute.

Tradeoff Study Matrix with S11 Increased by 5%Criteria

Weight ofImportance

Umpire’sAssistant

SeeingEye dog

Accuracy 0.75 0.74 0.33Silence of Signaling 0.25 0.83 0.17

Sum of weight timesscore 0.76 0.29

0 0

0

100.1

F F F S F

© 2009 Bahill

S iti it t i

8/2/2019 sensit (1)


Sensitivity matrix

4/12/2012 © 2009 Bahill77

Table X. Numerical estimates for semirelative-sensitivity function

for PI 3, the sum of the alternative ratings squared, for a plus 10%

parameter perturbationAlternatives

CriteriaWeight of

Importance

Umpire’s

Assistant

Seeing Eye

Dog

Accuracy of

the call3

10.38PI

Wt S 3

110.25

PI

SS 3

120.12PI

SS

Silence of

Signaling3

20.13

PI

Wt S 3

210.10PI

SS 3

220.02PI

SS

These are the same resultsthat were obtained in the analytic

semirelative sensitivity section.

B t h t b t th d d t ?

8/2/2019 sensit (1)


4/12/2012 78

But what about the second-order terms?Namely

When using the sum of weighted scores combining function

the second derivatives are all zero. So our estimations are all

right. This is not true for the product combining function,

most other combining functions (See Daniels, Werner and

Bahill [2001] for explanations of other combining functions.)or other performance indices.

In particular let’s try PI3.

2

0

( )( )

2!

f x x

1 1 11 2 21 2 1 12 2 22andSum Wt S Wt S Sum Wt S Wt S

1 2 1 2

1 11 21 2 12 22andWt Wt Wt Wt F S S F S S

© 2009 Bahill

D i ti f f ti f t i bl

8/2/2019 sensit (1)


4/12/2012 79

Derivative of a function of two variables

Let us examine the second-order terms, those insidethe { }, for two reasons

to see if they are large and must be included incomputing the first derivative

to estimate the effects of interactions on thesensitivity analysis

0 0 0 0 0 0 0 0

2 20 0 0 0

( , ) ( , ) ( , )( ) ( , )( )

1 ( , )( ) 2 ( , )( )( ) ( , )( )2!

x y

xx xy yy

f x y f x y f x y x x f x y y y

f x x f x x y y f y y

© 2009 Bahill

I t ti

8/2/2019 sensit (1)


4/12/2012 80

InteractionsPreviously we derived the analytic semirelative-

sensitivity function for the interaction of Wt1 and S11 as,

which is as big as the first-order semirelative-sensitivityfunctions.

0 03

1 11 0 0

21 113

1 11

1 11 NOP

0.25PI

Wt S

Wt SPI S Wt S

Wt S m

© 2009 Bahill

Interactions

8/2/2019 sensit (1)


4/12/2012 81

InteractionsFor a 10% change in parameter values, a simple-minded

approximation is

using our tradeoff study values we get

This does not match the analytic value.

What went wrong?

2

2

0 0 0 0

0 0

100.1 0.1

F F F F

S F

3

1 11

210 0.424PI

Wt SS F

© 2009 Bahill

Maybe the step size is too big

8/2/2019 sensit (1)


4/12/2012 82

Maybe the step size is too big Let’s reduce the perturbation step size to 0.1%?

This is closer, but it is still too big.

2

2

0 0 0 0

0 0

10000.001 0.001

F F F F S F

3

1 11

21000 0.393PI

Wt SS F

© 2009 Bahill

What went wrong?

8/2/2019 sensit (1)


4/12/2012 83

What went wrong?In the previous computations, we

changed both parameters at thesame time and then compared thevalue of the function to the value of the function at its normal operating

point. However, this is not thecorrect estimation for the second-partial derivative.

© 2009 Bahill

Estimating the second partials

8/2/2019 sensit (1)


4/12/2012 84

Estimating the second partials1

To estimate the second-partial derivatives we should

start with2

0 0 0 0 0( , ) ( , ) ( , ) f f f

0 0 0 02

0 0

( , ) ( , ) ( , ) ( , )( , )

f f f f f

2

0 0 0 0 0 0( , ) ( , ) ( , ) ( , ) ( , ) f f f f f

0

© 2009 Bahill

Estimating the second partials

8/2/2019 sensit (1)


Estimating the second partials2

4/12/2012 85 © 2009 Bahill

Values to be Used in Estimating the Second Derivative

Terms

Parameter values with a 0.1% step size,

that is 1Wt =0.00075 and 11S =0.00067Functionvalues

( , ) f 1Wt =0.75075

11S =0.67067

0.50063

0( , ) f 11S =0.67067 0.50025

0( , ) f 1Wt =0.75075 0.50038

0 0( , ) f 1Wt =0.75000

11S =0.670000.50000

2

3

1 11

0.50063 0.50038 0.50025 0.500000.5

*0.00075*0.00067

PI

Wt S m

Estimating the sensitivity functions

8/2/2019 sensit (1)


4/12/2012 86

Estimating the sensitivity functionsTo get the semirelative-sensitivity function we multiply

the second-partial derivative by the normal values of Wt1 and S11 to get

Now, this is the same result that we derived in the

analytic semirelative sensitivity section.

3

1 11 0 0 0 0

2

31 11 1 11

1 11 NOP

0.5 0.25PI

Wt S

PI S Wt S Wt S

Wt S

© 2009 Bahill

Lessons learned

8/2/2019 sensit (1)


4/12/2012 87

Lessons learned For a tradeoff study using the sum combining

function and a simple performance index,anything works.

Otherwise, the perturbation step size should be

small. Five and 10% perturbations are not

acceptable. It is incorrect to estimate the second partial

derivative by changing two parameters at thesame time and then comparing that value of the

function to the value of the function at itsnormal operating point. Estimating secondderivatives requires evaluation of four not two

numerator terms.

© 2009 Bahill

Sensitivity analysis of a risk analysis

8/2/2019 sensit (1)


4/12/2012 88

Sensitivity analysis of a risk analysisLet

be the probability of occurrence, the severity, and therisk, for the jth failure mode. Risk is

Use the performance index* and

calculate the semirelative-sensitivity functions

The largest sensitivities are always those for the largestrisk. This means that we should spend extra time andeffort estimating the probability and severity of thehighest ranked risk, which seems eminently sensible.

, and j j jP S R

j j j R P S

1

n

j

j

PI R

0 j

PI

P j j jS S P R 0 j

PI

S j j jS R S R

© 2009 Bahill

Linearity

8/2/2019 sensit (1)


4/12/2012 89

Linearity Although the model is linear, the

sensitivity functions are not.

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 91


Analytic

absolute relative

semirelative

Empirical

direct observation




© 2009 Bahill

Empirical sensitivity functions

8/2/2019 sensit (1)


4/12/2012 92


Sinusoidal variation of parameters,

also called frequency-domain experiments

response-surface methodology

© 2009 Bahill

Sinusoidal variation of parameters

8/2/2019 sensit (1)


4/12/2012 93

Sinusoidal variation of parameters Make two runs of the system

1. all parameters are set at their normal values

2. all parameters are modulated sinusoidally

Compute the power spectrum of each

Form the ratio of the two spectra at each frequency

Spikes will be observed at the modulation frequencies

at frequencies related to nonlinearities

at frequencies related to product effects

© 2009 Bahill

A first-order negative-feedback system

8/2/2019 sensit (1)


4/12/2012 94

A first order, negative feedback system

K ______ s + A

H

R (s ) Y (s )+

-

© 2009 Bahill

A first-order negative-feedback control

8/2/2019 sensit (1)


4/12/2012 95

A first order, negative feedback, controlsystem (continued)

Input: one Hertz, unit-amplitude sinusoid

Duration: one second Sampling: 2048 evenly spaced samples

Modulation frequencies: 5, 30 and 170 Hz

HK AsK

s RsY s M

)()()(

© 2009 Bahill

Modulation equations

8/2/2019 sensit (1)


4/12/2012 96

Modulation equations A = 0.1 (1 + 0.5 sin (5 2t))

H = 50 (1 + 0.5 sin (30

2t))

K = 1.0 (1 + 0.5 sin (170 2t))

© 2009 Bahill

Step response with modulated parameters

8/2/2019 sensit (1)


4/12/2012 98

Location of spikes If a parameter is modulated at a we expect to see

power at a

If there is a parabolic nonlinearity we also expect powerat 2 a

because 2 sin2 x = 1 - cos 2 x

If the system is sensitive to the product of twoparameters modulated at a and b, then we expectpower at a b because 2 sin x sin y = cos( x-y ) - cos( x+y )

These product terms are called interactions.

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 100

Split peaks are due to the one Hertz input.

This technique probably produces relative-sensitivities.

© 2009 Bahill

The semirelative-sensitive functions*

8/2/2019 sensit (1)


The semirelative sensitive functions

4/12/2012 101 © 2009 Bahill

1.50

~22

00 1.0

000

s H K As

AK

S

ss M

K

1.50

~22

0 50

000

20

s H K As

H K

S

M

H

1.50

~22

00 1.0

000

s H K As

AK S

M

A

An M/M1 queue

8/2/2019 sensit (1)


4/12/2012 102

An M/M1 queue service rate =

= 0.8 + 0.2 sin(46/4096 2t) arrival rate =

= 0.4 + 0.2 sin(4/4096 2t)

© 2009 Bahill

An M/M1 queue

8/2/2019 sensit (1)


4/12/2012 103

An M/M1 queue service rate =

= 0.8 + 0.2 sin(46/4096 2t) arrival rate =

= 0.4 + 0.2 sin(4/4096 2t)

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 105

An M/M1 queue is a low pass filter

The interaction peaks are not the same height.

© 2009 Bahill

Bode diagram of a low-pass filter

8/2/2019 sensit (1)


4/12/2012 107

p variation of parameters

There must be an input signal. Shape of the spikes depends on parameters of the input

signal.

The output cannot be stationary.

The frequency response of the system (e.g. low-pass,

high-pass, resonance, etc.) must be known.

The range of linearity of the system must be known.*

Parameters of the FFT and the windows must beunderstood.

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 109

yp y

Analytic

absolute relative

semirelative

Empirical

direct observation sinusoidal variation of parameters

design of experiments (DoE)


© 2009 Bahill


8/2/2019 sensit (1)


4/12/2012 110

p y 3

Design of experiments (DoE) should be used when

experiments are expensive. When doing a Taguchi 3-level design pick a normal

value, a high value and a low value.

If the high and low are some percentage change,

then you are doing a relative sensitivity analysis. If the high and low are plus and minus a unit, then

you are doing an absolute sensitivity analysis.

Alternatively, the high and low could be realistic

design options, in which case it does notcorrespond to any of our sensitivity functions.

© 2009 Bahill

Altitude 0 ft

9%

Percent change in air density

over the parameter ranges tobe expected for a typical July

afternoon in United States

8/2/2019 sensit (1)


4/12/2012 111 © 2009 Bahill

Low High

Temperature

Relative Humidity

Barometric Pressure P e r c e n t C h a n g e i n A

i r D e n s i t y

70 ºF 2600 feet

85 ºF50%

760 mm Hg

90%745 mm Hg

10%775 mm Hg

100 ºF

5200 feet

1%

3%

5%

7%

-1%

-3%

-5%

-7%

-9%

ballparks.

Medium


8/2/2019 sensit (1)


4/12/2012 112

yp y

Analytic

absolute relative

semirelative

Empirical

direct observation sinusoidal variation of parameters



© 2009 Bahill

The Pinewood Derby

8/2/2019 sensit (1)


4/12/2012 114

in the Pinewood Derby

baseline for Overall Happiness scoringfunction

baseline for Percent Happy Scoutsscoring function

importance weight for OverallHappiness evaluation criterion

baseline for Number of Repeat Racesscoring function

input value for Percent Happy Scouts

evaluation criterion input value for Number of Repeat

Races evaluation criterion

© 2009 Bahill

Of the 89 parameters only 3 couldh h f d l

8/2/2019 sensit (1)


4/12/2012 115

p ychange the preferred alternative

1. The Tradeoff Function with 90% for Performance and 10% for

Utilization of Resources the preferredalternative was a round robin with besttime scoring

with 57% for Performance and 43% forResources the preferred alternativeswitched to the double eliminationtournament

© 2009 Bahill

Sensitivity Analysis of Pinewood Derby (simulation data)0 9

8/2/2019 sensit (1)


4/12/2012 116 © 2009 Bahill

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Performance Weight

O v e r a

l l S c o r e

Single elimination

Double elimination

Round robin, mean-time

Round robin, best-time

Round robin, points

Sensitivity of Pinewood Derby (prototype data)

8/2/2019 sensit (1)


4/12/2012 117 © 2009 Bahill

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Performance Weight

O v e r a l l S c o r e

Double elimination

Round robin, best-time

Round robin, points

Of the 89 parameters only 3 couldh h f d l i

8/2/2019 sensit (1)


4/12/2012 118

change the preferred alternative2

2. The slope of thePercent Happy Scouts scoring function

3. The baseline for the

Percent Happy Scouts scoring function

© 2009 Bahill

8/2/2019 sensit (1)


4/12/2012 120

If a system (or its model) is very sensitive to

parameters over which the customer has no control,then it may be the wrong system for that customer.

If the sensitivity analysis reveals the most important

parameters and that result is a surprise, then it may be

the wrong system. If a system is more sensitive to its parameters than to

its inputs, then it may be the wrong system or thewrong operating point.

If the sensitivities of the model are different from thesensitivities of the physical system, then it may be the

wrong model.

© 2009 Bahill

Validation2

8/2/2019 sensit (1)


4/12/2012 121

If you delete a requirement, then your completeness

measure (a traceability matrix) should show a vacuity. After you make a decision, do a sensitivity analysis and

see if changing a parameter would change your

decision.

Domain experts should agree with the sensitivityanalysis about which criteria in a tradeoff study are themost important.

Domain experts should agree with the sensitivity

analysis about which risks are the most important. Do a sensitivity analysis of prioritized lists: see if

changing the most important criteria would change theprioritization.

© 2009 Bahill

Verification

8/2/2019 sensit (1)


4/12/2012 122

Unplanned excessive sensitivity to any parameter is a

verification mistake. Sensitivity to interactions should be flagged and

studied: such interactions may be unexpected and

undesirable.

© 2009 Bahill

Résumé

8/2/2019 sensit (1)


4/12/2012 123

After you

build a model, or write a set of requirements, or

do a tradeoff study, or

design a system,

you should study that thing to see if itmakes sense.

One of the best ways to study a thing iswith a sensitivity analysis.

© 2009 Bahill

Describe this talk to your Vice President

8/2/2019 sensit (1)


Professor Bahill modeled our potion production

process and did a sensitivity analysis of it. So we nowhave a better understanding of our process. Hissensitivity analysis accounts for nonlinearities andparameter interactions. His equations for estimating

parameters are correct (because Szidarovszkyderived them). This analysis shows which parametersare the most important for making our potent.

I recommend that we name our potion Love Potion

Number Nine. We should buy the copyright for that

song and play it in the background of our TVcommercials.

Play an audio clip.4/12/2012 © 2009 Bahill124

8/2/2019 sensit (1)


sensit (1)

Documents

Transcript of sensit (1)