Bootstrap Methods and Their Application

Bootstrap Methods and theirApplication

Anthony Davison

c2006A short course based on the book

Bootstrap Methods and their Application,by A. C. Davison and D. V. HinkleycCambridge University Press, 1997

Anthony Davison: Bootstrap Methods and their Application, 1

Summary

Bootstrap: simulation methods for frequentist inference.

Useful when

standard assumptions invalid (n small, data not normal,. . .);

standard problem has non-standard twist; complex problem has no (reliable) theory; or (almost) anywhere else.

Aim to describe

basic ideas; confidence intervals; tests; some approaches for regression


Motivation

Motivation

Basic notions

Confidence intervals

Several samples

Variance estimation

Tests

Regression


Motivation

AIDS data

UK AIDS diagnoses 19881992. Reporting delay up to several years! Problem: predict state of epidemic at end 1992, withrealistic statement of uncertainty.

Simple model: number of reports in row j and column kPoisson, mean

jk = exp(j + k).

Unreported diagnoses in period j Poisson, meank unobs

jk = exp(j)

k unobs

exp(k).

Estimate total unreported diagnoses fromperiod j by replacing j and k by MLEs. How reliable are these predictions? How sensible is the Poisson model?


Motivation

Diagnosis Reporting-delay interval (quarters): Totalperiod reports

to end

Year Quarter 0 1 2 3 4 5 6 14 of 19921988 1 31 80 16 9 3 2 8 6 174

2 26 99 27 9 8 11 3 3 2113 31 95 35 13 18 4 6 3 2244 36 77 20 26 11 3 8 2 205

1989 1 32 92 32 10 12 19 12 2 2242 15 92 14 27 22 21 12 1 2193 34 104 29 31 18 8 6 2534 38 101 34 18 9 15 6 233

1990 1 31 124 47 24 11 15 8 2812 32 132 36 10 9 7 6 2453 49 107 51 17 15 8 9 2604 44 153 41 16 11 6 5 285

1991 1 41 137 29 33 7 11 6 2712 56 124 39 14 12 7 10 2633 53 175 35 17 13 11 3064 63 135 24 23 12 258

1992 1 71 161 48 25 3102 95 178 39 3183 76 181 2734 67 133


Motivation

AIDS data

Data (+), fits of simple model (solid), complex model(dots)

Variance formulae could be derived painful! but useful? Effects of overdispersion, complex model, . . .?

++++++

++++

+

+++++

+++

+++++

++

+

+++++

+

+

++

+

+

Time

Dia

gnos

es

1984 1986 1988 1990 1992

010

020

030

040

050

0


Motivation

Goal

Find reliable assessment of uncertainty when

estimator complex

data complex

sample size small

model non-standard


Basic notions

Motivation

Basic notions


Several samples

Variance estimation

Tests

Regression


Basic notions

Handedness data

Table: Data from a study of handedness; hand is an integer measure

of handedness, and dnan a genetic measure. Data due to Dr Gordon

Claridge, University of Oxford.

dnan hand dnan hand dnan hand dnan hand

1 13 1 11 28 1 21 29 2 31 31 12 18 1 12 28 2 22 29 1 32 31 23 20 3 13 28 1 23 29 1 33 33 64 21 1 14 28 4 24 30 1 34 33 15 21 1 15 28 1 25 30 1 35 34 16 24 1 16 28 1 26 30 2 36 41 47 24 1 17 29 1 27 30 1 37 44 88 27 1 18 29 1 28 31 19 28 1 19 29 1 29 31 110 28 2 20 29 2 30 31 1


Basic notions

Handedness data

Figure: Scatter plot of handedness data. The numbers show the mul-

tiplicities of the observations.

15 20 25 30 35 40 45

12

34

56

78

dnan

hand

1 1

1

2 2 1 5

2

1

5

2

3

1

4

1

1

1 1

1

1


Basic notions

Handedness data

Is there dependence between dnan and hand for thesen = 37 individuals?

Sample product-moment correlation coefficient is = 0.509.

Standard confidence interval (based on bivariate normalpopulation) gives 95% CI (0.221, 0.715).

Data not bivariate normal!

What is the status of the interval? Can we do better?


Basic notions

Frequentist inference

Estimator for unknown parameter .

Statistical model: data y1, . . . , yniid F , unknown

Handedness data

y = (dnan, hand) F puts probability mass on subset of R2

is correlation coefficient

Key issue: what is variability of when samples arerepeatedly taken from F?

Imagine F known could answer question by

analytical (mathematical) calculation simulation


Basic notions

Simulation with F known

For r = 1, . . . , R:

generate random sample y1 , . . . , y

niid F ;

compute r using y

1 , . . . , y

n;

Output after R iterations:

1,

2, . . . ,

R

Use 1,

2, . . . ,

R to estimate sampling distribution of (histogram, density estimate, . . .)

If R, then get perfect match to theoretical calculation(if available): Monte Carlo error disappears completely

In practice R is finite, so some error remains


Basic notions

Handedness data: Fitted bivariate normal model

Figure: Contours of bivariate normal distribution fitted to handedness data;

parameter estimates are b1 = 28.5, b2 = 1.7, b1 = 5.4, b2 = 1.5, b = 0.509.

The data are also shown.

0.000

0.005

0.010

0.015

0.020

10 15 20 25 30 35 40 450

2

4

6

8

10

dnan

hand

1 1

1

2 2 1 5

2

1

5

2

3

1

4

1

1

1 1

1

1


Basic notions

Handedness data: Parametric bootstrap samples

Figure: Left: original data, with jittered vertical values. Centre and

right: two samples generated from the fitted bivariate normal distribu-

tion.

0 10 20 30 40 50

02

46

810

dnan

hand

Correlation 0.509

0 10 20 30 40 50

02

46

810

dnan

hand

Correlation 0.753

0 10 20 30 40 50

02

46

810

dnanha

nd

Correlation 0.533


Basic notions

Handedness data: Correlation coefficient

Figure: Bootstrap distributions with R = 10000. Left: simulation from

fitted bivariate normal distribution. Right: simulation from the data by

bootstrap resampling. The lines show the theoretical probability density

function of the correlation coefficient under sampling from a fitted bivariate

normal distribution.

Correlation coefficient

Prob

abilit

y de

nsity

0.5 0.0 0.5 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Correlation coefficient

Prob

abilit

y de

nsity

0.5 0.0 0.5 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5


Basic notions

F unknown

Replace unknown F by estimate F obtained

parametrically e.g. maximum likelihood or robust fit ofdistribution F (y) = F (y;) (normal, exponential, bivariatenormal, . . .)

nonparametrically using empirical distribution function(EDF) of original data y1, . . . , yn, which puts mass 1/n oneach of the yj

Algorithm: For r = 1, . . . , R:

generate random sample y1 , . . . , y

niid F ;

compute r using y

1 , . . . , y

n;

Output after R iterations:

1,

2, . . . ,

R


Basic notions

Nonparametric bootstrap

Bootstrap (re)sample y1, . . . , y

niid F , where F is EDF of

y1, . . . , yn Repetitions will occur!

Compute bootstrap using y1, . . . , y

n.

For handedness data take n = 37 pairs y = (dnan, hand)

with equal probabilities 1/37 and replacement from originalpairs (dnan, hand)

Repeat this R times, to get 1, . . . ,

R

See picture

Results quite different from parametric simulation why?


Basic notions

Handedness data: Bootstrap samples

Figure: Left: original data, with jittered vertical values. Centre and

right: two bootstrap samples, with jittered vertical values.

10 15 20 25 30 35 40 45

12

34

56

78

dnan

hand

Correlation 0.509

10 15 20 25 30 35 40 45

12

34

56

78

dnan

hand

Correlation 0.733

10 15 20 25 30 35 40 45

12

34

56

78

dnan

hand

Correlation 0.491


Basic notions

Using the

Bootstrap replicates r used to estimate properties of .

Write = (F ) to emphasize dependence on F

Bias of as estimator of is

(F ) = E( | y1, . . . , yniid F ) (F )

estimated by replacing unknown F by known estimate F :

(F ) = E( | y1, . . . , yniid F ) (F )

Replace theoretical expectation E() by empirical average:

(F ) b = = R1Rr=1

r


Basic notions

Estimate variance (F ) = var( | F ) by

v =1

R 1

Rr=1

(r

)2 Estimate quantiles of by taking empirical quantiles of

1, . . . ,

R

For handedness data, 10,000 replicates shown earlier give

b = 0.046, v = 0.043 = 0.2052


Basic notions

Handedness data

Figure: Summaries of the b. Left: histogram, with vertical line showing b.

Right: normal QQ plot of b.

Histogram of t

t*

Den

sity

0.5 0.0 0.5 1.0

0.0

0.5

1.0

1.5

2.0

2.5

4 2 0 2 4

0.

50.

00.

5

Quantiles of Standard Normal

t*


Basic notions

How many bootstraps?

Must estimate moments and quantiles of and derivedquantities. Nowadays often feasible to take R 5000

Need R 100 to estimate bias, variance, etc. Need R 100, prefer R 1000 to estimate quantilesneeded for 95% confidence intervals

4 2 0 2 4

4

2

02

4

R=199

Theoretical Quantiles

Z*

4 2 0 2 4

4

2

02

4

R=999

Theoretical Quantiles

Z*


Basic notions

Key points

Estimator is algorithm

applied to original data y1, . . . , yn gives original applied to simulated data y1 , . . . , y

n gives

can be of (almost) any complexity

for more sophisticated ideas (later) to work, must often besmooth function of data

Sample is used to estimate F F F heroic assumption

Simulation replaces theoretical calculation

removes need for mathematical skill does not remove need for thought check code very carefully garbage in, garbage out!

Two sources of error

statistical (F 6= F ) reduce by thought simulation (R 6=) reduce by taking R large (enough)



Motivation

Basic notions


Several samples

Variance estimation

Tests

Regression



Normal confidence intervals

If approximately normal, then . N( + , ), where

has bias = (F ) and variance = (F ) If , known, (1 2) confidence interval for would be(D1)

z1/2,

where (z) = . Replace , by estimates:

(F ).= (F )

.= b =

(F ).= (F )

.= v = (R 1)1

r

(r )2,

giving (1 2) interval b zv1/2.

Handedness data: R = 10, 000, b = 0.046, v = 0.2052, = 0.025, z = 1.96, so 95% CI is (0.147, 0.963)




Normal approximation reliable? Transformation needed?

Here are plots for = 12 log{(1 + )/(1 )}:

Transformed correlation coefficient

Den

sity

0.5 0.0 0.5 1.0 1.5

0.0

0.5

1.0

1.5

4 2 0 2 4

0.

50.

00.

51.

01.

5


Tran

sfor

med

cor

rela

tion

coef

ficie

nt




Correlation coefficient: try Fishers z transformation:

= () = 12 log{(1 + )/(1 )}

for which compute

b = R1

Rr=1

r , v =1

R 1

Rr=1

(r

)2,

(1 2) confidence interval for is

1{ b z1v

1/2

}, 1

{ b zv

1/2

} For handedness data, get (0.074, 0.804)

But how do we choose a transformation in general?



Pivots

Hope properties of 1, . . . ,

R mimic effect of sampling fromoriginal model.

Amounts to faith in substitution principle: may replaceunknown F with known F false in general, but oftenmore nearly true for pivots.

Pivot is combination of data and parameter whosedistribution is independent of underlying model.

Canonical example: Y1, . . . , Yniid N(, 2). Then

Z =Y

(S2/n)1/2 tn1,

for all , 2 so independent of the underlyingdistribution, provided this is normal

Exact pivot generally unavailable in nonparametric case.



Studentized statistic

Idea: generalize Student t statistic to bootstrap setting Requires variance V for computed from y1, . . . , yn Analogue of Student t statistic:

Z =

V 1/2

If the quantiles z of Z known, then

Pr (z Z z1) = Pr

(z

V 1/2 z1

)= 1 2

(z no longer denotes a normal quantile!) implies that

Pr( V 1/2z1 V

1/2z

)= 1 2

so (1 2) confidence interval is ( V 1/2z1, V1/2z)



Bootstrap sample gives (, V ) and hence

Z =

V 1/2

R bootstrap copies of (, V ):

(1, V

1 ), (

2, V

2 ), . . . , (

R, V

R)

and corresponding

z1 =1

V1/21

, z2 =2

V1/22

, . . . , zR =R

V1/2R

.

Use z1 , . . . , z

R to estimate distribution of Z for example,order statistics z(1) < < z

(R) used to estimate quantiles

Get (1 2) confidence interval

V 1/2z((1)(R+1)) , V1/2z((R+1))



Why Studentize?

Studentize, so ZD N(0, 1) as n. Edgeworth series:

Pr(Z z | F ) = (z) + n1/2a(z)(z) +O(n1);

a() even quadratic polynomial.

Corresponding expansion for Z is

Pr(Z z | F ) = (z) + n1/2a(z)(z) +Op(n1).

Typically a(z) = a(z) +Op(n1/2), so

Pr(Z z | F ) Pr(Z z | F ) = Op(n1).



If dont studentize, Z = ( )D N(0, ). Then

Pr(Z z | F ) = ( z1/2

)+n1/2a

( z1/2

)( z1/2

)+O(n1)

and

Pr(Z z | F ) = ( z1/2

)+n1/2a

( z1/2

)( z1/2

)+Op(n

1).

Typically = +Op(n1/2), giving

Pr(Z z | F ) Pr(Z z | F ) = Op(n1/2).

Thus use of Studentized Z reduces error from Op(n1/2) to

Op(n1) better than using large-sample asymptotics, for

which error is usually Op(n1/2).



Other confidence intervals

Problem for studentized intervals: must obtain V , intervalsnot scale-invariant

Simpler approaches: Basic bootstrap interval: treat as pivot, get

(((R+1)(1)) ), (

((R+1)) ).

Percentile interval: use empirical quantiles of 1 , . . . ,

R:

((R+1)),

((R+1)(1)).

Improved percentile intervals (BCa, ABC, . . .) Replace percentile interval with

((R+1)),

((R+1)(1)),

where , chosen to improve properties. Scale-invariant.



Handedness data

95% confidence intervals for correlation coefficient ,R = 10, 000:

Normal 0.147 0.963Percentile 0.047 0.758Basic 0.262 1.043

BCa ( = 0.0485, = 0.0085) 0.053 0.792

Student 0.030 1.206

Basic (transformed) 0.131 0.824Student (transformed) 0.277 0.868

Transformation is essential here!



General comparison

Bootstrap confidence intervals usually too short leads tounder-coverage

Normal and basic intervals depend on scale.

Percentile interval often too short but is scale-invariant.

Studentized intervals give best coverage overall, but

depend on scale, can be sensitive to V length can be very variable best on transformed scale, where V is approximatelyconstant

Improved percentile intervals have same error in principleas studentized intervals, but often shorter so lowercoverage



Caution

Edgeworth theory OK for smooth statistics bewarerough statistics: must check output.

Bootstrap of median theoretically OK, but very sensitive tosample values in practice.

Role for smoothing?

.

.

..

.

.

.

.

..

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

..

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

..

.

.

..

.

..

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

....

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

..

.

..

.

.

.

..

.

.

.

.

.

..

.

...

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

....

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

..

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

..

.

.

..

.

.

.

.

.

..

..

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

...

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

...

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

..

..

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

..

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

..

.

.

.

..

.

.

.

.

.

.

.

.

....

.

.

.

.

.

.

..

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

..

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

..

..

.

...

..

..

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.


T*-t

for m

edia

ns

-2 0 2

-10

-5

05

10



Key points

Numerous procedures available for automatic constructionof confidence intervals

Computer does the work

Need R 1000 in most cases

Generally such intervals are a bit too short

Must examine output to check if assumptions (e.g.smoothness of statistic) satisfied

May need variance estimate V see later


Several samples

Motivation

Basic notions


Several samples

Variance estimation

Tests

Regression


Several samples

Gravity data

Table: Measurements of the acceleration due to gravity, g, given as

deviations from 980,000 103 cms2, in units of cms2 103.

Series1 2 3 4 5 6 7 876 87 105 95 76 78 82 8482 95 83 90 76 78 79 8683 98 76 76 78 78 81 8554 100 75 76 79 86 79 8235 109 51 87 72 87 77 7746 109 76 79 68 81 79 7687 100 93 77 75 73 79 7768 81 75 71 78 67 78 80

75 62 75 79 8368 82 82 8167 83 76 78

73 7864 78


Several samples

Gravity data

Figure: Gravity series boxplots, showing a reduction in variance, a shift

in location, and possible outliers.40

6080

100

g

1 2 3 4 5 6 7 8

series


Several samples

Gravity data

Eight series of measurements of gravitational acceleration gmade May 1934 July 1935 in Washington DC

Data are deviations from 9.8 m/s2 in units of 103 cm/s2

Goal: Estimate g and provide confidence interval

Weighted combination of series averages and its varianceestimate

=

8i=1 yi ni/s

2i8

i=1 ni/s2i

, V =

(8i=1

ni/s2i

)1,

giving = 78.54, V = 0.592

and 95% confidence interval of 1.96V 1/2 = (77.5, 79.8)


Several samples

Gravity data: Bootstrap

Apply stratified (re)sampling to series, taking each series asa separate stratum. Compute , V for simulated data

Confidence interval based on

Z =

V 1/2,

whose distribution is approximated by simulations

z1 =1

V1/21

, . . . , zR =R

V1/2R

,

giving

( V 1/2z((R+1)(1)) , V1/2z((R+1)))

For 95% limits set = 0.025, so with R = 999 usez(25), z

(975), giving interval (77.1, 80.3).


Several samples

Figure: Summary plots for 999 nonparametric bootstrap simulations.

Top: normal probability plots of t and z = (t t)/v1/2. Line on

the top left has intercept t and slope v1/2, line on the top right has

intercept zero and unit slope. Bottom: the smallest tr also has the

smallest v, leading to an outlying value of z.

.

..

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

..

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

...

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

...

.

.

.

.

.

..

.

...

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

..

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

..

.

.

.

.

..

.

.

...

.

.

.

.

.

.

..

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

...

.

.

.

.

.

.

.

.

.

..

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

..

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

..

.

.

.

.

.

.

..

.

..

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

..

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

..

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

..

.

.

.

..

.

.

.

.

..

.

..

.

.

.

.

.

.

.

.

..

.

.

.

.

..

..

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.


t*

-2 0 2

7778

7980

81

.

..

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

...

..

.

.

.

.

.

.

..

.

..

.

.

.

.

.

..

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

..

..

.

.

.

.

.

.

.

..

.

.

.

.

..

.

.

.

.

.

.

..

.

.

..

...

.

.

...

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

..

.

.

.

..

.

.

.

.

.

..

..

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

..

..

.

..

.

.

.

.

..

.

..

.

.

.

.

.

..

.

.

.

.

..

..

..

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

..

..

.

.

..

.

.

.

.

..

.

.

.

.

.

.

.

.

.

..

.

.

..

.

...

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

..

.

.

..

.

..

.

...

.

.

.

.

.

.

..

.

.

.

.

.

.

.

..

.

.

.

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

...

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

..

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

..

.

.

.

..

.

..

.

.

.

...

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

...

.

.

.

.

...

...

.

.

.

.

..

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

..

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

..

..

.

.

..

.

.

.

..

.

..

.

.

.

.

.

..

.

..

.

...

.

.

..

.

.

..

.

..

..

.

.

.

..

.

.

.

..

.

.

.

..

...

.

..

..

.

.

.

.

.

..

.

.

.

.

..

.

.

.

.

.

.

..

.

.

..

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

..

...

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

...

.

.

.

.

.

..

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

..

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

..

.

.

.

..

.

.

.

.

..

....

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

...

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

...

.

.

.

.

.

.

.

..

.

.

.

.

.

.

..

..

..

.

.

.

.

.

.

.

..

..

.

.

.

.

..

.

.

..

.

.

.

.

.

.

..

.

.

..

.

..

.

.

.

..

.

..

..

..

..

.

.

.

..

..

.

.

.

.

..

.

.

..

.

.

.

.

..

.

.

.

.

.

...

.

.

.

..

..

..

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

..

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

..

.

.

.

.

..

.

.

.

.

.

.

..

.

.

.

..

...

.

.

.

.

..

.

.

.

.

..

.

.

.

..

.

.

.

..

.

..

.

.

.

..

.

.

.

.

..

.

..

.

.

.

.

.

.

..

..

.

.

.

.

..

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

Quantiles of Standard Normalz*

-2 0 2

-15

-10

-5

05

.

.

..

.

.

. . .

.

.

.

. .

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

..

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

..

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

. .

.

.

.

.

..

.

.

..

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

...

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

..

.

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.. .

.

.

.

.

.

. .

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

..

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

. .

.

.

.

.

.

.

. .

.

.

.

.

.

..

. .

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

..

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

..

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

..

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

t*

sqrt(

v*)

77 78 79 80 81

0.1

0.2

0.3

0.4

0.5

0.6

0.7

.

.

..

.

.

. . .

.

.

.

. .

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

..

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

..

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

..

.

.

.

.

..

.

.

..

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

...

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

..

.

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

....

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.. .

.

.

.

.

.

. .

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

..

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

..

.

.

.

.

.

.

. .

.

.

.

.

.

..

. .

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

..

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

..

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

..

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

z*

sqrt(

v*)

-15 -10 -5 0 5

0.1

0.2

0.3

0.4

0.5

0.6

0.7


Several samples

Key points

For several independent samples, implement bootstrap bystratified sampling independently from each

Same basic ideas apply for confidence intervals


Variance estimation

Motivation

Basic notions


Several samples

Variance estimation

Tests

Regression


Variance estimation

Variance estimation

Variance estimate V needed for certain types of confidenceinterval (esp. studentized bootstrap)

Ways to compute this:

double bootstrap delta method nonparametric delta method jackknife


Variance estimation

Double bootstrap

Bootstrap sample y1, . . . , y

n and corresponding estimate

Take Q second-level bootstrap samples y1 , . . . , y

n fromy1, . . . , y

n, giving corresponding bootstrap estimates

1 , . . . ,

Q ,

Compute variance estimate V as sample variance of1 , . . . ,

Q

Requires total R(Q+ 1) resamples, so could be expensive

Often reasonable to take Q.= 50 for variance estimation, so

need O(50 1000) resamples nowadays not infeasible


Variance estimation

Delta method

Computation of variance formulae for functions of averagesand other estimators

Suppose = g() estimates = g(), and . N(, 2/n)

Then provided g() 6= 0, have (D2)

E() = g() +O(n1)

var() = 2g()2/n+O(n3/2)

Then var().= 2g()2/n = V

Example (D3): = Y , = log

Variance stabilisation (D4): if var().= S()2/n, find

transformation h such that var{h()}.=constant

Extends to multivariate estimators, and to = g(1, . . . , d)


Variance estimation


Variance estimation

Nonparametric delta method

Write parameter = t(F ) as functional of distribution F General approximation:

V.= VL =

1

n2

nj=1

L(Yj;F )2.

L(y;F ) is influence function value for for observation at ywhen distribution is F :

L(y;F ) = lim0

t {(1 )F + Hy} t(F )

,

where Hy puts unit mass at y. Close link to robustness. Empirical versions of L(y;F ) and VL are

lj = L(yj; F ), vL = n2

l2j ,

usually obtained by analytic/numerical differentation.


Variance estimation

Computation of lj

Write in weighted form, differentiate with respect to

Sample average:

= y =1

n

yj =

wjyj

wj1/n

Change weights:

wj 7 + (1 )1n , wi 7 (1 )

1n , i 6= j

so (D5)

y 7 y = yj + (1 )y = (yj y) + y,

giving lj = yj y and vL =1n2(yj y)

2 = n1n n1s2

Interpretation: lj is standardized change in y when increasemass on yj


Variance estimation

Nonparametric delta method: Ratio

Population F (u, x) with y = (u, x) and

= t(F ) =

x dF (u, x)/

u dF (u, x),

sample version is

= t(F ) =

x dF (u, x)/

u dF (u, x) = x/u

Then using chain rule of differentiation (D6),

lj = (xj uj)/u,

giving

vL =1

n2

(xj uju

)2


Variance estimation

Handedness data: Correlation coefficient

Correlation coefficient may be written as a function ofaverages xu = n1

xjuj etc.:

=xu xu{

(x2 x2)(u2 u2)}1/2 ,

from which empirical influence values lj can be derived

In this example (and for others involving only averages),nonparametric delta method is equivalent to delta method

GetvL = 0.029

for comparison with v = 0.043 obtained by bootstrapping.

vL typically underestimates var() as here!


Variance estimation

Delta methods: Comments

Can be applied to many complex statistics

Delta method variances often underestimate true variances:

vL < var()

Can be applied automatically (numerical differentation) ifalgorithm for written in weighted form, e.g.

xw =

wjxj, wj 1/n for x

and vary weights successively for j = 1, . . . , n, setting

wj = wi + , i 6= j,

wi = 1

for = 1/(100n) and using the definition as derivative


Variance estimation

Jackknife

Approximation to empirical influence values given by

lj ljack,j = (n 1)( j),

where j is value of computed from sample

y1, . . . , yj1, yj+1, . . . , yn

Jackknife bias and variance estimates are

bjack = 1

n

ljack,j, vjack =

1

n(n 1)

(l2jack,j nb

2jack

) Requires n+ 1 calculations of

Corresponds to numerical differentiation of , with = 1/(n 1)


Variance estimation

Key points

Several methods available for estimation of variances

Needed for some types of confidence interval

Most general method is double bootstrap: can be expensive

Delta methods rely on linear expansion, can be appliednumerically or analytically

Jackknife gives approximation to delta method, can fail forrough statistics


Tests

Motivation

Basic notions


Several samples

Variance estimation

Tests

Regression


Tests

Ingredients

Ingredients for testing problems:

data y1, . . . , yn; model M0 to be tested; test statistic t = t(y1, . . . , yn), with large values givingevidence against M0, and observed value tobs

P-value, pobs = Pr(T tobs |M0) measures evidenceagainst M0 small pobs indicates evidence against M0.

Difficulties:

pobs may depend upon nuisance parameters, those of M0; pobs often hard to calculate.


Tests

Examples

Balsam-fir seedlings in 5 5 quadrats Poisson sample?

0 1 2 3 4 3 4 2 2 10 2 0 2 4 2 3 3 4 21 1 1 1 4 1 5 2 2 34 1 2 5 2 0 3 2 1 13 1 4 3 1 0 0 2 7 0

Two-way layout: row-column independence?

1 2 2 1 1 0 12 0 0 2 3 0 00 1 1 1 2 7 31 1 2 0 0 0 10 1 1 1 1 0 0


Tests

Estimation of pobs

Estimate pobs by simulation from fitted null hypothesismodel M0.

Algorithm: for r = 1, . . . , R:

simulate data set y1 , . . . , y

n from M0; calculate test statistic tr from y

1 , . . . , y

n.

Calculate simulation estimate

p =1 +#{tr tobs}

1 +R

ofpobs = Pr(T tobs | M0).

Simulation and statstical errors:

p pobs pobs


Tests

Handedness data: Test of independence

Are dnan and hand positively associated? Take T = (correlation coefficient), with tobs = 0.509; thisis large in case of positive association (one-sided test)

Null hypothesis of independence: F (u, x) = F1(u)F2(x) Take bootstrap samples independently fromF1 (dnan1, . . . , dnann) and from F2 (hand1, . . . , handn),then put them together to get bootstrap data(dnan1, hand

1), . . . , (dnan

n, hand

n).

With R = 9, 999 get 18 values of , so

p =1 + 18

1 + 9999= 0.0019 :

hand and dnan seem to be positively associated To test positive or negative association (two-sided test),take T = ||: gives p = 0.004.


Tests

Handedness data: Bootstrap from M0

15 20 25 30 35 40 45

12

34

56

78

dnan

hand

310

1

32

2

4

2

1

1

1

2 1

2

1

1

Test statistic

Prob

abilit

y de

nsity

0.6 0.2 0.2 0.6

0.0

0.5

1.0

1.5

2.0

2.5


Tests

Choice of R

Take R big enough to get small standard error for p,typically 100, using binomial calculation:

var(p) = var

(1 + #{tr tobs}

1 +R

).=

1

R2Rpobs(1 pobs) =

pobs(1 pobs)

R

so if pobs.= 0.05 need R 1900 for 10% relative error (D7)

Can choose R sequentially: e.g. if p.= 0.06 and R = 99, can

augment R enough to diminish standard error.

Taking R too small lowers power of test.


Tests

Duality with confidence interval

Often unclear how to impose null hypothesis on samplingscheme

General approach based on duality between confidenceinterval I1 = (,) and test of null hypothesis = 0

Reject null hypothesis at level in favour of alternative > 0, if 0 <

Handedness data: 0 = 0 6 I0.95, but 0 = 0 I0.99, soestimated significance level 0.01 < p < 0.05: weakerevidence than before

Extends to tests of = 0 against other alternatives:

if 0 6 I1 = (, ), have evidence that < 0

if 0 6 I12 = (, ), have evidence that 6= 0


Tests

Pivot tests

Equivalent to use of confidence intervals Idea: use (approximate) pivot such as Z = ( )/V 1/2 asstatistic to test = 0

Observed value of pivot is zobs = ( 0)/V1/2

Significance level is

Pr

(

V 1/2 zobs |M0

)= Pr(Z zobs |M0)

= Pr(Z zobs | F ).= Pr(Z zobs | F )

Compare observed zobs with simulated distribution ofZ = ( )/V 1/2, without needing to construct null

hypothesis model M0 Use of (approximate) pivot is essential for success


Tests

Example: Handedness data

Test zero correlation (0 = 0), not independence; = 0.509,V = 0.1702:

zobs = 0

V 1/2=

0.509 0

0.170= 2.99

Observed significance level is

p =1 +#{zr zobs}

1 +R=

1 + 215

1 + 9999= 0.0216

Test statistic

Prob

abilit

y de

nsity

6 4 2 0 2 4 6

0.0

0.1

0.2

0.3


Tests

Exact tests

Problem: bootstrap estimate is

pobs = Pr(T tobs | M0) 6= Pr(T t |M0) = pobs,

so estimate the wrong thing

In some cases can eliminate parameters from nullhypothesis distribution by conditioning on sufficientstatistic

Then simulate from conditional distribution

More generally, can use MetropolisHastings algorithm tosimulate from conditional distribution (below)


Tests

Example: Fir data

Data Y1, . . . , Yniid Pois(), with unknown

Poisson model has E(Y ) = var(Y ) = : base test ofoverdispersion on

T =

(Yj Y )2/Y

. 2n1;

observed value is tobs = 55.15 Unconditional significance level:

Pr(T tobs | M0, )

Condition on value w of sufficient statistic W =

Yj:

pobs = Pr(T tobs | M0,W = w),

independent of , owing to sufficiency of W Exact test: simulate from multinomial distribution ofY1, . . . , Yn given W =

Yj = 107.


Tests

Example: Fir data

Figure: Simulation results for dispersion test. Left panel: R = 999 val-

ues of the dispersion statistic t obtained under multinomial sampling:

the data value is tobs = 55.15 and p = 0.25. Right panel: chi-squared

plot of ordered values of t, dotted line shows 249 approximation to

null conditional distribution.

20 40 60 80

0.0

0.01

0.02

0.03

0.04

t*

.

..

.

..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

.

...

.

.

chi-squared quantiles

disp

ersio

n st

atist

ic t*

30 40 50 60 70 80

2040

6080


Tests

Handedness data: Permutation test

Are dnan and hand related?

Take T = (correlation coefficient) again

Impose null hypothesis of independence:F (u, x) = F1(u)F2(x), but condition so that marginaldistributions F1 and F2 are held fixed under resamplingplan permutation test

Take resamples of form

(dnan1, hand1), . . . , (dnann, handn)

where (1, . . . , n) is random permutation of (1, . . . , n)

Doing this with R = 9, 999 gives one- and two-sidedsignificance probabilities of 0.002, 0.003

Typically values of p very similar to those forcorresponding bootstrap test


Tests

Handedness data: Permutation resample

15 20 25 30 35 40 45

12

34

56

78

dnan

hand

Test statistic

Prob

abilit

y de

nsity

0.6 0.2 0.2 0.6

0.0

0.5

1.0

1.5

2.0

2.5

3.0


Tests

Contingency table

1 2 2 1 1 0 12 0 0 2 3 0 00 1 1 1 2 7 31 1 2 0 0 0 10 1 1 1 1 0 0

Are row and column classifications independent:

Pr(row i, column j) = Pr(row i) Pr(column j)?

Standard test statistic for independence is

T =i,j

(yij yij)2

yij, yij =

yiyjy

Get Pr(224 38.53) = 0.048, but is T. 224?


Tests

Exact tests: Contingency table

For exact test, need to simulate distribution of Tconditional on sufficient statistics row and column totals

Algorithm (D8) for conditional simulation:

1. choose two rows j1 < j2 and two columns k1 < k2 atrandom

2. generate new values from hypergeometric distributionof yj1k1 conditional on margins of 2 2 table

yj1k1 yj1k2yj2k1 yj2k2

3. compute test statistic T every I = 100 iterations, say

Compare observed value tobs = 38.53 with simulated T

get p.= 0.08


Tests

Key points

Tests can be performed using resampling/simulation

Must take account of null hypothesis, by

modifying sampling scheme to satisfy null hypothesis inverting confidence interval (pivot test)

Can use Monte Carlo simulation to get approximations toexact tests simulate from null distribution of data,conditional on observed value of sufficient statistic

Sometimes obtain permutation tests very similar tobootstrap tests


Regression

Motivation

Basic notions


Several samples

Variance estimation

Tests

Regression


Regression

Linear regression

Independent data (x1, y1), . . ., (xn, yn) with

yj = xTj + j , j (0,

2)

Least squares estimates , leverages hj, residuals

ej =yj x

Tj

(1 hj)1/2. (0, 2)

Design matrix X is experimental ancillary should beheld fixed if possible, as

var() = 2(XTX)1

if model y = X + correct


Regression

Linear regression: Resampling schemes

Two main resampling schemes

Model-based resampling:

yj = xTj +

j ,

j EDF(e1 e, . . . , en e)

Fixes design but not robust to model failure Assumes j sampled from population

Case resampling:

(xj , yj) EDF{(x1, y1), . . . , (xn, yn)}

Varying design X but robust Assumes (xj , yj) sampled from population Usually design variation no problem; can prove awkward indesigned experiments and when design sensitive.


Regression

Cement data

Table: Cement data: y is the heat (calories per gram of cement) evolved

while samples of cement set. The covariates are percentages by weight

of four constituents, tricalciumaluminate x1, tricalcium silicate x2, tet-

racalcium alumino ferrite x3 and dicalcium silicate x4.

x1 x2 x3 x4 y

1 7 26 6 60 78.52 1 29 15 52 74.33 11 56 8 20 104.34 11 31 8 47 87.65 7 52 6 33 95.96 11 55 9 22 109.27 3 71 17 6 102.78 1 31 22 44 72.59 2 54 18 22 93.110 21 47 4 26 115.911 1 40 23 34 83.812 11 66 9 12 113.313 10 68 8 12 109.4


Regression

Cement data

Fit linear model

y = 0 + 1x1 + 2x2 + 3x3 + 4x4 +

and apply case resampling

Covariates compositional: x1 + + x4.= 100% so X

almost collinear smallest eigenvalue of XTX isl5 = 0.0012

Plot of 1 against smallest eigenvalue of XTX reveals

that var(1) strongly variable

Relevant subset for case resampling post-stratification ofoutput based on l5?


Regression

Cement data

.

.

.

.

.

.

...

.

.

.

.

.

.

.

..

.

..

.

..

...

. ..

.

.

.

.

.

.

..

.

.

.

.

.

.

..

.

.

...

.

.

.

.

.

.

.

.

.

.

.

..

.

. ..

... .

.

.

.

. ..

.

..

..

.

.

. ..

.

.

.

.

..

.

.

.

.

.

.

..

...

..

.

..

.

.

.

.

..

.

.

.

.

.

.

..

..

.. .

.

..

.

..

.

.

.

.

..

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

. ..

..

.

.

.

.

.

.

.

.

...

.

...

.

.

.

..

..

.

.

.

.

.

.

.

.

.

.

....

.

.

.. ..

.

.

.

.

.

.

..

.

.

.

.

.

..

..

..

.

.

.

..

...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

.

.

.

.

...

...

..

.

.

.

.

..

...

.

.

.

.

.

.

.

..

. ..

.

.

...

. .

.

.

.

..

.

.

.

.

.

.

.

.

.

.

..

..

.

...

.

.

.

.

.

...

.

.

.

.

.

.

.

.

...

.

..

.

.

.

.

.

.

.

..

.

.

... .

.

.

.

.

.

.

.

..

.

.

.

.

.

.

..

.

.

.

.

.

..

...

.

. .

.

....

.

.

.

.

.

.

..

.

..

.

.. .

.. ..

..

.

..

.

.

.

..

. .

.

. .

.

.

.

.

.

...

.

...

.

.

..

.

.

.

.

.

..

.

.

.

.

.

..

.

.

.

.

..

.

.

.

.

..

.

.

.

.

.

.

.

..

..

.

.

.

.

.

. .

.

.

.

.

..

.

..

..

.

.

.

.

.

.

.

. .

.

.

.

...

.

.

.

.

.

.

.

.

...

.

.

.

. .

.

.

.

.

.

.

.

..

..

.

.

.

.

.

.

.

.

.

..

.

.

.

.

..

.

.

.

.. ..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. .

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

... .

.

..

.

.

.

.

.

.

.

..

.

.

..

.

.

.

.

.

.. .

..

.

.

.

..

.

.

..

.

.

.

.

..

. .

.

.

.

.

..

.

.

.

..

.

. .

..

.

..

..

.

.

.

.

..

.

.

.

..

.

. .

.

.

.

.

.

.

.

.

.

.

..

.

.

..

.

..

.

.

..

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

..

..

.

.

.

.

.

.

.

..

.

.

. .

.

. ...

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

....

..

.

.

..

.

..

.

.

..

.

.

.

.

..

..

.

.

.

..

.

.

.

...

. ..

.

.

.

.

..

. .

.

...

.

.

..

.

.

. .

.

.

.

...

.

.

.

.

..

..

.

.

.

.

..

.

.

.

.

..

.. .

.

.

.

.

.

.

.

.

..

.

.

..

.

.

..

.

..

.

..

.. .

.

.

. .

.

.

. . .

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

. ..

..

... .

.

....

.

.

.

.

..

.

.

..

.

..

.

.

.

.

..

.

.

..

. . .

..

.

.

.

.

...

.

.

.

.

..

.

.

.

.

.

.. ..

.

.

.

.

.

.

...

.

.

.

.

.

.

.

.

.

smallest eigenvalue

beta

1hat

*

1 5 10 50 500

-10

-5

05

10

.

.

.

.

.

.

.. .

.

.

.

.

.

.

.

. .

.

. .

.

.

.

...

. ..

.

.

.

.

.

.

. .

.

.

.

.

.

.

..

.

.

.

..

.

.

.

.

.

..

.

.

.

.

.

.

.

. ..

.

.

..

.

. .

. .

..

..

..

.

.

. . ..

.

.

..

.

.

.

.

..

.

.

.

...

..

.. .. .

.

.

..

.

..

.

.

..

.

..

.. .

...

.

. ....

.

.

..

.

.

.

..

.

..

.

.

.

.

.

.

.

.

.

.

.. .

.

.

.

.

.

.

..

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

..

.

.

..

. ...

.

.

.

.

.

..

.

.

.

.

.

..

.. .

.

.

.

.

...

. ..

.

.

.

.

.

.

.

.

.

.

.

...

. .

.

..

.

.

.

........

.

.

.

.

.

.

..

.

..

.

.

.

.

.

..

.

.

. ...

. ..

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

..

..

.

. ..

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

...

.

..

.

.

.

.

.

.

.

...

.

.

.

...

.

.

.

.

.

.

..

.

.

.

.

..

..

.

..

..

. .

.

..

.

..

.

..

..

.

.

.

.

.

.

..

.

..

. .. .

.. ..

.

.

.

..

.

.

...

...

.

.

.

..

.

..

.

.

.

. ..

.

.

..

.

..

.

.

..

.

....

..

.

.

.

.

....

..

.

..

.

.

..

.

.

.

..

.

.

.

.

...

.

..

...

..

..

..

..

.

.

.

.

.

. .

.

.

...

.

.

.

.

.

..

.

.

...

.

.

..

.

.

.

..

.

.

.

.

. ..

.

..

.

.

.

.

.

.

..

.

.

.

.

..

.

..

.. ..

..

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

...

.

.

..

.

.

.

..

.

.

..

.

..

.

.

.

.

.

.

..

.

..

.

.

.

.

.

.

...

.

.

.

.

.

...

.

.

.

.

.

...

..

. .

..

...

.

.

..

.

.

.

.

..

.

.

.

..

.

. .

.

.

.

.

.

.

.

.

.

.

..

.

.

..

.

.

.

.

.

..

.

.

.

.

.

.

..

..

.

.

.

.

.

.

..

..

.

.

.

.

.

..

. ..

.

.

.

.

..

..

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.

.

.

.

.

.

..

.

.

.

.

.. ..

.

.

.

.

.

.

.

. ..

..

.

.

.

.

.

.

.

.

..

. ..

.

.

.

.

.

..

.

.

.

..

.

.

.

.

.

.

..

..

.

..

Bootstrap Methods and Their Application

Documents

Transcript of Bootstrap Methods and Their Application