Springer Series in Statistics978-1-4612-0795-5/1.pdf · Springer Series in Statistics Advisors: ......
Transcript of Springer Series in Statistics978-1-4612-0795-5/1.pdf · Springer Series in Statistics Advisors: ......
Springer Series in Statistics
Advisors: P. Bickel, P. Diggle, s. Fienberg, K. Krickeberg, 1. Olkin, N. Wermuth, s. Zeger
Springer Science+Business Media, LLC
Springer Series in Statistics
Andersen/Borgan/Gill/Keiding: Statistical Models Based on Counting Processes. Andrews/Herzberg: Data: A Collection of Problems from Many Fields for the Student
and Research Worker. Anscombe: Computing in Statistical Science through APL. Berger: Statistical Decision Theory and Bayesian Analysis, 2nd edition. Bolfarine/Zacks: Prediction Theory for Finite Populations. Bremaud: Point Processes and Queues: Martingale Dynamics. Brockwell/Davis: Time Series: Theory and Methods, 2nd edition. DaleylVere-Jones: An Introduction to the Theory of Point Processes. Dzhaparidze: Parameter Estimation and Hypothesis Testing in Spectral Analysis of
Stationary Time Series. Fahrmeir/Tutz: Multivariate Statistical Modelling Based on Generalized Linear
Models. Farrell: Multivariate Calculation. Federer: Statistical Design and Analysis for Intercropping Experiments. FienberglHoaglin/KruskallTanur (Eds.): A Statistical Model: Frederick Mosteller's
Contributions to Statistics, Science and Public Policy. Fisher/Sen: The Collected Works of Wassily Hoeffding. Good: Permutation Tests: A Practical Guide to Resampling Methods for Testing
Hypotheses. GoodmanlKruskal: Measures of Association for Cross Classifications. Grandell: Aspects of Risk Theory. Haberman: Advanced Statistics, Volume I: Description of Populations. Hall: The Bootstrap and Edgeworth Expansion. Hardie: Smoothing Techniques: With Implementation in S. Hartigan: Bayes Theory. Heyer: Theory of Statistical Experiments. Huet/Bouvier/GruetlJolivet: Statistical Tools for Nonlinear Regression: A Practical
Guide with S-PLUS Examples. Jolliffe: Principal Component Analysis. KolenlBrennan: Test Equating: Methods and Practices. Kotz/Johnson (Eds.): Breakthroughs in Statistics Volume I. Kotz/Johnson (Eds.): Breakthroughs in Statistics Volume II. Kres: Statistical Tables for Multivariate Analysis. Le Cam: Asymptotic Methods in Statistical Decision Theory. Le Cam/Yang: Asymptotics in Statistics: Some Basic Concepts. Longford: Models for Uncertainty in Educational Testing. Manoukian: Modem Concepts and Theorems of Mathematical Statistics. Miller, Jr.: Simultaneous Statistical Inference, 2nd edition. Mosteller/Wallace: Applied Bayesian and Classical Inference: The Case of The
Federalist Papers.
(continued after index)
Jun Shao Dongsheng Tu Department of Statistics University of Wisconsin, Madison 1210 West Dayton Street Madison, WI 53706-1685 USA
Institute of System Science Academia Sinica Beijing, 100080 People's Republic of China
With 4 figures.
Library of Congress Cataloging-in-Publication Data Shao, Jun.
The jackknife and bootstrap / Jun Shao, Dongsheng Tu. p. cm. - (Springer series in statistics)
Includes bibliographical references and index. ISBN 978-1-4612-6903-8 ISBN 978-1-4612-0795-5 (eBook) DOI 10.1007/978-1-4612-0795-5 1. Jackknife (Statistics). 2. Bootstrap (Statistics).
3. Resampling (Statistics). 4. Estimation theory. 1. Tu, Dongsheng. II. Title. III. Series. QA276.6.S46 1995 519.5' 44-dc20 95-15074
Printed on acid-free paper.
© 1995 Springer Science+Business Media New York Originally published by Springer-Verlag New York, Inc in 1995 Softcover reprint ofthe hardcover Ist edition 1995 Ali rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher Springer Science+Business Media, LLC, except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use of general descriptive names, trade names, trademarks, etc., in this publication, even if the former are not especially identified, is not to be taken as a sign that such names, as understood by the Trade Marks and Merchandise Marks Act, may accordingly be used freely byanyone.
Production managed by Hal Henglein; manufacturing supervised by Joe Quatela. Photocomposed pages prepared from the authors' LaTeX file.
9 8 7 6 5 4 3 2 (Second corrected printing, 1996)
ISBN 978-1-4612-6903-8 SPIN 10544454
Preface
The jackknife and bootstrap are the most popular data-resampling methods used in statistical analysis. The resampling methods replace theoretical derivations required in applying traditional methods (such as substitution and linearization) in statistical analysis by repeatedly resampling the original data and making inferences from the resamples. Because of the availability of inexpensive and fast computing, these computer-intensive methods have caught on very rapidly in recent years and are particularly appreciated by applied statisticians.
The primary aims of this book are
(1) to provide a systematic introduction to the theory of the jackknife, the bootstrap, and other resampling methods developed in the last twenty years;
(2) to provide a guide for applied statisticians: practitioners often use (or misuse) the resampling methods in situations where no theoretical confirmation has been made; and
(3) to stimulate the use of the jackknife and bootstrap and further developments of the resampling methods.
The theoretical properties of the jackknife and bootstrap methods are studied in this book in an asymptotic framework. Theorems are illustrated by examples. Finite sample properties of the jackknife and bootstrap are mostly investigated by examples and/or empirical simulation studies. In addition to the theory for the jackknife and bootstrap methods in problems with independent and identically distributed (Li.d.) data, we try to cover, as much as we can, the applications of the jackknife and bootstrap in various complicated non-Li.d. data problems.
Chapter 1 introduces some basic ideas and motivations for using the jackknife and bootstrap. It also describes the scope of our studies in this book. Chapters 2 and 3 contain general theory for the jackknife and the bootstrap, respectively, which sets up theoretical fundamentals. Some technical tools are introduced and discussed in these two chapters for readers
vii
Vlll Preface
interested in theoretical studies. Beginning with Chapter 4, each chapter focuses on an important topic concerning the application of the jackknife, the bootstrap, and other related methods. Chapter 4 studies bootstrap confidence sets in some depth: we consider various bootstrap confidence sets developed in recent years and provide asymptotic and empirical comparisons. Bootstrap hypothesis testing is also studied in Chapter 4. Chapter 5 discusses some computational aspects of the jackknife and bootstrap methods. Chapter 6 considers sample survey problems, one of the non-i.i.d. data problems in which the use of the resampling methods (e.g., the jackknife and balanced repeated replication) has a long history. Chapter 7 focuses on applications of the jackknife and bootstrap to linear models, one of the most useful models in statistical applications. Chapter 8 contains some recent developments of the jackknife and bootstrap in various other important statistical fields such as nonlinear regression, generalized linear models, Cox's regression, nonparametric density estimation, nonparametric regression, and multivariate analysis. Applications of the jackknife and bootstrap for dependent data (time series) are studied in Chapter 9. The last chapter introduces two resampling methods that are generalizations of the bootstrap, namely the Bayesian bootstrap and the random weighting. Except for the first chapter, each chapter ends with conclusions and discussions.
Some useful asymptotic results that are often cited in this book are provided in Appendix A. A list of notation is given in Appendix B.
Some knowledge of mathematical statistics (with a standard textbook such as Bickel and Doksum, 1977) is assumed. The reader should be familiar with concepts such as probability, distribution, expectation, estimators, bias, variance, confidence sets, and hypothesis tests. For reading Chapters 6-9, some knowledge of the fields under consideration is required: sample surveys for Chapter 6; linear models for Chapter 7; nonlinear models, generalized linear models, nonparametric regression, and multivariate analysis for the respective sections of Chapter 8; and time series for Chapter 9. Some knowledge of prior and posterior distributions in Bayesian analysis is needed for reading Chapter 10. The mathematical level of the book is flexible: a practical user with a knowledge of calculus and a notion of vectors and matrices can understand all of the basic ideas, discussions, and recommendations in the book by skipping the derivations and proofs (we actually omitted some difficult proofs); with a knowledge of advanced calculus, matrix algebra, and basic asymptotic tools in mathematical statistics (Appendix A and Chapter 1 of Serfling, 1980), one can fully understand the derivations and most of the proofs. A few places (e.g., Section 2.2 and part of Sections 2.4 and 3.3) involving more advanced mathematics (such as real analysis) can be skipped without affecting the reading of the rest of the book.
The Edgeworth and Cornish-Fisher expansions are very important tools
Preface ix
in studying the accuracy of the bootstrap distribution estimators and bootstrap confidence sets. However, the derivations and rigorous proofs of these expansions involve difficult mathematics, which can be found in a recent book by Hall (1992d) (some special cases can be found in Appendix A). Thus, we only state these expansions (with the required regularity conditions) when they are needed, without providing detailed proofs. This does not affect the understanding of our discussions.
Although conceived primarily as a research monograph, the book is suitable for a second-year graduate level course or a research seminar. The following are outlines for various possible one-semester courses.
I
(I) AN INTRODUCTION TO JACKKNIFE AND BOOTSTRAP (non-theoretical)
Chapter 1, Sections 2.1 and 2.3, Sections 3.4 and 3.5, Chapter 4 (except Sections 4.2 and 4.3), and Chapter 5. Technical proofs should be skipped. If there is time, include Sections 6.1, 6.2, 6.3, 7.1, 7.2, and 7.3.
(II) JACKKNIFE AND BOOTSTRAP FOR COMPLEX DATA (non-theoretical)
Chapter 1, Chapter 6 (except Section 6.4), Chapter 7 (except Section 7.5), Chapter 8, and Chapter 9. Technical proofs should be skipped. If there is time, include some materials from Chapter 5 or Section 1O.l.
(III) THEORY OF JACKKNIFE AND BOOTSTRAP (theoretical)
Chapters 1-5. If there is time, include some materials from Chapters 6 and 7 or Chapter 10.
(IV) JACKKNIFE AND BOOTSTRAP FOR COMPLEX DATA (theoretical)
Chapters 1, 6-9. If there is time, include Chapter 5 or Chapter 10.
Of course, one may combine either (I)-(II) or (III)-(IV) for a twosemester course.
This book is essentially based on the authors' lecture notes for graduate level courses taught at Purdue University in 1988, at the University of Waterloo in 1989, and at the University of Ottawa in 1991 and 1992. We are very grateful to students and colleagues who provided helpful comments. Special thanks are due to C. R. Rao, who provided some critical comments on a preliminary draft of this book; J. N. K. Rao, who read part of the manuscript and provided useful suggestions; and anonymous referees and Springer-Verlag Production and Copy Editors, who helped to improve the presentation. We also would like to express our appreciation to the National Sciences and Engineering Research Council of Canada for support during the writing of the book.
Madison and Ottawa May, 1995
J. Shao D. Tu
Contents
Preface
Chapter 1. Introduction
1.1 Statistics and Their Sampling Distributions
1.2 The Traditional Approach
1.3 The Jackknife . . . . . . . . . . .
1.4 The Bootstrap ......... .
1.5 Extensions to Complex Problems
1.6 Scope of Our Studies . . . . . . .
Chapter 2. Theory for the Jackknife
2.1 Variance Estimation for Functions of Means
2.2
2.3
2.1.1 Consistency ....... .
2.1.2 Other properties ...... .
2.1.3 Discussions and examples ..
Variance Estimation for Functionals
2.2.1 Differentiability and consistency
2.2.2 Examples...........
2.2.3 Convergence rate . . . . . . .
2.2.4 Other differential approaches
The Delete-d Jackknife ....
2.3.1 Variance estimation .
2.3.2 Jackknife histograms.
2.4 Other Applications . . .
2.4.1 Bias estimation.
2.4.2 Bias reduction .
xi
vii
1
1
2 4 9
17
19
23
23
24
28
29 32
33
37
42
44 49 50
55
60 61
64
xii
2.4.3 Miscellaneous results .
2.5 Conclusions and Discussions.
Chapter 3. Theory for the Bootstrap 3.1 Techniques in Proving Consistency
3.1.1 Bootstrap distribution estimators.
3.1.2 Mallows' distance .....
3.1.3 Berry-Esseen's inequality
3.1.4 Imitation ..
3.1.5 Linearization
3.1.6 Convergence in moments
3.2 Consistency: Some Major Results.
3.2.1 Distribution estimators
3.2.2 Variance estimators ..
3.3 Accuracy and Asymptotic Comparisons
3.3.1
3.3.2
3.3.3
3.3.4
3.3.5
Convergence rate . . . . . . . . .
Asymptotic minimaxity .....
Asymptotic mean squared error .
Asymptotic relative error
Conclusions . . . . .
3.4 Fixed Sample Performance
3.4.1 Moment estimators .
3.4.2 Distribution estimators
3.4.3 Conclusions.
3.5 Smoothed Bootstrap
3.5.1 Empirical evidences and examples
3.5.2 Sample quantiles
3.5.3 Remarks.
3.6 Nonregular Cases
3.7 Conclusions and Discussions .
Chapter 4. Bootstrap Confidence Sets and Hypothesis Tests 4.1 Bootstrap Confidence Sets . . .
4.1.1 The bootstrap-t
4.1.2 The bootstrap percentile.
Contents
68 69
71
72
72 73
74 76 78 79
80 80 86 91 91
97
99 102
104
104
105
108
112 113 113 116 117 118 127
129 129
131
132
Contents xiii
4.1.3 The bootstrap bias-corrected percentile ..... .. 133
4.1.4 The bootstrap accelerated bias-corrected percentile. 135
4.1.5 The hybrid bootstrap 140
4.2 Asymptotic Theory. 141
4.2.1 Consistency. 141
4.2.2 Accuracy . . 144
4.2.3 Other asymptotic comparisons 152
4.3 The Iterative Bootstrap and Other Methods. 155
4.3.1 The iterative bootstrap ........ 155
4.3.2 Bootstrap calibrating ......... 160
4.3.3 The automatic percentile and variance stabilizing. 161
4.3.4 Fixed width bootstrap confidence intervals 164
4.3.5 Likelihood based bootstrap confidence sets 165
4.4 Empirical Comparisons. . . . . . . . . . . . . . . . 166
4.4.1 The bootstrap-t, percentile, BC, and BCa . 166
4.4.2 The bootstrap and other asymptotic methods 170
4.4.3 The iterative bootstrap and bootstrap calibration. 173
4.4.4 Summary...... 176
4.5 Bootstrap Hypothesis Tests 176
4.5.1 General description. 177
4.5.2 Two-sided hypotheses with nuisance parameters 179
4.5.3 Bootstrap distance tests . . . 182
4.5.4 Other results and discussions 184
4.6 Conclusions and Discussions. . . 188
Chapter 5. Computational Methods
5.1 The Delete-l Jackknife ....
5.1.1 The one-step jackknife
5.1.2 Grouping and random subsampling .
5.2 The Delete-d Jackknife ....
5.2.1 Balanced subsampling
5.2.2 Random subsampling
5.3 Analytic Approaches for the Bootstrap .
5.3.1 The delta method ....
5.3.2 Jackknife approximations
190
190
191
195
197
197
198
200
201
202
xiv
5.4
5.5
5.3.3 Saddle point approximations ...
5.3.4 Remarks...............
Simulation Approaches for the Bootstrap
5.4.1 The simple Monte Carlo method
5.4.2 Balanced bootstrap resampling
5.4.3 Centering after Monte Carlo
5.4.4 The linear bootstrap . . . . .
5.4.5 Antithetic bootstrap resampling
5.4.6 Importance bootstrap resampling .
5.4.7 The one-step bootstrap
Conclusions and Discussions . .
Contents
203
205
206
207
211
215
219
221
223
228
230
Chapter 6. Applications to Sample Surveys
6.1 Sampling Designs and Estimates
232
232
238
238
241
244
246
6.2 Resampling Methods ...... .
6.2.1 The jackknife ...... .
6.2.2 The balanced repeated replication
6.2.3 Approximated BRR methods
6.2.4 The bootstrap
6.3 Comparisons by Simulation 251
6.4 Asymptotic Results. . . . . 258
6.4.1 Assumptions . . . . 258
6.4.2 The jackknife and BRR for functions of averages 260
6.4.3 The RGBRR and RSBRR for functions of averages. 264
6.4.4 The bootstrap for functions of averages '" 267
6.4.5 The BRR and bootstrap for sample quantiles 268
6.5 Resampling Under Imputation
6.5.1 Hot deck imputation ..
6.5.2 An adjusted jackknife .
6.5.3 Multiple bootstrap hot deck imputation
6.5.4 Bootstrapping under imputation
6.6 Conclusions and Discussions ...... .
Chapter 7. Applications to Linear Models
7.1 Linear Models and Regression Estimates
7.2 Variance and Bias Estimation ..... .
270
271
273
277
278
281
283
283
285
Contents
7.2.1 Weighted and unweighted jackknives
7.2.2 Three types of bootstraps ..... .
7.2.3 Robustness and efficiency ..... .
7.3 Inference and Prediction Using the Bootstrap
7.3.1
7.3.2
7.3.3
7.3.4
Confidence sets . . . . . . . . . . .
Simultaneous confidence intervals .
Hypothesis tests
Prediction .
7.4 Model Selection . .
7.4.1 Cross-validation
7.4.2 The bootstrap .
7.5 Asymptotic Theory ...
7.5.1 Variance estimators
7.5.2 Bias estimators ...
7.5.3
7.5.4
7.5.5
Bootstrap distribution estimators .
Inference and prediction
Model selection . . . .
7.6 Conclusions and Discussions.
Chapter 8. Applications to Nonlinear, Nonparametric,
xv
285
289
292
295
295
298
301
303
306
307
311
313
313
318
320
324
326
329
and Multivariate Models 331 8.1 Nonlinear Regression. . . . . . . . . . . . . . . . . 331
8.2
8.1.1 Jackknife variance estimators ....... .
8.1.2 Bootstrap distributions and confidence sets
8.1.3 Cross-validation for model selection
Generalized Linear Models. . . . . .
8.2.1
8.2.2
8.2.3
Jackknife variance estimators
Bootstrap procedures ....
Model selection by bootstrapping .
8.3 Cox's Regression Models ...... .
8.3.1 Jackknife variance estimators
8.3.2 Bootstrap procedures ....
8.4 Kernel Density Estimation. . . . . .
8.4.1 Bandwidth selection by cross-validation
8.4.2 Bandwidth selection by bootstrapping
8.4.3 Bootstrap confidence sets . . . . . . .
333
335
337
338
340
341
343
345
346
349
350
351
353
356
xvi
8.5 Nonparametric Regression .......... .
8.5.1 Kernel estimates for fixed design .. .
8.5.2 Kernel estimates for random regressor
8.5.3 Nearest neighbor estimates
8.5.4 Smoothing splines . . . . .
8.6 Multivariate Analysis ....... .
8.6.1
8.6.2
8.6.3
8.6.4
Analysis of covariance matrix
Multivariate linear models.
Discriminant analysis ...
Factor analysis and clustering .
8.7 Conclusions and Discussions ..... .
Contents
360
360
364
366
370
373
373
376
379
382
384
Chapter 9. Applications to Time Series and Other Dependent Data 386
387
392
394
395
397
400
401
403
406
407
407
410
411
413
414
9.1 m-Dependent Data .... .
9.2 Markov Chains ...... .
9.3 Autoregressive Time Series
9.3.1 Bootstrapping residuals
9.3.2 Model selection
9.4 Other Time Series ..
9.4.1 ARMA(p, q) models
9.4.2 Linear regression with time series errors
9.4.3 Dynamical linear regression .. .
9.5 Stationary Processes .......... .
9.5.1 Moving block and circular block
9.5.2 Consistency of the bootstrap
9.5.3 Accuracy of the bootstrap.
9.5.4 Remarks ....... .
9.6 Conclusions and Discussions.
Chapter 10. Bayesian Bootstrap and Random Weighting 416
10.1 Bayesian Bootstrap. . . . . . . . . . . . . . . . . . . . . 416
10.1.1 Bayesian bootstrap with a noninformative prior. 417
10.1.2 Bayesian bootstrap using prior information 420
lO.1.3 The weighted likelihood bootstrap 422
10.1.4 Some remarks. . . . . . . . . . . . 424
Contents xvii
10.2 Random Weighting 425
10.2.1 Motivation 425
10.2.2 Consistency 427
10.2.3 Asymptotic accuracy. 429
10.3 Random Weighting for Functionals and Linear Models 434
10.3.1 Statistical functionals ....... 434
10.3.2 Linear models. . . . . . . . . . . . 437
10.4 Empirical Results for Random Weighting 440
10.5 Conclusions and Discussions.
Appendix A. Asymptotic Results A.1 Modes of Convergence . . . .
A.2 Convergence of Transformations.
A.3 0(·), 0(·), and Stochastic O( .), 0(·) A.4 The Borel-Cantelli Lemma. . . . . .
A.5 The Law of Large Numbers .....
A.6 The Law of the Iterated Logarithm .
A.7 Uniform Integrability ....
A.8 The Central Limit Theorem
A.9 The Berry-Esseen Theorem
A.10 Edgeworth Expansions ...
A.ll Cornish-Fisher Expansions.
Appendix B. Notation
References
Author Index
Subject Index
445
447 447
448
448
449
449
450
450
451
451
452
454
455
457
493
499