Summer School in Statisticsfor Astronomers VI

June 7-11, 2010

Robustness, Nonparametrics and Some Inconvenient Truths

Tom Hettmansperger

Dept. of Statistics

Penn State University

Robust methods

t-tests and F-test

rank tests

Least squares

Nonparametrics

Some ideas we will explore:

Robustness

Nonparametric Bootstrap

Nonparametric Density Estimation

Nonparametric Rank Tests

Tests for (non-)Normality

The goal: To make you worry or at least thinkcritically about statistical analyses.

AbstractPopulation Distribution, Model

Real World Data

Probability andExpectation

StatisticalInference

Statistical Model, Population Distribution

Research Hypothesis or Question in English

Measurement, Exp. Design, Data Collection

Translate Res. Hyp. or Quest. into a statement in terms of the model parameters

Select a relevant statistic

Carry out statistical inference Graphical displays Model criticism

Sampling Distributions P-values Significance levels Confidence coefficients

State Conclusions and Recommendations in English

Parameters in a population or model

Typical Values: mean, median, mode

Spread: variance (standard deviation), interquartile range (IQR)

Outliers

Shape: probability density function (pdf), cumulative distribution function (cdf)

NGC 4382 (n = 59) 26.215 26.506 26.542 26.551 26.553 26.607 26.612 26.674 26.687 26.699 26.703 26.727 26.740 26.747 26.765 26.779 26.790 26.800 26.807 ... 27.161 27.169 27.179

Research Question: How large are the luminosities in NGC 4382?

Measure of luminosity (data below)

Traditional model: normal distribution of luminosity

Translate Res. Q.: What is the mean luminosity of the population?(Here we use the mean to represent the typical value.)

The relevant statistic is the sample mean.

n

SX 2

orig: 26.905 + .0524no: 26.917 + .047424.000: 26.867 + .1094

Statistical Inference: 95% confidence interval for the meanusing a normal approximation to the sampling distribution of the mean.

27.227.026.826.626.426.2

Median

Mean

27.00026.97526.95026.92526.90026.87526.850

1st Quartile 26.765Median 26.9743rd Quartile 27.042Maximum 27.179

26.853 26.957

26.915 27.010

0.170 0.246

A-Squared 1.54P-Value < 0.005

Mean 26.905StDev 0.201Variance 0.040Skewness -1.06045Kurtosis 1.08094N 59

Minimum 26.215

Anderson-Darling Normality Test

95% Confidence I nterval for Mean

95% Confidence I nterval for Median

95% Confidence I nterval for StDev95% Confidence I ntervals

Summary for NGC 4382

NGC 4382_24NGC 4382_25NGC 4382_26NGC 4382_origNGC 4382_no

27.5

27.0

26.5

26.0

25.5

25.0

24.5

24.0

Data

Boxplot of NGC 4382 no, NGC 4382, NGC 4382_26, NGC 4382_25, ...

Variable N N* Mean SE Mean StDevNGC 4382_no 58 0 26.917 0.0237 0.181NGC 4382_orig 59 0 26.905 0.0262 0.201NGC 4382_26 59 0 26.901 0.0280 0.215NGC 4382_25 59 0 26.884 0.0400 0.307NGC 4382_24 59 0 26.867 0.0547 0.420

Minimum Q1 Median Q326.506 26.776 26.974 27.04626.215 26.765 26.974 27.04226.000 26.765 26.974 27.04225.000 26.765 26.974 27.04224.000 26.765 26.974 27.042

Outliers can have arbitrarily large impact on the sample mean, samplestandard deviation, and sample variance.

A single outlier can increase the width of the t-confidence interval and inflate the margin of error for the sample mean. Inference canbe adversely affected.

First Inconvenient Truth:

Second Inconvenient Truth:

It is bad for a small portion of the data to dictatethe results of a statistical analysis.

Third Very Inconvenient Truth:

The construction of a 95% confidence interval for the population variance is very sensitive to theshape of the underlying model distribution.

The standard interval computed in most statisticalpackages assumes the model distribution is normal.

If this assumption is wrong, the resulting confidencecoefficient can vary significantly.

I am not aware of a stable 95% confidence interval for the population variance.

The ever hopeful statisticians

Robustness: structural and distributional

Influence or sensitivity curves: The rate of change in a statistic as an outlier is varied.

Breakdown: The smallest fraction of the data thatmust be altered to carry the statistic beyond anypreset bound.

We want bounded influence and high breakdown.

Structural: We would like to have an estimator and a test statistic that are not overly sensitiveto small portions of the data.

Distributional robustness:

We want a sampling distribution for the teststatistic that is not sensitive to changes ormisspecifications in the model or populationdistribution.

This type of robustness provides stablep-values for testing and stable confidencecoefficients for confidence intervals.

NGC 4382_24NGC 4382_25NGC 4382_26NGC 4382_origNGC 4382_no

27.00

26.95

26.90

26.85

26.80

26.75

Data

95% conf int for pop mean, x denotes the sample median

Mean

Median

NGC 4382_24NGC 4382_25NGC 4382_26NGC 4382_origNGC 4382_no

27.025

27.000

26.975

26.950

26.925

26.900

26.875

26.850

Data

95% conf int for pop median, + denotes sample mean

Message: The sample mean is not structurally robust; whereas, the median is structurally robust.

It takes only one observation to move the sample mean anywhere. It takes roughly 50% of the data to move the median. (Breakdown)

Sensitivity Curve: ]ˆˆ)[1()( 1 nnnxSC

SCmean(x) = x

SCmedian(x) = (n+1)x(r) if x < x(r)

(n+1)x if x(r) < x < x(r+1)

(n+1)x(r+1) if x(r+1) < x when n = 2r

Mean

Median

x

Influence

Mean has linear, unbounded influence.Median has bounded influence.

Some good news: The sampling distribution of the sample mean depends only mildly on the populationor model distribution. (A Central Limit Theorem effect)

Provided our data come from a model with finitevariance, for large sample size

S

Xn

][ has an approximate standardnormal distribution (mean 0 and variance 1).

This means that the sample mean enjoys distributionalrobustness, at least approximately. We say that thesample mean is asymptotically nonparametric.

More inconvenient truth: the sample variance is neitherstructurally robust (unbounded sensitivity and breakdowntending to 0), but also lacks distributional robustness.

Again, from the Central Limit Theorem:

][ 22 Sn

Provided our data come from a model with finitefourth moment, for large sample size

has an approximate normal distribution with mean 0 and variance:

4

44 )(

),1(

XEwhere

is called the kurtosis

Approx true

Conf Coeff

3 .948

4.2 .877

5 .834

9 .674

The kurtosis and is a measure of the tail weight of a model distribution. It is independent of location and scale and has value 3 for any normal model.

Assuming 95% confidence:

43210-1-2-3

99

95

90

80

70

605040

30

20

10

5

1

t5

Perc

ent

Mean 0.08663StDev 1.105N 50AD 0.211P-Value 0.850

Normal - 95% CIProbability Plot of t5, Kurtosis = 9

A very inconvenient truth:A test for normality will alsomislead you!!

Some questions:

1. If statistical methodology based on sample meansand sample variances is non robust, what can we do?

Are you concerned about the last least squares analysisyou carried out? (t-tests and F-tests) If not, you should be!

2. What if we want to simply replace the mean by themedian as the typical value? The sample median is robust, at least structurally. What about the distribution?

3. The mean and the t-test go together. What test goeswith the median?

We know that:

n

Sbyestimated

nmeanSE ,)(

How to find SE(median) and estimate it.

Two ways:

1. Nonparametric Bootstrap (computational)

2. Estimate the standard deviation of the approximating normal distribution. (theoretical)

Result for NGC4382: SE(median) = .028

(.027 w/o the outlier, .028 w outlier = 24)

Nonparametric Bootstrap;

1. Draw a sample of size 59 from the original NGC4382data. Sample with replacement.

2. Compute and store the sample median.

3. Repeat B times. (I generally take B = 4999)

4. The histogram of the B medians is an estimate of sampling distribution of the sample median.

5. Compute the standard deviation of the B medians.This is the approximate SE of the sample median.

Theoretical (Mathematical Statistics) Moderately Difficult

Let M denote the sample median. Provided the density (pdf) of the model distribution is not 0 at the model median,

][ Mn has an approximate normal distributionwith mean 0 and variance 1/[4f2(

In other words, SE(median) = )(2

1

fn

and we must estimate the value of the density at the population median.

where f(x) is the density and is the model median.

Let f(x) denote a pdf. Based on a sample of sizen we wish to estimate f(x0) where x0 is given.

Define:

n

ii

h

XxK

hnxf

10

0

11)(ˆ

Where K(t) is called the kernel and

dttKtdtttKdttK K )(,0)(,1)( 22

Nonparametric density estimation:

Then a bit of calculation yields:

...)(2

1)())(ˆ( 2

02

00 KxfhxfxfE

And a bit more:

...)()(1

)(ˆ( 200 dttKxf

nhxfV

And so we want: nhandh 0

The density estimate does not much depend on K(t), thekernel. But it does depend strongly on h, the bandwidth.

We often choose a Gaussian (normal) kernel: 2

2

1

2

1 tetK

Next we differentiate the integrated mean squared error andset it equal to 0 to find the optimal bandwidth (indept of x0).

5/1

5/1

22

2

)]([

)(

n

dxxf

dttKh

K

opt

If we choose the Gaussian kernel and if f is normal then:

IQRSwherenhn 75.,minˆ)ˆ06.1( 5/1

Recall, SE(median) =)(2

1

fn

For NGC4382: n = 59, M = 26.974

approxasyMf

MSE .,031.)(ˆ259

1)(

Bootstrap result for NGC4382: SE(median) = .028 finite sample approx

Final note: both bootstrap and density estimate are robust.

The median and the sign test (for testing H0: = 0) are related through the L1 norm.

2)]([#)(

][#2)sgn()()(

))]([sgn(||)(

nSEwithXS

nXXSDd

d

XXXD

i

ii

iii

To test H0: = 0 we use S+(Xi > 0 which has a nullbinomial sampling distribution with parameters n and .5.

This test is nonparametric and very robust.

Research Hypothesis: NGC4494 and NGC4382 differin luminosity.

Luminosity measurements (data)

NGC 4494 (m = 101) 26.146 26.167 26.173…26.632 26.641 26.643NGC 4382 (n = 59) 26.215 26.506 26.542…27.161 27.169 27.179

Statistical Model Two normal populations with possiblydifferent means but with the same variance.

Translation: H0: 4494 = vs. H0:

Select a statistic: The two sample t statisticNGC 4494 (m = 101), NGC 4382 (n = 59)

m

S

n

S

XX

nmnm

SnSm

XXt

24382

24494

43824494

24382

24494

43824494

)11

2

)1()1(

The two sided t-test with significanc level .05rejects the null hyp when |t| > 2.

Recall that means and variances are not robust.

VERY STRANGE!

Ratio of variances

Ratio

of

sample

sizes

1/4 1/1 3/1

1/4 .01 .05 .15

1/1 .05 .05 .05

4/1 .18 .05 .01

Table of true values of the significance level whenthe assumed level is .05.

Another inconvenient truth: the true significance levelcan differ from .05 when some model assumptions fail.

An even more inconvenient truth:

These problems extend to analysis of variance and regression.

Seek alternative tests and estimates.

We already have alternatives to the mean and t-test: the robust median and sign test.

We next consider nonparametric rank tests and estimatesfor comparing two samples. (Competes with the two sample t-test and difference in sample means.)

)(,...,

)(,...,

1

1

yGfromsampleaYY

xFfromsampleaXX

n

m

Generally suppose: )()( yFyG

To test H0: or to estimate we introduce

2

)1()(]0[#)0(

nnYRXYS jij

)( jYR is the rank of Yj in the combined data.

The robust estimate of is

)(ˆ, ijji XYmedian

Provides the robustness Provides the comparison

As opposed to

)(1

ij XYmn

XY

which is not robust.

Research Hypothesis: NGC4494 and NGC4382 differin luminosity.

Luminosity measurements (data)

X: NGC 4494 (n = 101) 26.146 26.167 26.173…26.632 26.641 26.643Y: NGC 4382 (n = 59) 26.215 26.506 26.542…27.161 27.169 27.179

Statistical Model Two normal populations with possiblydifferent medians but with the same scale.

Translation: H0: vs. H0: 0

Mann-Whitney Test and CI: NGC 4494, NGC 4382

N MedianX: NGC 4494 101 26.659Y: NGC 4382 59 26.974

Point estimate for Delta is 0.253

95.0 Percent CI for Delta is (0.182, 0.328)

Mann-Whitney test:Test of Delta = 0 vs Delta not equal 0 is significant at 0.0000 (P-Value)

NGC 4382NGC 4494

27.2

27.0

26.8

26.6

26.4

26.2

26.0

Data

85% CI-Boxplot of NGC 4494, NGC 4382

What to do about truncation.

1. See a statistician

2. Read the Johnson, Morrell, and Schick reference. and thensee a statistician.

Here is the problem: Suppose we want to estimate the difference in locationsbetween two populations: F(x) and G(y) = F(y – d).

But (with right truncation at a) the observations come from

ayforandayfordaF

dyFyG

axforandaxforaF

xFxF

a

a

1)(

)()(

1)(

)()(

Suppose d > 0 and so we want to shift the X-sample to the right toward the truncation point. As we shift the Xs, some will pass the truncation point andwill be eliminated from the data set. This changes the sample sizes and requires adjustment when computing the corresponding MWW to see ifit is equal to its expectation. See the reference for details.

d̂

Computation of shift estimate with truncation

d m n S+ E(S+)

.253 88 59 .510 4750.5 4366.0

.283 84 59 .360 4533.5 4248.0

.303 83 59 .210 4372.0 4218.5

.323 81 59 .080 4224.5 4159.5

.333 81 59 -.020 4144.5 4159.5

.331 81 59 -.000 4161.5 4159.5

Comparison of NGC4382 and NGC 4494

Point estimate for d is .253 W = 6595.5 (sum of ranks of Ys)

S+ = 4825.5

m = 101 and n = 59

Recall the two sample t-test is sensitive to the assumptionof equal variances.

The Mann-Whitney test is less sensitive to the assumptionof equal scale parameters.

The null distribution is nonparametric. It does not dependon the common underlying model distribution.

It depends on the permutation principle: Under the nullhypothesis, all (m+n)! permutations of the data areequally likely. This can be used to estimate thep-value of the test: sample the permutations,compute and store the MW statistics, then find theproportion greater than the observed MW.

Here’s a bad idea:

Test the data for normality using, perhaps, the Kolmogorov-Smirnov test.

If the test ‘accepts’ normality then use a t-test, and if it rejects normality then use a rank test.

You can use the K-S test to reject normality.

The inconvenient truth is that it may accept manypossible models, some of which can be very disruptive to the t-test and sample means.

Absolute MagnitudePlanetary Nebulae

Milky WayAbs Mag (n = 81) 17.537 15.845 15.449 12.710 15.499 16.450 14.695 14.878 15.350 12.909 12.873 13.278 15.591 14.550 16.078 15.438 14.741 …

Abs Mag-6.0-7.2-8.4-9.6-10.8-12.0-13.2-14.4

Dotplot of Abs Mag

Abs Mag

Perc

ent

-5.0-7.5-10.0-12.5-15.0-17.5

99.9

99

95

90

80706050403020

10

5

1

0.1

Mean

0.567

-10.32StDev 1.804N 81AD 0.303P-Value

Probability Plot of Abs Mag

Normal - 95% CI

Abs Mag - Threshold

Perc

ent

101

99.999

9080706050403020

10

5

32

1

0.1

Shape

0.224P-Value >0.500

2.680Scale 5.027Thresh -14.79N 81AD

Probability Plot of Abs Mag

3-Parameter Weibull - 95% CI

But don’t be too quick to “accept” normality:

)68.2(

0)(exp{)(

)(

:

1

exampletheinshapec

scaleb

thresholdt

otherwiseandtxforb

tx

b

txcxf

onDistributiWeibull

cc

c

Null Hyp: Pop distribution, F(x) is normal

)())](1)(([))()(( 12 xdFxFxFxFxFnAD n

|)()(|max xFxFD n

The Kolmogorov-Smirnov Statistic

The Anderson-Darling Statistic

A Strategy:

Use robust statistical methods whenever possible.

If you must use traditional methods (sample means,t and F tests) then carry out a parallel analysis usingrobust methods and compare the results. Start toworry if they differ substantially.

Always explore your data with graphical displays.Attach probability error statements whenever possible.

What more can we do robustly?

1. Multiple regression

2. Analysis of designed experiments (AOV)

3. Analysis of covariance

4. Multivariate analysis

There’s more:

The rank based methods are 95% efficient relative to the least squares methods when the underlying model isnormal.

They may be much more efficient when the underlying model has heavier tails than a normal distribution.

But time is up.

References:1. Hollander and Wolfe (1999) Nonpar Stat Methods

2. Sprent and Smeeton (2007) Applied Nonpar Stat Methods

3. Kvam and Vidakovic (2007) Nonpar Stat with Applications to Science and Engineering

4. Johnson, Morrell, and Schick (1992) Two-Sample Nonparametric Estimation and Confidence Intervals Under Truncation, Biometrics, 48, 1043-1056.

5. Hettmansperger and McKean (2010) Robust Nonparametric Statistics, 2nd Ed.

6. Efron and Tibshirani (1993) An Introduction to the bootstrap

7. Arnold Notes, Bendre Notes

Thank you for listening!