SufficientSufficientStatisticsStatistics

DayuDayu

11.1111.11

Some AbbreviationsSome Abbreviations• i.i.d. : independent, identically

distributed

ContentContent• Estimator, Biased, Mean Square Error

(MSE) and Minimum-Variance Unbiased Estimator (MVUE)When MVUE is unique?

• Lehmann–Scheffé Theorem– Biased– Complete– Sufficient

• the Neyman-Fisher factorization criterion

How to construct MVUE is unique?• Rao-Blackwell theorem

EstimatorEstimator• The probability mass function (or

density) of X is partially unknown, i.e. of the form f(x;θ) where θ is a parameter, varying in the parameter space Θ.

Unbiased• An estimator is said to be unbiased

for a function if it equals in expectation i.e.

• E.g using mean of a sample to estimate mean of the population

is unbiased

t(x)ˆ̂

}ˆ{E

x

n

ii

n

ii

n

ii n

nxE

nxE

nx

nExE

111

1)(

1)(

1)

1()(

Mean Squared Error (MSE)Mean Squared Error (MSE)• MSE of an estimator T of an

unobservable parameter θ isMSE(T)=E[(T- θ)2]

• Since E(Y2)=V(Y)+[E(Y)]2

MSE(T)=var(T)+[bias(T)]2

where bias(T)=E(T- θ)=E(T)- θ• For the unbiased one, MSE=V(T)

since biasd(T)=0

ExamplesExamples

Two estimators for σ2 :

Results from MLE, biased, butsmaller variance

Unbiased, but bigger variance

Minimum-Variance Unbiased Minimum-Variance Unbiased Estimator (MVUE)Estimator (MVUE)

• An unbiased estimator of minimum MSE also has minimum variance.• MVUE is an unbiased estimator of

parameters, whose variance is minimized for all values of the parameters.

• Two theorems– Lehmann-Scheffé theorem can show that MVUE

is unique. – Constructing a MVUE: Rao-Blackwell theorem

Lehmann–Scheffé TheoremLehmann–Scheffé Theorem

• any estimator that is complete, sufficient, and unbiased is the unique best unbiased estimator of its expectation.

• The Lehmann-Scheffé Theorem states that if a complete and sufficient statistic T exists, then the UMVU estimator of g(θ) (if it exists) must be a function of T.

CompletenessCompleteness• Suppose a random variable X has a probability

distribution belonging to a known family of probability distributions, parameterized by θ,

• A function g(X) is an unbiased estimator of zero if the expectation E(g(X)) remains zero regardless of the value of the parameter θ. (by the definition of unbiased)

• Then X is a complete statistic precisely if it admits (up to a set of measure zero) no such unbiased estimator of zero except 0 itself.

Example of CompletenessExample of Completeness• suppose X1, X2 are i.i.d. random variables,

normally distributed with expectation θ and variance 1.

• Not complete: Then X1 — X2 is an unbiased estimator of zero. Therefore the pair (X1, X2) is not a complete statistic.

• Complete: On the other hand, the sum X1 + X2 can be shown to be a complete statistic. That means that there is no non-zero function g such that E(g(X1 + X2 )) remains zero regardless of changes in the value of θ.

Detailed ExplanationsDetailed Explanations

• X1 + X2~(2θ,2)

SufficiencySufficiency• Consider an i.i.d. sample X1, X2,.. Xn

• Two people A and B:– A observe the entire sample X1, X2,.. Xn

– B observes only one number T,T=T(X1, X2,.. Xn)

• Intuitionly, Who has more information?

• Under what condition, B will have as much information about θ as A has?

SufficiencySufficiency• Definition:

– A statistic T(X) is sufficient for θ precisely if the conditional probability distribution of the data X given the statistic T(X) does not depend on θ.

• How to find?: the Neyman-Fisher factorization criterion: If the probability density function of X is f(x;θ), then T satisfies the factorization criterion if and only if functions g and h can be found such that

• h(x): a function that does not depend on θ• g(T(x),θ): a function that depends on data

only throught T(x)• E.g.

• T=x1+x2+.. +xn is a sufficient statistic for p for Bernoulli Distribution B(p)

g(T(x),p)∙1 h(x)=1

Example 2Example 2Test T=x1+x2+.. +xn for Poisson Distribution Π(λ):

g(T(x), λ)h(x): independent of λ

Hence, T=x1+x2+.. +xn is sufficient!

Notes on Sufficient StatisticsNotes on Sufficient Statistics• Note that the sufficient statistic is not

unique. If T(x) is sufficient, so are T(x)/n and log(T(x))

Rao-Blackwell theoremRao-Blackwell theorem• named after

– C.R. Rao (1920- ) is a famous Indian statistician and currently professor emeritus at Penn State University

– David Blackwell (1919-) is Professor Emeritus of Statistics at the UC Berkeley

• describes a technique that can transform an absurdly crude estimator into an estimator that is optimal by the mean-squared-error criterion or any of a variety of similar criteria.

Rao-Blackwell theoremRao-Blackwell theorem• Definition: A Rao–Blackwell estimator δ1(X) of an

unobservable quantity θ is the conditional expected value E(δ(X) | T(X)) of some estimator δ(X) given a sufficient statistic T(X). – δ(X) : the "original estimator"

– δ1(X): the "improved estimator".

• The mean squared error of the Rao–Blackwell estimator does not exceed that of the original estimator.

Conditional ExpectationConditional Expectation

BxBxP

xPBx

BxxP

xfBxxPBxfE

bxfXxB

Bx

)(

)(,0

)|(

)()|()|)((

})(|{

Example IExample I• Phone calls arrive at a switchboard

according to a Poisson process at an average rate of λ per minute.

• λ is not observable• Observe: the numbers of phone calls that

arrived during n successive one-minute periods are observed.

• It is desired to estimate the probability e−λ that the next one-minute period passes with no phone calls.

t=x1+x2+.. +xn is sufficient

Original estimator:

Example IIExample II• To estimate λ for X1 … Xn ~ P(λ)

• Original estimator: X1

We know t= X1 +…+ Xn is sufficient

• Improved estimator by R-B theorem:E[X1| X1 +…+ Xn =t] cannot compute directly

We know Σ[E(Xi| X1 +…+ Xn =t)]

=E(ΣXi| X1 +…+ Xn =t)=t

• Since X1 … Xn are i.i.d. so every term is t/n

In fact, it’s x

Thank you!Thank you!