Maximum Likelihood Estimator

All of Statistics (Chapter 9)

Outline

• MLE• Properties of MLE

– Consistency– Asymptotic normality – Efficiency– Invariance

Definition of MLE

• Likelihood Function =

• Log Likelihood Function

• MLE is the value that maximizes

• Joint density function

Definition of MLE

• MLE is the value that maximizes

Properties of MLE

MLE has the following nice properties:• Consistency:

• Asymptoticly Normal:

• Asymptotic optimality: MLE has the smallest variance

• Invariance Property

0 0ˆ( ) 1MLEP n

Consistency:

1. is the maximizer of 2. is the maximizer of . 3. we have in probability by Law of large numbers

4. Based on 1,2,3

Scaled Log-likelihood Function

Expectation:

0 0ˆ( ) 1P n

Outline of Proof

0 0ˆ( ) 1P n

is the maximizer of . 2:

For any , we have

Since

Proof:

in probability

• Law of large numbers: sample average converges to expectation in probability (proved by Chebyshev's inequality)

• Sample average:

• Expectation:

3:

1. is the maximizer of2. is is the maximizer of . 3. we have in probability by LLN.

Target: Based on 1,2,3

Consistency:

The distributions of the estimators become more and more concentrated near the true value of the parameter being estimated.

MLE is Asymptotically Normal

Fisher Information• Notation:

Fisher Information is defined as

Measure how quickly pdf will change

Larger fisher information pdf changes quickly at

can be well distinguished from the distribution with other parameters

easier to estimate based on data0

0

Taking the derivative:

Since is pdf:

Equivalently,

Writing (4) as an expectation

Differentiate(4):

=?

Second term:

First term:

Theorem. (Asymptotic normality of MLE.)

Since MLE is maximizer of , we have

By Mean Value Theory:

Convergence in Distribution by Central Limit Theory

First, consider the numerator

Next, consider the denominator : Since Convergence in Prob. by LLN.

(1)

(2)

Combine (1) and (2), we get

Theorem. (Asymptotic normality of MLE.)

With the normal property, we can generate confidence bounds and hypothesis tests for the parameters.

Asymptotic optimal (efficient)

• Cramér–Rao bound expresses a lower bound on the variance of estimators

• The variance of an unbiased estimator is bounded by:

• MLE:

• MLE has the smallest asymptotic variance and we say that the MLE is asymptotically efficient and asymptotically optimal.

MLE is a unbiased estimator with smallest variance

Functional invariance( | ), f X ~ x

( )g

( | ), ( )f g X ~ x ( )g

An invertible mapping

where

. .

Outline of Proof* 1 1( ) ( ) ( ( )) ( ( )) ( )L f f g L g L x x xx | x |

* ( )L xˆ( )g Thus, the maximum of is attained at

Discussion

• Questions??

Asymptotic optimal (efficient)MLE has the smallest asymptotic variance and we say that the MLE is

asymptotically efficient and asymptotically optimal.

Statistic Y is called efficient estimator of iff the variance of Y attains the Rao-Cramer lower bound.

Let Y is a statistic with mean then we have

When Y is an unbiased estimator of , then the Rao-Cramer inequality becomes

When n converges to infinity, MLE is a unbiased estimator with smallest variance