8.2Fisher Information CR Inequality J I

Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

8.2 Fisher Information CR Inequality

• X ∼ f (x, θ), θ ∈ Θ = (a, b).

• Regular cases:

f (x, θ) > 0,∂

∂θ

∫f =

∫∂

∂θf,

http://lsec.cc.ac.cn/~guoxh

Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

Since∫f (x, θ) = 1, then∫

∂f (x, θ)

∂θdx = 0,

i.e. ∫∂ ln f (x, θ)

∂θf (x, θ)dx = 0.


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

⇒∫

[∂2 ln f (x, θ)

∂θ2f (x, θ)

+∂ ln f (x, θ)

∂θ

∂f (x, θ)

∂θ]dx = 0,

⇒∫

(∂ ln f (x, θ)

∂θ)2f (x, θ)dx

+

∫∂2 ln f (x, θ)

∂θ2f (x, θ)dx = 0.


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

Therefore,

I(θ) =:

∫ (∂ ln f (x, θ)

∂θ

)2

f (x, θ)dx

= E

[∂ ln f (x, θ)

∂θ

]2

= −∫∂2 ln f (x, θ)

∂θ2f (x, θ)dx

= −E∂2 ln f (x, θ)

∂θ2.


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

• The larger of I(θ), the more information

of θ.

• If I(θ) = 0, then ∂ ln f∂θ = 0. No informa-

tion in ln f (x, θ).

• ∂ ln f (x,θ)∂θ is an important function.


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

MLE θ̂ satisfies:

n∑i=1

∂ ln f (xi, θ̂)

∂θ= 0.


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

Example 1: X ∼ N(θ, σ20),

f (x, θ) =1√

2πσ0

exp{−(x− θ)2

2σ20

}.

Then

ln f (x, θ) = − ln(√

2πσ0)

−(x− θ)2

2σ20

.


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

⇒∂ ln f (x, θ)

∂θ=x− θσ2

0

,

⇒∂2 ln f (x, θ)

∂θ2= − 1

σ20

.

Then

I(θ) =1

σ20

.


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

Example 2: X ∼ B(1, θ), then

ln f (x, θ) = x ln θ

+ (1− x) ln(1− θ),

∂2 ln f (x, θ)

∂θ2= − x

θ2− (1− x)

(1− θ)2.


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

Hence,

I(θ) = E

[x

θ2+

1− x(1− θ)2

]

=1

θ+

1

1− θ=

1

θ(1− θ).


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

• SupposeX1, · · · , Xn iid. ∼ f (x, θ), then

L(θ) =

n∏i=1

f (xi, θ)

and

lnL(θ) =

n∑i=1

ln f (xi, θ).


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

⇒∂ lnL(θ)

∂θ=

n∑i=1

∂ ln f (xi, θ)

∂θ.

In(θ) = E[∂ lnL(θ)

∂θ]2

=

n∑i=1

E(∂ ln f (xi, θ)

∂θ)2

= nI(θ).


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

• Rao-Cramer inequalityµ

Y = u(X1, · · · , Xn) ∼ θ,

E(Y ) = k(θ).

⇒ k(θ) =

∫u(x1, · · · , xn)∏

f (xi, θ)dxi


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

k′(θ) =

∫u(x1, · · · , xn)

(

n∑i=1

∂ ln f (xi, θ)

∂θ)∏

f (xi, θ)dxi.


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

Let Z =∑n

i=1∂ ln f (xi,θ)

∂θ , then

EZ = 0,

V ar(Z) = nE

[∂ ln f (X, θ)

∂θ

]2

= nI(θ).


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

Since k′(θ) = E(Y Z), then

k′(θ) ≤√V ar(Y )V ar(Z)

⇒

V ar(Y ) ≥ [k′(θ)]2

V ar(Z)

=[k′(θ)]2

nI(θ).


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

• If k(θ) = θ, then Y is unbiased and

V ar(Y ) ≥ 1

nI(θ)

CR-lower bound.


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

Example 1µX1, · · · , Xn iid. ∼N(θ, σ2)with known σ2. Then

V ar(Y ) ≥ σ2

n.

Example 2µX1, · · · , Xn iid. ∼ B(1, θ).

Then

V ar(Y ) ≥ θ(1− θ)

n.


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

Def 1: Y is a efficient unbiased estimator

of θ⇐⇒ the variance of Y attains the CR

lower bound.

Def 2: EfficiencyThe CR lower bound

V ar(Y )

is called the efficiency of unbiased estima-

tor Y .


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

Example 3: X1, · · · , Xn iid. ∼ N(µ, θ),

0 < θ < +∞ with known µ.

E(S2) = θ.

and

ln f (x, θ) = −(x− µ)2

2θ− ln θ

2+ c.


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

I(θ) = −E

[∂ ln2 f (X, θ)

∂θ2

]

= E

[(X − µ)2

θ3− 1

2θ2

]=

1

2θ2.

and1

nI(θ)=

2θ2

n.


Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

Since

V ar(S2) =2(n− 1)

(n− 1)2θ2 =

2θ2

n− 1

>2θ2

n,

then

eff(S2) =n− 1

n.


Contents

Home Page

Title Page

JJ II

J I

of 100

Go Back

Full Screen

Close

Quit

Dra

ft

Take S20 = 1

n

∑ni=1(Xi − µ)2, then

eff(S20) = 1.

• Asymptotically efficient:

eff(Yn)→ 1 (n→∞).


8.2Fisher Information CR Inequality J I

Documents

Transcript of 8.2Fisher Information CR Inequality J I