crlb notes

Estimation: chapter 2+3

Minimum variance unbiased estimation + the CRLB

Natasha [email protected]://www.ece.uic.edu/~devroye

Spring 2011

• Find a few estimators

Estimation: a first example

• Estimate the DC level, A, of a signal given noisy measurements x[0], x[1], ... x[N-1] where

x[n] are samples of this!

• Compare their performance { • mean?• variance?• pdf?

Estimation: a first example

• Estimators of the DC level, A

Estimation: definitions

Estimation: definitions

How would you pick a ``good’’ estimator?

Vector versions....

Minimum variance unbiased estimation

Why?

So?

Minimum variance unbiased estimation

Give a counter-example! (b1pg.20)

The Cramer-Rao Lower Bound

• the CRLB give a lower bound on the variance of ANY UNBIASED estimator

• does NOT guarantee bound can be obtained

• IF find an estimator whose variance = CRLB then it’s MVUE

• otherwise can use Ch.5 tools (Rao-Blackwell-Lehmann-Scheffe Theorem and Neyman-Fisher Factorization Theorem) to construct a better estimator from any unbiased one - possibly the MVUE if conditions are met

The Cramer-Rao Lower Bound (CRLB)

• Use?

• Intuition?

The Cramer-Rao Lower Bound (CRLB)

CRLB examples

CRLB proof

CRLB T or F

What is I(θ)?

• Why “information”?

• non-negative• additive for independent observations

Vector form of the CRLB

Vector form of the CRLB

Vector form of the CRLB examples

What can we conclude?

CRLB for transformations

Vector CRLB for transformations

Example of vector CRLB with transformation

CRLB for General Gaussian Case

• When observations are Gaussian and one knows the dependence of the mean and covariance matrix on the unknown parameters, we know the closed form of the CRLB (or Fisher information matrix):

CRLB for Gaussians examples

Example: range estimation

!"#$

%&'(

)*+

,)+)+)+ !"

#"

! $$##%#"#&

!

!!

!

"#$$"%#!"!#$

-.&/0%12345607833 "+#)333333/9/:7.939;7.3#&<,($"=

>7?71;73>7@67?219/8333 "+#'( !!)

A7&05.73-1%73B76&C8 !!

!"#$%&'()*$+,-.*

D&/E61%127E

FG1273H&5001&/

#$"

"+#)

#$

"+#''( !!)D4I

J3K%L

&+#)

/%0$,1$!2"3

)'*(*

#$45

678$9$:)('.$6;<&=)<&,($/>,?*.=

Continuous time signal model

!"#$

%&'(

)*+

,)+)+)+ !"

#"

! $$##%#"#&

!

!!

!

"#$$"%#!"!#$

-.&/0%12345607833 "+#)333333/9/:7.939;7.3#&<,($"=

>7?71;73>7@67?219/8333 "+#'( !!)

A7&05.73-1%73B76&C8 !!

!"#$%&'()*$+,-.*

D&/E61%127E

FG1273H&5001&/

#$"

"+#)

#$

"+#''( !!)D4I

J3K%L

&+#)

/%0$,1$!2"3

)'*(*

#$45

678$9$:)('.$6;<&=)<&,($/>,?*.=

Taken from http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522.htm


Discrete time signal model


!"#$

%&'()*+,-*./+! 0+#"1!"2*3

#4$5+0+#6$!7

89+:;<=*+

>&?22<&@+AB<2*

C&.+"! " !"#

#DD#DE547654 #$%#!$ %$$#$&$' ( !&

)"

#$%&'(&$)%*

&#"1!

!*!

#"! !"1!

+,-&'(&$)%*

"#.! "! " !"

#

''(

'')

*

#++%

#%++%#!

#++

$

#54

#5476

#E54

54

%$+$$#

+$$$$#$&

$$$#

$'

(

(((

(

&

&4$F&(5G ;&2++ @B@HI*.B+2&'()*2+2=&.=<@J+&=+$(

/0123&45678067'1&-9:&,7210;&<'=3;

!"#$

%&'()*+,-*./+! 0+#"1!"2*3

#4$5+0+#6$!7

89+:;<=*+

>&?22<&@+AB<2*

C&.+"! " !"#

#DD#DE547654 #$%#!$ %$$#$&$' ( !&

)"

#$%&'(&$)%*

&#"1!

!*!

#"! !"1!

+,-&'(&$)%*

"#.! "! " !"

#

''(

'')

*

#++%

#%++%#!

#++

$

#54

#5476

#E54

54

%$+$$#

+$$$$#$&

$$$#

$'

(

(((

(

&

&4$F&(5G ;&2++ @B@HI*.B+2&'()*2+2=&.=<@J+&=+$(

/0123&45678067'1&-9:&,7210;&<'=3;



!"#$

%&'()**+,(-.)/0)10(2345(16-7+.(8&1(-9:/)+(;(<=%>

!!

!!

"

# $#

"%

# "$#

"%

#

"

#

&&'

())*

+

,

,#

&&

'

(

))

*

+

,

,

#

&&'

())*

+

,

"$,#

&&'

())*

+

,

,-

#

?

@

@

#@

@

# @

@

#

?

@

@

ABAB

ABCDE

AFG)1B

!

" "#

!"

"" "#

!"

"" $

$

%

" $

$

$

#

#&

#

#&

"&"&

$

$ $

$

$

..

/

/

.

/

/

./

/

H+7:(9/I()/0(J66*(

/&/KL61&(.61M-

NO*+&9.(2)+P7+7-QQQ R-6()**1&O9M).9&/>(/$'('$ "

$

ST6/(0&(PT)/:6(&8(G)19)U+6-QQ

!"#$%&'()*+")*,#&-!./



!"#$

!

"

#

!

"

#

"#

$

%&&

&!

'

$%&

&

&!

'

%&&

&! !!!

!!! "#

$%&

'((

)

"#

$%&

'((

)

"#

$%&

'((

*

+

%

&

%

&

%

&

&

'(

&"

#

'(

&"

'(#')*+,(

,-

-../012.+03412.3+567826.2.0+449 +33,:;<2./02=>267?18,+49

!)!"

! %&&!$%

& '(

. /

. /

!!

!

0

01

0

01

)

+

%&()

%(()(

'

$

$

%(()(

'

$

#

!

!

"

#

!

#!

&

&&

%

&&

'(

'(&

&"

#

'(&

&"

#')*+,(

2

2- @A2AB1:,10

C2D+,.1*+4

D+,.1*+4. /

!

!0

01

0

01)%&()

%(()(*+,!

&

&&

'(

'(&2

!"#$%"&'&()&*"'+,-".

*+,! 6.2EFGH2IJK22(LM'-2?>312:N2EHOFK

/'%0"&1+2$*'2$3%&4/5(&673%289



!"#$

%&'()*+,-&-*'.-/&*0-*/11'2-*/+*+,-*3456*7(*+,-*.-8/9:

;<&-=#

;>2/1< ?

?!"#

$

%&'( !"#

@7*)-+*+,-*3456*7(*+,-*!"#$%A*B&-*C+1/(&DE*7D*F/1G&H*1-&B8+:*

;<GI"

;>2/1< ?

?

?

!"#%&'(

)(

!"

$*(+%

(*(+%

$( ##

>

?

> $$%

&''(

)

*

*+ 0'+,* ( J*)#

$,"*?

3456*'&*'(2-1&-89*F17F71+'7(/8*+7:

K LM4*N-/&B1-

K 4NL*6O*N-/&B1-

3456*'&*'(2-1&-89*F17F71+'7(/8*+7:

K LM4*N-/&B1-

K 4NL*6O*N-/&B1-

L7*+,-*3456*+-88&*B&A

K 3,77&-*&')(/8*0'+,*8/1)-*%!"#

K P(&B1-*+,/+*LM4*'&*8/1)-

K 6-++-1*7(*M-/1Q9"8/1)-*+/1)-+&

K O,'=,*'&*Q-++-1R**

S T7BQ8-*+1/(&G'++-.*-(-1)9R

S T7BQ8-*4NL*Q/(.0'.+,R

&"#$%'()*+,"*+-#'.&/0'12-#*34

!"#$

%&'()*+,-&-*'.-/&*0-*/11'2-*/+*+,-*3456*7(*+,-*.-8/9:

;<&-=#

;>2/1< ?

?!"#

$

%&'( !"#

@7*)-+*+,-*3456*7(*+,-*!"#$%A*B&-*C+1/(&DE*7D*F/1G&H*1-&B8+:*

;<GI"

;>2/1< ?

?

?

!"#%&'(

)(

!"

$*(+%

(*(+%

$( ##

>

?

> $$%

&''(

)

*

*+ 0'+,* ( J*)#

$,"*?

3456*'&*'(2-1&-89*F17F71+'7(/8*+7:

K LM4*N-/&B1-

K 4NL*6O*N-/&B1-

3456*'&*'(2-1&-89*F17F71+'7(/8*+7:

K LM4*N-/&B1-

K 4NL*6O*N-/&B1-

L7*+,-*3456*+-88&*B&A

K 3,77&-*&')(/8*0'+,*8/1)-*%!"#

K P(&B1-*+,/+*LM4*'&*8/1)-

K 6-++-1*7(*M-/1Q9"8/1)-*+/1)-+&

K O,'=,*'&*Q-++-1R**

S T7BQ8-*+1/(&G'++-.*-(-1)9R

S T7BQ8-*4NL*Q/(.0'.+,R

&"#$%'()*+,"*+-#'.&/0'12-#*34

crlb notes

Documents

Transcript of crlb notes