Part 3Part 3. Spectrum . Spectrum EstimationEstimation 3.1...

ENEE630 ENEE630 PartPart--33

Part 3Part 3. Spectrum . Spectrum EstimationEstimation3.1 3.1 Classic Methods for Spectrum EstimationClassic Methods for Spectrum Estimation

Electrical & Computer EngineeringElectrical & Computer Engineering

University of Maryland, College Park

Acknowledgment: ENEE630 slides were based on class notes developed by Profs. K.J. Ray Liu and Min Wu. The slides were made by Prof. Min Wu,

ith d t f M W i H Ch C t t i @ d d

UMD ENEE630 Advanced Signal Processing (ver.1111)

with updates from Mr. Wei-Hong Chuang. Contact: [email protected]

LogisticsLogistics

Last Lecture: lattice predictorcorrelation properties of error processes©

200

3)

– correlation properties of error processes– joint process estimator in lattice– inverse lattice filter structureat

ed b

y M

.Wu

inverse lattice filter structure

Today: S t ti ti b k d d l i l th d63

0 S

lides

(cre

– Spectrum estimation: background and classical methods

CP

ENEE

624/

6

Homework setUM

C

UMD ENEE630 Advanced Signal Processing (ver.1111) Nonparametric spectral estimation [2]

Summary of Related Readings on PartSummary of Related Readings on Part--IIII

2.1 Stochastic Processes and modelingHaykin (4th Ed) 1.1-1.8, 1.12-1.14 Hayes 3.3 – 3.7 (3.5); 4.7

2.2 Wiener filteringHaykin (4th Ed) Chapter 2Hayes 7.1, 7.2, 7.3.1

2 3 2 4 Li di ti d L i D bi i2.3-2.4 Linear prediction and Levinson-Durbin recursionHaykin (4th Ed) 3.1 – 3.3Hayes 7.2.2; 5.1; 5.2.1 – 5.2.2, 5.2.4– 5.2.5

2.5 Lattice predictorHaykin (4th Ed) 3.8 – 3.10

UMD ENEE630 Advanced Signal Processing (F'10) Parametric spectral estimation [3]

Hayes 6.2; 7.2.4; 6.4.1

Summary of Related Readings on PartSummary of Related Readings on Part--IIIIII

Overview Haykins 1.16, 1.10

3 1 Non-parametric method3.1 Non-parametric methodHayes 8.1; 8.2 (8.2.3, 8.2.5); 8.3

3 2 P t i th d3.2 Parametric methodHayes 8.5, 4.7; 8.4

3.3 Frequency estimationHayes 8.6

Review – On DSP and Linear algebra: Hayes 2.2, 2.3

UMD ENEE630 Advanced Signal Processing (F'10) Parametric spectral estimation [4]

– On probability and parameter estimation: Hayes 3.1 – 3.2

Spectrum Estimation: BackgroundSpectrum Estimation: Background

Spectral estimation: determine the power distribution in frequency of a random process©

200

3)

frequency of a random process– E.g “Does most of the power of a signal reside at low or high

frequencies?” “Are there resonances in the spectrum?”

ated

by

M.W

u ©

Applications:– Needs of spectral knowledge in spectrum domain non-causal

Wi fil i i l d i d ki b f i30 S

lides

(cre

a

Wiener filtering, signal detection and tracking, beamforming, etc.– Wide use in diverse fields: radar, sonar, speech, biomedicine,

geophysics, economics, …

CP

EN

EE

624/

63

g p y , ,

Estimating p.s.d. of a w.s.s. process is equivalent to estimate autocorrelation at all lags

UM

C


Spectral Estimation: ChallengesSpectral Estimation: Challenges

When a limited amount of observation data are available– Can’t get r(k) for all k and/or may have inaccurate estimate of r(k)©

200

3)

Can t get r(k) for all k and/or may have inaccurate estimate of r(k)– Scenario-1: transient measurement (earthquake, volcano, …) – Scenario-2: constrained to short period to ensure (approx.)

ated

by

M.W

u ©

stationarity in speech processing

30 S

lides

(cre

a

N

Mkknunukr 10][][1)(ˆ

Ob d d t h b t d b iCP

EN

EE

624/

63

kn

MkknunukN

kr1

,1,0 ],[][)(

Observed data may have been corrupted by noise

UM

C


Spectral Estimation: Major ApproachesSpectral Estimation: Major Approaches

Nonparametric methods– No assumptions on the underlying model for the data

© 2

003)

p y g– Periodogram and its variations (averaging, smoothing, …)– Minimum variance method

ated

by

M.W

u ©

Parametric methods– ARMA, AR, MA models

M i t th d30 S

lides

(cre

a

– Maximum entropy method

Frequency estimation (noise subspace methods)For harmonic processes that consist of a sum of sinusoids orC

P E

NE

E62

4/63

– For harmonic processes that consist of a sum of sinusoids or complex-exponentials in noise

High-order statistics

UM

C


g

Example of Speech SpectrogramExample of Speech Spectrogramu

© 2

002)

eate

d by

M. W

u08

G S

lides

(cre

UM

CP

ENEE

40U


Figure 3 of SPM May’98 Speech Survey

)

8000

“Sprouted grains and seeds are used in salads and dishes such as chop suey”W

ilson

© 2

004)

6000

cy (k

Hz)

by C

arol

Esp

y-W

2000

4000

Freq

uenc

F2

lides

(cre

ated

b

1.0 2.0 3.0 4.0 0.0

Time (sec)

EN

EE

408G

Sl

6000

8000

(kH

z)

“S t d”

UM

CP

2000

4000

requ

ency

(“Sprouted”


0.1 0.3 0.5

Fr

fricative stopconsonantglide vowel stopconsonantvowel

Section 3.1 Classical Nonparametric MethodsSection 3.1 Classical Nonparametric Methods©

200

3) Recall: given a w.s.s. process {x[n]} with

ated

by

M.W

u ©

)(]][][[]][[

krknxnxEmnxE x

30 S

lides

(cre

a

The power spectral density (p.s.d.) is defined as

)(]][][[ krknxnxE

k

fkjekrfP 2)()(

CP

EN

EE

624/

63

21

21

f

k

As we can take DTFT on a specific realization of a random process,What is the relation between the DTFT of a specific signal and the

UM

C ):2or ( f


What is the relation between the DTFT of a specific signal and the p.s.d. of the random process?

Ensemble Average of Squared Fourier MagnitudeEnsemble Average of Squared Fourier Magnitude

p.s.d. can be related to the ensemble average of the squared Fourier magnitude |X()|2

© 2

003)

q g | ( )|

ated

by

M.W

u ©

22][

121)(Consider

M

fnjM enx

MfP

30 S

lides

(cre

a 12 Mn

M M

M M

mnfjemxnx )(2][][1

i.e., take DTFT on (2M+1) samples and CP

EN

EE

624/

63

Mn Mm

emxnxM

][][12

, ( ) pexamine normalized magnitude

Note: for each frequency f, PM(f) is a random variable

UM

C


Ensemble Average of PEnsemble Average of PMM(f)(f)©

200

3)

M M

mnfjM emnr

MfPE )(2)(

121)]([

ated

by

M.W

u © Mn MmM 12

M

fkjekrkMM

22)()12(

121

Mfkjekr

k22)(1

30 S

lides

(cre

a MkM 212

Mk

ekrM2

)(12

1

M

fkM

fk2

22

2 1CP

EN

EE

624/

63

Mk

fkj

Mk

fkj ekrkM

ekr2

2

2

2 )(12

1)(

UM

C


Now, what if M goes to infinity?

P.S.D. and Ensemble Fourier MagnitudeP.S.D. and Ensemble Fourier Magnitude©

200

3) If the autocorrelation function decays fast enough s.t.

ated

by

M.W

u ©

)for rapidly 0 (i.e., )(

kr(k)krkk

fkj 2

30 S

lides

(cre

a

then

k

fkjMM

fPekrfPE )()()]([lim 2

p.s.d.

)( ][12

1lim)(2

2

M

Mn

fnj

Menx

MEfP

CP

EN

EE

624/

63

Thus

Mn

UM

C


3.1.1 Periodogram Spectral Estimator3.1.1 Periodogram Spectral Estimator©

200

3) (1) This estimator is based on (**)

ated

by

M.W

u ©

Given an observed data set {x[0], x[1], …, x[N-1]},the periodogram is defined as

30 S

lides

(cre

a p g21

2PER ][1)(

N

fnjenxN

fP

CP

EN

EE

624/

63

0)(

nNf

UM

C


An Equivalent Expression of PeriodogramAn Equivalent Expression of Periodogram20

04) The periodogram estimator can be given in terms of )(kr

N 1

d by

M.W

u ©

2

fkjN

NkekrfP 2

1

)1(PER )()(

0for )()( ];[][1)(1

0

kkrkrknxnxN

krkN

nSlid

es (c

reat

ed

where0n

– The quality of the estimates for the higher lags of r(k) may be poorer since they involve fewer terms of lag products in theM

CP

ENEE

630

poorer since they involve fewer terms of lag products in the averaging operation

Exercise: to show this from the periodogram definition in last page

UM


Exercise: to show this from the periodogram definition in last page

(2) Filter Bank Interpretation of Periodogram(2) Filter Bank Interpretation of Periodogram©

200

3)

For a particular frequency of f0:21

2 ][1)( 0

N

kfj kfP

ated

by

M.W

u ©

0

20PER ][)( 0

k

kfj kxeN

fP

21 N

30 S

lides

(cre

a

00

][][

nk

kxknhN

where

h i0

;0,1),...,1(for )2exp(1][ 0 Nnnfj

Nnh

CP

EN

EE

624/

63 where

otherwise0

– Impulse response of the filter h[n]: a windowed version of a

UM

C


complex exponential

Frequency Response of h[n]Frequency Response of h[n]©

200

3) )()1(exp)(i)(sin)( 0

0 ffNjffNffNfH

ated

by

M.W

u ©

)()(p)(sin

)( 00

ffjffN

f

sinc-like function centered at f0:

H(f) is a bandpass filterC f i f30

Slid

es (c

rea 0:

– Center frequency is f0

– 3dB bandwidth 1/N

CP

EN

EE

624/

63U

MC


Periodogram: Filter Bank PerspectivePeriodogram: Filter Bank Perspective

Can view the periodogram as an estimator of power spetrum that has a built-in filterbank©

200

3)

p– The filter bank ~ a set of bandpass filters– The estimated p.s.d. for each frequency f0 is the power of one

ated

by

M.W

u ©

p q y 0 poutput sample of the bandpass filter centering at f0

30 S

lides

(cre

aC

P E

NE

E62

4/63

21

0PER ][][)(

N

kxknhNfP

UM

C

00

n

k


E.g. White Gaussian ProcessE.g. White Gaussian Process[Lim/Oppenheim Fig.2.4] Periodogram of zero-mean white Gaussian noise using N-point data record: N=128, 256, 512, 1024

© 2

003)

ated

by

M.W

u ©

30 S

lides

(cre

aC

P E

NE

E62

4/63

The random fluctuation (measured by variance) of the i d d t d ith i i N

UM

C


periodogram does not decrease with increasing N periodogram is not a consistent estimator

(3) How Good is Periodogram for Spectral Estimation?(3) How Good is Periodogram for Spectral Estimation?00

3/20

04)

?)( p.s.d. will, If PER fPPN

Estimation: Tradeoff between bias and varianced by

M.W

u ©

20

Slid

es (c

reat

ed

For white Gaussian process, we can show that at fk = k/N

ENE

E62

4/63

0 U

MC

P E


Performance of Periodogram: SummaryPerformance of Periodogram: Summary

The periodogram for white Gaussian process is an unbiased estimator but not consistent

© 2

003)

– The variance does not decrease with increasing data length– Its standard deviation is as large as the mean (equal to the quantity

to be estimated)ated

by

M.W

u ©

to be estimated)

Reasons for the poor estimation performance– Given N real data points, the # of unknown parameters {P(f0), … 30

Slid

es (c

rea

p , p { ( 0),P(fN/2)} we try to estimate is N/2, i.e. proportional to N

Similar conclusions can be drawn for processes withCP

EN

EE

624/

63

Similar conclusions can be drawn for processes with arbitrary p.s.d. and arbitrary frequencies– Asymptotically unbiased (as N goes to infinity) but inconsistent

UM

C


3.1.2 Averaged Periodogram3.1.2 Averaged Periodogram

As one solution to the variance problem of periodogram– Average K periodograms computed from K sets of data records

© 2

003)

g p g p

ated

by

M.W

u ©

1 (m)

PERPER AV )(1)(K

fPfP

30 S

lides

(cre

a 0

V )()(m

fK

f

where 212

(m)

PER ][1)(

L

fnjenxfP

CP

EN

EE

624/

63

0PER ][)(

n

m enxL

fP

And the K sets of data records are

UM

C

... ;10 ],[ ];1[ ..., ],0[{ 100 LnnxLxx}10 ],1[{ 1 LnnxK


}],[{ 1K

Performance of Averaged PeriodogramPerformance of Averaged Periodogram

– If K sets of data records are uncorrelated with each other,we have: ( fi = i/L )

© 2

003)

( fi )

ated

by

M.W

u ©

1

30 S

lides

(cre

a

)(1 2ifP

K

CP

EN

EE

624/

63

KK for 0Varand,Var i.e., KK for 0Var and ,Var i.e.,

UM

C

KK for 0Varand,Var i.e., ,,i.e., consistent estimatei.e., consistent estimate


,,i.e., consistent estimate

Practical Averaged PeriodogramPractical Averaged Periodogram

Usually we partition an available data sequence of length Ninto K non overlapping blocks each block has length L (i e N=KL)©

200

3)

– into K non-overlapping blocks, each block has length L (i.e. N=KL)

ated

by

M.W

u ©

i.e. 1 ..., ,1 ,0 ,][][ LnmLnxnxm

110 K

Since the blocks are contiguous, the K sets of data records 30 S

lides

(cre

a 1...,,1,0 Km

gmay not be completely uncorrelated– Thus the variance reduction factor is in general less than K

CP

EN

EE

624/

63

Periodogram averaging is also known as the Bartlett’s method

UM

C


Averaged Periodogram for Fixed Data SizeAveraged Periodogram for Fixed Data Size

Given a data record of fixed size N, will the result be better if we segment the data into more and more subrecords?

© 2

003)

We examine for a real-valued stationary process:

ated

by

M.W

u ©

1 1 )(K m

30 S

lides

(cre

a )(ˆ)(1)( )0(PER

0

)(

PERPER AV fPEfPK

EfPEm

CP

EN

EE

624/

63

identical distribution for all mNote

1

2)0()0(PER )(ˆ)(ˆ

LfljelrfP

UM

C

where

)1(Ll

lL

lnxnxlr1

)0( ][1)(ˆ

an equivalent expression to definition in terms of x[n]


n

lnxnxL

lr0

][)( of x[n]

Mean of Averaged PeriodogramMean of Averaged Periodogram©

200

3)at

ed b

y M

.Wu

©30

Slid

es (c

rea

CP

EN

EE

624/

63U

MC


Mean of Averaged Periodogram (cont’d)Mean of Averaged Periodogram (cont’d)©

200

3) fkrkwfPE )}(][{DTFT)](ˆ[ PER AV 1

multiplication in time

ated

by

M.W

u ©

dPfW )()(21

21

)( fP

convolution in frequency

Biased estimate (both averaged and regular periodogram)– The convolution with the window function w[k] lead to the mean of30

Slid

es (c

rea )( fP

– The convolution with the window function w[k] lead to the mean of the averaged periodogram being smeared from the true p.s.d

Asymptotic unbiased as L

CP

EN

EE

624/

63

– To avoid the smearing, the window length L must be large enough so that the narrowest peak in P(f) can be resolved

This gives a tradeoff between bias and variance

UM

C


gSmall K => better resolution (smaller smearing/bias) but larger variance

NonNon--parametric Spectrum Estimation: Recapparametric Spectrum Estimation: Recap

Periodogram – Motivated by relation between p.s.d. and squared magnitude of DTFT

of a finite-size data record– Variance: won’t vanish as data length N goes infinity ~ “inconsistent”– Mean: asymptotically unbiased w r t data length N in generalMean: asymptotically unbiased w.r.t. data length N in general

equivalent to apply triangular window to autocorrelation function(windowing in time gives smearing/smoothing in freq domain)

unbiased for white Gaussian unbiased for white Gaussian

Averaged periodogram– Reduce variance by averaging K sets of data record of length L eachReduce variance by averaging K sets of data record of length L each– Small L increases smearing/smoothing in p.s.d. estimate thus higher

bias equiv. to triangular windowing


Windowed periodogram: generalize to other symmetric windows

Case Study on NonCase Study on Non--parametric Methodsparametric Methods Test case: a process consists of narrowband components

(sinusoids) and a broadband component (AR)– x[n] = 2 cos(1 n) + 2 cos(2 n) + 2 cos(3 n) + z[n]

where z[n] = a1 z[n-1] + v[n], a1 = 0.85, 2 = 0.1 /2 = 0 05 /2 = 0 40 /2 = 0 421/2 = 0.05, 2/2 = 0.40, 3/2 = 0.42

– N=32 data points are available periodogram resolution f = 1/32

Examine typical characteristics of various non-parametric pspectral estimators

(Fig.2.17 from Lim/Oppenheim book)


3.1.3 Periodogram with Windowing3.1.3 Periodogram with Windowing

Review and Motivation

© 2

003)

The periodogram estimator can be given in terms of )(k

ated

by

M.W

u © The periodogram estimator can be given in terms of )(kr

1

2PER )()(

NfkjekrfP

30 S

lides

(cre

a )1( Nk

where )()( ];[][1)(1

kN

krkrknxnxN

kr

– The higher lags of r(k), the poorer estimates since the estimates

CP

EN

EE

624/

63

)()(][][)(0nN 0for k

g g ( ) pinvolve fewer terms of lag products in the averaging operation

Solution: weigh the higher lags less

UM

C


– Trade variance with bias

WindowingWindowing Use a window function to weigh the higher lags less

© 2

003)

ated

by

M.W

u ©

30 S

lides

(cre

a

w(0)=1 preserves variance r(0)

CP

EN

EE

624/

63

Effect: periodogram smoothing – Windowing in time Convolution/filtering the periodogram

UM

C


g g p g– Also known as the Blackman-Tukey method

Common Lag WindowsCommon Lag Windows Much of the art in non-parametric spectral estimation is in

choosing an appropriate window (both in type and length)

© 2

003)

ated

by

M.W

u ©

30 S

lides

(cre

aC

P E

NE

E62

4/63

UM

C


Table 2.1 common lag window (from Lim-Oppenheim book)

Discussion: Estimate r(k) via Time AverageDiscussion: Estimate r(k) via Time Average Normalizing the sum of (N-k) pairs

by a factor of 1/N ? v.s. by a factor of 1/(N-k) ?Biased (low variance) Unbiased (may not non-neg. definite)

• Hints on showing the non-negative definiteness: using )(kr

definiteness: using to construct correlation matrix

)(1 kr


HW#8)(For 2 kr

3.1.43.1.4 Minimum Variance Spectral Estimation Minimum Variance Spectral Estimation (MVSE)(MVSE)

Recall: filter bank perspective of periodogram– The periodogram can be viewed as estimating the p s d byThe periodogram can be viewed as estimating the p.s.d. by

forming a bank of narrowband filters with sinc-like response– The high sidelobe can lead to “leakage” problem:

large output power due to p.s.d outside the band of interest

MVSE designs filters to minimize the leakage from out-of-MVSE designs filters to minimize the leakage from out ofband spectral components– Thus the shape of filter is dependent on the frequency of interest

d d t d tiand data adaptive(unlike the identical filter shape for periodogram)


– MVSE is also referred to as the Capon spectral estimator

Main Steps of MVSE MethodMain Steps of MVSE Method

Design a bank of bandpass filters Hi(f) with center frequency fi so thati

– Each filter rejects the maximum amount of out-of-band power– And passes the component at frequency fi without distortion

Filter the input process { x[n] } with each filter in the filter bank and estimate the power of each output processbank and estimate the power of each output process

Set the power spectrum estimate at frequency fi to be the power estimated above divided by the filter bandwidth


Formulation of MVSEFormulation of MVSE

The MVSE designs a filter H(f) for each f f i t t ffrequency of interest f0

minimize the output power

dffPfH )()( 221

1

minimize the output power

dffPfH )()(21

subject to 1)( 0 fH

(i.e., to pass the components at f0 w/o distortion)


Deriving MVSE SolutionsDeriving MVSE Solutions


Solution of MVSE (cont’d)Solution of MVSE (cont’d)

The optimal filter: eRhT 1

The optimal filter:

f

eReh

TH 1

TTHTH 1It follows that eRRhhRh TTHTH 1

ehH 1 eRe

ehTH 1


MVSE: SummaryMVSE: Summary

If choosing the bandpass filters to be FIR of length p, its 3dB-b.w. is approximately 1/pg p, pp y p

Thus the MVSE ismatrixncorrelatio

is ˆ ppR

eRe

pfPTH 1MV

ˆ)(

matrix ncorrelatio

)2exp(

1fj

e

(i.e. normalize by filter b.w.)

eRe

))1(2exp( pfj

e

MVSE is a data adaptive estimator and provides improved resolution over periodogram– Also referred to as “High-Resolution Spectral Estimator”


Also referred to as High Resolution Spectral Estimator– Does not assume a particular underlying model for the data

Recall: Case Recall: Case Study on NonStudy on Non--parametric parametric Methods Methods Test case: a process consists of narrowband components

(sinusoids) and a broadband component (AR)– x[n] = 2 cos(1 n) + 2 cos(2 n) + 2 cos(3 n) + z[n]

where z[n] = a1 z[n-1] + v[n], a1 = 0.85, 2 = 0.1 /2 = 0 05 /2 = 0 40 /2 = 0 421/2 = 0.05, 2/2 = 0.40, 3/2 = 0.42

– N=32 data points are available periodogram resolution f = 1/32

Examine typical characteristics of various non-parametric pspectral estimators

(Fig.2.17 from Lim/Oppenheim book)


Deriving MVSE SolutionsDeriving MVSE Solutions


Output Power From H(f) filterOutput Power From H(f) filter

From the filter bank perspective of periodogram:0

)1(

2][)(Nn

fnjenhfH

Thus

dffPelhekh fljfkj )(][][0

221

1

02

0 0

21

)(2)(][][ klfj dfefPlhkh

ffNlNk

)(][][)1(2

1)1(

)1( )1( 2

1 )(][][Nk Nl

dfefPlhkh

0 0

)(][][ klrlhkhEquiv. to filter r(k) with { h(k) h*(-k) } and evaluate at

)1( )1(

)(][][Nk Nl

klrlhkh


and evaluate at output time k=0

MatrixMatrix--Vector Form of MVSE FormulationVector Form of MVSE Formulation

Define

The constraint can be written in vector form as 1ehH

)( 0fH

Thus the problem becomes

hRh THmin subject to 1ehH

hj


Solution of MVSESolution of MVSE )1(2Re ehhRhJ HTHdef

Use Lagrange multiplier approach for solving the constrained optimization problem

– Define real-valued objective function s.t. the stationary condition can be derived in a simple and elegant way based on the theorem for complex derivative/gradient operatorsp g p

)1()1(min*

,ehehhRhJ HHTH

h

eheRh HT 1

11 and

)1()1( * heehhRh HHTH

00either * ehRJ T

eReT

TH

1

11

00or

00either

**

*

eRhJ

ehRJTTH

h

h

eRe

eRhTH

T

1

1


00 ehRehR THT

Part 3Part 3. Spectrum . Spectrum EstimationEstimation 3.1...

Documents

Transcript of Part 3Part 3. Spectrum . Spectrum EstimationEstimation 3.1...