Fourier Methods of Spectral Estimation

C.S.Ramalingam

Department of Electrical EngineeringIIT Madras

C.S.Ramalingam Fourier Methods of Spectral Estimation

Outline

Definition of Power Spectrum

Deterministic signal example

Power Spectrum of a Random Process

The Periodogram Estimator

The Averaged Periodogram

Blackman-Tukey Method

Use of Data Windowing in Spectral Analysis

Spectrogram: Speech Signal Example

C.S.Ramalingam Fourier Methods of Spectral Estimation

What is Spectral Analysis?

Spectral analysis is the estimation of the frequency content ofa random process

By “frequency content” we mean the distribution of powerover frequency

Also called Power Spectral Density, or simply spectrum

What frequency components are present?What is the intensity of each component?

C.S.Ramalingam Fourier Methods of Spectral Estimation

The Earliest Spectral Analyzer

All colours are present with equal intensity

C.S.Ramalingam Fourier Methods of Spectral Estimation

What is Frequency?

Our notion of frequency comes from sin(2πf0t) and cos(2πf0t)

both are called “sinusoids”

Frequency ≡ Sinusoidal Frequency: f0 cycles/sec (Hz)

exp(j2πf0t) is the basis function needed for representing acomponent with frequency f0

for an arbitrary frequency component, it becomes exp(j2πft)

As f varies from −∞ to ∞, we get the Fourier basis set!

f is sometimes called “Fourier frequency”

C.S.Ramalingam Fourier Methods of Spectral Estimation

Spectral Analysis ≡ Expansion Using Fourier Basis

Spectral analysis is nothing but expanding a signal x(t) usingthe Fourier basis

X (f ) =

∫ ∞−∞

x(t) exp(−j2πft) dt ← inner product!

“X (f ) is the continuous-time Fourier transform of x(t)”

|X (f1)| large ⇒ dominant frequency component at f = f1

|X (f1)| = 0⇒ no frequency component at f = f1

Plot of |X (f )|2 as a function of f is called the “powerspectrum”

C.S.Ramalingam Fourier Methods of Spectral Estimation

Deterministic Signal Example

−2 0 2 4 6 8 10 12 14 16

−1

0

1

Time (ms)

Gated sinusoid with f0 = 1 kHz

−4000 −3000 −2000 −1000 0 1000 2000 3000 40000

20

40

60

Frequency (Hz)

Mag

nitu

de

−4000 −3000 −2000 −1000 0 1000 2000 3000 4000−20

0

20

40

Frequency (Hz)

Log

Mag

nitu

de

C.S.Ramalingam Fourier Methods of Spectral Estimation

What About PSD of a Random Process?

“Spectral analysis is the estimation of the frequency contentof a random process”

An ensemble of sample waveforms constitute a randomprocess

X (f )?=

∫ ∞−∞

x(t) exp(−j2πft) dt

Does it exist?Even if it does, is it meaningful?

We’ll focus on discrete-time random processes, i.e., ensembleof x [n], where n ∈ Z

C.S.Ramalingam Fourier Methods of Spectral Estimation

PSD of a WSS Random Process

Let x [n] be a complex wide-sense stationary process

Its autocorrelation sequence (ACS) is defined as

rxx [k] = E{x∗[n] x [n + k]}

Wiener-Khinchine Theorem:

Pxx(f ) =∞∑−∞

rxx [k] exp(−j2πfk) − 1

2≤ f ≤ 1

2

That is, ACSDTFT←→ PSD

C.S.Ramalingam Fourier Methods of Spectral Estimation

An Alternative Definition for PSD

If the ACS decays sufficiently rapidly,

Pxx(f ) = limM→∞

E

1

2M + 1

∣∣∣∣∣M∑−M

x [n] exp(−j2πfn)

∣∣∣∣∣2

The so-called “Direct Method” is based on the above formula

C.S.Ramalingam Fourier Methods of Spectral Estimation

Why is the Problem Difficult?

ACS is not available

Finite number of samples from one realization

We are only given x [0], x [1], . . . , x [N − 1]

No “best” spectral estimator exists

Many practical signals, such as speech, are non-stationary

Pxx(f ) obtained from given data is a random variable

Bias versus Variance trade-off

C.S.Ramalingam Fourier Methods of Spectral Estimation

The Periodogram Estimator

Recall

Pxx(f ) = limM→∞

E

1

2M + 1

∣∣∣∣∣M∑−M

x [n] exp(−j2πfn)

∣∣∣∣∣2

In practice we drop

limM→∞

because data are finite

the expectation operator E since we have only one realization

The Periodogram estimator is defined as

P̂PER(f )def=

1

N

∣∣∣∣∣N−1∑n=0

x [n] exp(−j2πfn)

∣∣∣∣∣2

“Direct Method”, since it deals with the data directly

C.S.Ramalingam Fourier Methods of Spectral Estimation

Example: Two Sine Waves + Noise

x [n] =√

10 exp(j 2π 0.15n) +√

20 exp(j 2π 0.2n) + z [n]

z [n] ∼ complex N (0, 1), N = 20

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5−30

−20

−10

0

10

20

30

40

50

Frequency

Mag

nitu

de (

dB)

C.S.Ramalingam Fourier Methods of Spectral Estimation

Periodogram is a Biased Estimator For Finite Data

For finite N, periodogram is a biased estimator

Bias is the difference between the true and expected values

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5−30

−20

−10

0

10

20

30

40

50

Frequency

Mag

nitu

de (

dB)

averagednoiseless

N = 20

C.S.Ramalingam Fourier Methods of Spectral Estimation

Periodogram: Bias Decreases With Increasing N

If data length is increased, bias decreases:

limN→∞

E{

P̂xx(f )}

= Pxx(f )

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5−30

−20

−10

0

10

20

30

40

50

60

Frequency

Mag

nitu

de (

dB)

N = 100

C.S.Ramalingam Fourier Methods of Spectral Estimation

The More (Samples) the Merrier?

For most estimators, bias and variance decrease withincreasing N

An estimator is said to be consistent if

limN→∞

Pr(∣∣∣θ̂ − θ∣∣∣ > ε

)= 0

where θ̂ is the estimate of θ

This implies that, as N →∞,

bias → 0variance → 0

C.S.Ramalingam Fourier Methods of Spectral Estimation

Is the Periodogram Consistent?

Consider white noise sequence for various N

True Pxx(f ) = constant

If the Periodogram estimator were consistent,P̂xx(f )→ constant as N increases

Consider noise sequences of length 32, 64, 128, and 256

N = 32; % white noise sequencex = randn(N,1); % of length 32

Does P̂xx(f ) tend to a constant as N increases?

C.S.Ramalingam Fourier Methods of Spectral Estimation

White Noise Example

−0.5 0 0.5

0

40PSD of White Noise

−0.5 0 0.5

0

40

−0.5 0 0.5

0

40

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5

0

40

Frequency

N=256

N=32

As N increases, variance of the estimate does not decrease

Periodogram is an inconsistent estimator

C.S.Ramalingam Fourier Methods of Spectral Estimation

What Went Wrong?

“In practice we drop

limM→∞

because data are finite

the expectation operator E since we have only one realization”

For white noise, increasing the data length did not help

What can be done to capture the benefits of E{·} ?

C.S.Ramalingam Fourier Methods of Spectral Estimation

Averaging: The Poor Man’s Expectation Operator

Expectation operator can be approximated by averaging

Averaged Periodogram:

P̂AVPER(f ) =1

M

M∑m=1

P̂(m)PER (f )

where P̂(m)PER (f ) is periodogram of m-th segment of length N

For independent data records

var{

P̂AVPER(f )}

=1

Mvar{

P̂PER(f )}

C.S.Ramalingam Fourier Methods of Spectral Estimation

Averaged Periodogram of White Noise

Result of averaging 8 periodograms

−0.5 −0.25 0 0.25 0.5−10

−5

0

5

10

15

20

25

30

35

40

Frequency

Mag

nitu

de (

dB)

Averaged Periodogram

C.S.Ramalingam Fourier Methods of Spectral Estimation

Variance Decreases, But Bias Increases!

Two Sines + Noise example

−0.5 −0.25 0 0.25 0.5−50

−40

−30

−20

−10

0

10

20Averaged Periodogram for Two Sines + Noise

Frequency

Mag

nitu

de (

dB)

N=256, M=1N=64, M=4N=16, M=16

C.S.Ramalingam Fourier Methods of Spectral Estimation

Welch’s Method

Overlapping blocks by 50%

Reduces variance without worsening bias

−0.5 −0.25 0 0.25 0.5−25

−20

−15

−10

−5

0

5

10

15Welch’s Method

Frequency

Mag

nitu

de (

dB)

Block Length = 64No. of blocks = 7

Overlap = 50%

C.S.Ramalingam Fourier Methods of Spectral Estimation

Why Did The Periodogram Fail?

Periodogram was defined as

P̂PER(f ) = 1N

∣∣∣∑N−1n=0 x [n] exp(−j2πfn)

∣∣∣2Equivalent to

P̂PER(f ) =N−1∑−(N−1)

r̂xx [k] exp(−j2πfk)

where

r̂xx [k] =

1

N

N−1−k∑n=0

x∗[n] x [n + k] k = 0, 1, . . . ,N − 1

r̂∗xx [−k] k = −(N − 1), . . . ,−1

Note that r̂xx [N − 1] = x∗[0]x [N − 1]/N

No averaging ⇒ estimate with high variance!

C.S.Ramalingam Fourier Methods of Spectral Estimation

Blackman-Tukey Method

Recall

Pxx(f ) =∞∑−∞

rxx [k] exp(−j2πfk) − 1

2≤ f ≤ 1

2

In practice: (a) replace rxx [k] by estimate r̂xx [k], (b) truncatethe summation, and (c) apply “lag window”

P̂BTf ) =M∑−M

w [k] r̂xx [k] exp(−j2πfk)

where

0 ≤ w [k] ≤ w [0] = 1 w [k] = 0 for |k| > Mw [−k] = w [k] W (f ) ≥ 0

“Indirect Method”, since it does not deal with the datadirectly

C.S.Ramalingam Fourier Methods of Spectral Estimation

Example: Two Sine Waves + Noise

Data length N = 100, Correlation Lag M = 10

−0.5 −0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4 0.5−30

−20

−10

0

10

20

30

40

50

60

Frequency

Mag

nitu

de (

dB)

C.S.Ramalingam Fourier Methods of Spectral Estimation

Periodogram Vs. Blackman-Tukey

−0.5 −0.25 0 0.25 0.5

−20

0

20

40

60

Mag

nitu

de (

dB)

Periodogram

−0.5 −0.25 0 0.25 0.5

−20

0

20

40

60

Frequency

Mag

nitu

de (

dB)

Blackman−Tukey

Blackman-Tukey method: reduction in variance comes at theexpense of increased bias

Speech Analysis

C.S.Ramalingam Fourier Methods of Spectral Estimation

Data Windowing in Spectral Analysis

Useful for data containing sinusoids + noise

Sidelobes of a stronger sinusoid may mask the main lobe of anearby weak sinusoid

We multiply x [n] by data window w [n] before computingperiodogram

Weaker sinusoid becomes more visible

Main lobe of each sinusoid broadens: two close peaks maymerge into one

C.S.Ramalingam Fourier Methods of Spectral Estimation

Example: How Many Sine Waves Are There?

0.1 0.12 0.14 0.16 0.18 0.2 0.22−60

−40

−20

0

20

40

60

Frequency

Mag

nitu

de (

dB)

How Many Sinusoids Are There?

C.S.Ramalingam Fourier Methods of Spectral Estimation

Example: Three Sine Waves

0.1 0.12 0.14 0.16 0.18 0.2 0.22−60

−40

−20

0

20

40

60

Frequency

Mag

nitu

de (

dB)

Three Sinusoids: Rectangular Window

0.150.15 0.157

C.S.Ramalingam Fourier Methods of Spectral Estimation

Example: Three Sine Waves

0.1 0.12 0.14 0.16 0.18 0.2 0.22−60

−40

−20

0

20

40

60

Frequency

Mag

nitu

de (

dB)

Three Sinusoids: Hanning Window

0.150.15 0.157

C.S.Ramalingam Fourier Methods of Spectral Estimation

Commonly Used Windows

Name w [k] Fourier transform

Rectangular 1 WR(f ) =sin πf (2M + 1)

sin πf

Bartlett 1− |k |M

1

M

(sin πfM

sin πf

)2

Hanning 0.5 + 0.5 cosπk

M0.25 WR

(f − 1

2M

)+ 0.5 WR(f ) +

0.25 WR

(f + 1

2M

)Hamming 0.54 + 0.46 cos

πk

M0.23 WR

(f − 1

2M

)+ 0.54 WR(f ) +

0.23 WR

(f + 1

2M

)w [k] = 0 for |k | > M

C.S.Ramalingam Fourier Methods of Spectral Estimation

Hamming Vs. Hanning

−0.05 −0.025 0 0.025 0.05−60

−40

−20

0

Frequency

Mag

nitu

de (

dB)

−0.5 −0.25 0 0.25 0.5

−100

−50

0Fourier Transforms of Hamming and Hanning Windows

Frequency

Mag

nitu

de (

dB)

HammingHanning

C.S.Ramalingam Fourier Methods of Spectral Estimation

Three Sine Waves

0.1 0.12 0.14 0.16 0.18 0.2 0.22−60

−40

−20

0

20

40

60

Frequency

Mag

nitu

de (

dB)

Rectangular Vs. Hamming Vs. Hanning

0.150.15 0.157

C.S.Ramalingam Fourier Methods of Spectral Estimation

How Can We Analyze Non-Stationary Signals?

Consider a “linear chirp”, i.e., a signal whose frequencyincreases linearly from f1 Hz to f2 Hz over a time interval T

What is its magnitude spectrum?

0 0.01 0.02 0.03 0.04 0.05−1

0

1

Time

Am

plitu

de

0 200 400 600 800 10000

20

40

60

Frequency

Mag

nitd

ue (

dB)

C.S.Ramalingam Fourier Methods of Spectral Estimation

Need a More Useful Representation

In Fourier analysis, even if a signal is non-stationary, it is stillrepresented using stationary sinusoids

An unsatisfactory approach

Power spectrum is identical to x(−t), whose frequencydecreases from f2 to f1

x(t) and x(−t) differ only in the phase of the Fouriertransform

What we really want to know is how frequency varies withtime

Can it still be called “frequency” ?

C.S.Ramalingam Fourier Methods of Spectral Estimation

Spectrogram

Plot of power spectrum of short blocks of a signal as afunction of time

Over each short block, signal is considered to be stationary

Speech is a classic example of a commonly occurringnon-stationary signal

Voiced sounds: /a/, /e/, /i/, /o/, /u/ (quasi-periodic)

Unvoiced sounds: /s/, /sh/, /f/ (noise-like)

Plosives: /p/, /t/, /k/ (transient sounds)

C.S.Ramalingam Fourier Methods of Spectral Estimation

Spectrogram of Linear Chirp

Time

Fre

quen

cy

0 0.1 0.2 0.3 0.40

100

200

300

400

500

600

C.S.Ramalingam Fourier Methods of Spectral Estimation

Non-stationarity in Speech Signal

1.16 1.17 1.18 1.19 1.2 1.21 1.22 1.23 1.24−0.2

0

0.2

/k/

1.24 1.26 1.28 1.3 1.32 1.34 1.36 1.38−1

0

1

/ow/

1.75 1.8 1.85 1.9−0.05

0

0.05

Time

/s//s/

C.S.Ramalingam Fourier Methods of Spectral Estimation

Application to Speech Analysis

0 0.25 0.5 0.75 1 1.25 1.5 1.75 2−1

−0.5

0

0.5

1Should We Chase Those Cowboys?

Am

plitu

de

Time

Fre

quen

cy

0 0.25 0.5 0.75 1 1.25 1.5 1.75 20

1000

2000

3000

4000

5000

C.S.Ramalingam Fourier Methods of Spectral Estimation

Application to Speech Analysis

0 0.25 0.5 0.75 1 1.25 1.5 1.75 2−1

−0.5

0

0.5

1Should We Chase Those Cowboys?

Am

plitu

de

Time

Fre

quen

cy

0 0.25 0.5 0.75 1 1.25 1.5 1.75 20

1000

2000

3000

4000

5000

C.S.Ramalingam Fourier Methods of Spectral Estimation

Application to Speech Analysis

0 0.25 0.5 0.75 1 1.25 1.5 1.75 2−1

−0.5

0

0.5

1Should We Chase Those Cowboys?

Am

plitu

de

Time

Fre

quen

cy

0 0.25 0.5 0.75 1 1.25 1.5 1.75 20

1000

2000

3000

4000

5000

C.S.Ramalingam Fourier Methods of Spectral Estimation

Application to Speech Analysis

0 0.25 0.5 0.75 1 1.25 1.5 1.75 2−1

−0.5

0

0.5

1A

mpl

itude

Should We Chase Those Cowboys?

Time

Fre

quen

cy

0 0.25 0.5 0.75 1 1.25 1.5 1.750

1000

2000

3000

4000

5000

C.S.Ramalingam Fourier Methods of Spectral Estimation

Summary

Definition of Power Spectrum

Deterministic signal example

Power Spectrum of a Random Process

The Periodogram Estimator

The Averaged Periodogram

Bias versus Variance

Blackman-Tukey Method

Use of Data Windowing in Spectral Analysis

Spectrogram: Speech Signal Example

C.S.Ramalingam Fourier Methods of Spectral Estimation