1

Stochastic processes

Lecture 7Linear time invariant systems

2

Random process

3

1st order Distribution & density function

First-order distribution

First-order density function

4

2end order Distribution & density function

2end order distribution

2end order density function

5

EXPECTATIONS

• Expected value

• The autocorrelation

6

Some random processes

• Single pulse• Multiple pulses• Periodic Random Processes• The Gaussian Process• The Poisson Process• Bernoulli and Binomial Processes• The Random Walk Wiener Processes• The Markov Process

7

Recap: Power spectrum density

f

Sxx

(f)

8

Power spectrum density

• Since the integral of the squared absolute Fourier transform contains the full power of the signal it is a density function.

• So the power spectral density of a random process is:

• Due to absolute factor the PSD is always real

𝑆𝑥𝑥 ( 𝑓 )= 𝑙 𝑖𝑚𝑇→∞

𝐸 [|∫−𝑇𝑇 𝑠 (𝑡 )𝑒− 𝑗 2𝜋 𝑓𝑡𝑑𝑡|22𝑇 ]

9

Power spectrum density

• The PSD is a density function.– In the case of the random process the PSD is the density

function of the random process and not necessarily the frequency spectrum of a single realization.

• Example– A random process is defined as

– Where ωr is a unifom distributed random variable wiht a range from 0-π

– What is the PSD for the process and – The power sepctrum for a single realization

X (𝑡 )=sin (𝜔𝑟 𝑡)

10

Properties of the PSD

1. Sxx(f) is real and nonnegative

2. The average power in X(t) is given by:

3. If X(t) is real Rxx(τ) and Sxx(f) are also even

4. If X(t) has periodic components Sxx(f)has impulses

5. Independent on phase

11

Wiener-Khinchin 1

• If the X(t) is stationary in the wide-sense the PSD is the Fourier transform of the Autocorrelation

12

Wiener-Khinchin Two method for estimation of the PSD

X(t)

Fourier Transform

|X(f)|2

Sxx(f)

Autocorrelation

Fourier Transformt

X(t

)

f

X(f

)

Rxx

()

f

Sxx

(f)

13

The inverse Fourier Transform of the PSD

• Since the PSD is the Fourier transformed autocorrelation

• The inverse Fourier transform of the PSD is the autocorrelation

14

Cross spectral densities

• If X(t) and Y(t) are two jointly wide-sense stationary processes, is the Cross spectral densities

• Or

15

Properties of Cross spectral densities

1. Since is

2. Syx(f) is not necessary real

3. If X(t) and Y(t) are orthogonal Sxy(f)=0

4. If X(t) and Y(t) are independent Sxy(f)=E[X(t)] E[Y(t)] δ(f)

16

Cross spectral densities example

• 1 Hz Sinus curves in white noise

Where w(t) is Gaussian noise

0 5 10 15 20-10

0

10

X(t

)

t (s)

Signal X(t)

0 5 10 15 20-10

0

10

Y(t

)

t (s)

Signal Y(t)

𝑋 (𝑡 )=sin (2𝜋 𝑡 )+3𝑤 (𝑡)𝑌 (𝑡 )=sin(2𝜋𝑡+𝜋2 )+3𝑤(𝑡)

0 5 10 15 20 25-30

-25

-20

-15

-10

-5

0

5

Frequency (Hz)

Pow

er/f

requ

ency

(dB

/Hz)

Welch Cross Power Spectral Density Estimate

17

The periodogramThe estimate of the PSD

• The PSD can be estimate from the autocorrelation

• Or directly from the signal

𝑆 𝑥𝑥 [ω ]= ∑𝑚=−𝑁+1

𝑁− 1

𝑅𝑥𝑥 [𝑚]𝑒− 𝑗 ω𝑚  

𝑆 𝑥𝑥 [ω ]= 1𝑁 |∑

𝑛=0

𝑁− 1

𝑥 [𝑛]𝑒− 𝑗ω𝑛  |2

18

Bias in the estimates of the autocorrelation

N=12𝑅𝑥𝑥 [𝑚 ]= ∑

𝑛=0

𝑁−|𝑚|− 1

𝑥 [𝑛 ] 𝑥 [𝑛+𝑚]

-10 -5 0 5 10 15 20-2

-1

0

1

2

n

-10 -5 0 5 10 15 20-2

-1

0

1

2

n+m-15 -10 -5 0 5 10 15

-6

-4

-2

0

2

4

6

8Autocorrelation

M=-10

-10 -5 0 5 10 15 20-2

-1

0

1

2

n

-10 -5 0 5 10 15 20-2

-1

0

1

2

n+m-15 -10 -5 0 5 10 15

-6

-4

-2

0

2

4

6

8Autocorrelation

M=0

-10 -5 0 5 10 15 20-2

-1

0

1

2

n

-10 -5 0 5 10 15 20-2

-1

0

1

2

n+m-15 -10 -5 0 5 10 15

-6

-4

-2

0

2

4

6

8Autocorrelation

M=4

19

Variance in the PSD

• The variance of the periodogram is estimated to the power of two of PSD

𝑉𝑎𝑟 (𝑆𝑥𝑥 [𝜔 ] )=𝑆 𝑥𝑥(𝜔)  2

0 5 10-5

0

5Realization 1

t (s)0 50 100 150 200

0

5

10PSD: Realization 1

f (Hz)

0 5 10-5

0

5

t (s)

Realization 2

0 50 100 150 2000

5

10

f (Hz)

PSD: Realization 2

0 5 10-5

0

5

t (s)

Realization 3

0 50 100 150 2000

5

10

f (Hz)

PSD: Realization 3 0 50 100 150 200

0

0.2

0.4

0.6

0.8

1

f (Hz)

Sxx

(f)

True PSD

20

Averaging

• Divide the signal into K segments of M length

• Calculate the periodogram of each segment

• Calculate the average periodogram

21

Illustrations of Averaging

0 1 2 3 4 5 6 7 8 9 10-4

-2

0

2X

(t)

0 100 2000

5

10

0 100 2000

2

4

0 100 2000

5

10

0 100 2000

2

4

6

0 50 100 150 2000

5

10

f (Hz)

22

PSD units

• Typical units:• Electrical measurements: V2/Hz or dB V/Hz• Sound: Pa2/Hz or dB/Hz

• How to calculate dB I a power spectrum:PSDdB(f) = 10 log10 { PSD(f) }

.

-100 -50 0 50 1000

1

2

3

4

5

6x 10

8

f (Hz)

Sxx

(f)

V2 /

Hz

-100 -50 0 50 10040

50

60

70

80

90

f (Hz)

Sxx

(f)

dB V

/H

z

23

Agenda (Lec. 7)

• Recap: Linear time invariant systems• Stochastic signals and LTI systems

– Mean Value function– Mean square value – Cross correlation function between input and output– Autocorrelation function and spectrum output

• Filter examples • Intro to system identification

24

Focus continuous signals and system

Continuous signal:

Discrete signal:

0 20 40 60 80 100-1

0

1

t (s)

x(t)

0 2 4 6 8 10-1

-0.5

0

0.5

1

n

x[n]

25

Systems

26

Recap: Linear time invariant systems (LTI)

• What is a Linear system:– The system applies to superposition

)()()()( 2121 txTbtxTatxbtxaT

0 1 2 3 4 50

2

4

6

8

10

12

14

16

18

20Linear system

x(t)

y(t)

0 1 2 3 4 5-20

-15

-10

-5

0

5

10

15

20

25Nonlinear systems

x(t)

y(t)

x[n]2

Öx[n]

20 log(x[n])

27

Recap: Linear time invariant systems (LTI)

• Time invariant:• A time invariant systems is independent on explicit time

– (The coefficient are independent on time)

• That means If: y2(t)=f[x1(t)]

Then: y2(t+t0)=f[x1(t+t0)]

The same to Day tomorrow and in 1000 years

70 years45 years20 yearsA non Time invariant

28

Examples

• A linear systemy(t)=3 x(t)

• A nonlinear systemy(t)=3 x(t)2

• A time invariant system y(t)=3 x(t)

• A time variant system y(t)=3t x(t)

The impulse response

T{∙}

)]([][ tfnh )(th)(t

The output of a system if Dirac delta is input

-10 -5 0 5 10 15 20

0

t

y(t)

Impuls response

-10 -5 0 5 10 15 200

inf

t

x(t)

Impuls

30

Convolution

• The output of LTI system can be determined by the convoluting the input with the impulse response

31

Fourier transform of the impulse response

• The Transfer function (System function) is the Fourier transformed impulse response

• The impulse response can be determined from the Transfer function with the invers Fourier transform

32

Fourier transform of LTI systems

• Convolution corresponds to multiplication in the frequency domain

-10 -5 0 5 10 15 20

0

t

y(t)

Impuls response

-2 -1 0 1 20

0.5

1

1.5

f (Hz)

|H(f

)|

-10 -5 0 5 10 15 20-3

-2

-1

0

1

2

3

t

x(t)

Input

-2 -1 0 1 20

500

1000

1500

2000

2500

3000

f (Hz)

|X(f

)|

-2 -1 0 1 20

200

400

600

800

1000

1200

f (Hz)

|Y(f

)|

-10 -5 0 5 10 15 20-3

-2

-1

0

1

2

3

t

y(t)

Output

Time domain

Frequency domain

* =

x =

33

Causal systems

• Independent on the future signal

-10 -5 0 5 10 15 20

0

t

y(t)

Impuls response

34

Stochastic signals and LTI systems

• Estimation of the output from a LTI system when the input is a stochastic process

Α is a delay factor like τ

35

Statistical estimates of output

• The specific distribution function fX(x,t) is difficult to estimate. Therefor we stick to– Mean – Autocorrelation – PSD – Mean square value.

36

Expected Value of Y(t) (1/2)

• How do we estimate the mean of the output?

𝐸 [𝑌 (𝑡 ) ]=𝐸[∫−∞

∞

𝑋 (𝑡−𝛼 )h (𝛼 )𝑑𝛼 ]𝐸 [𝑌 (𝑡 ) ]=∫

−∞

∞

𝐸 [ 𝑋 (𝑡−𝛼 ) ] h (𝛼 ) 𝑑𝛼

𝐸 [𝑌 (𝑡 ) ]=∫−∞

∞

𝑚𝑥 (𝑡−𝛼)h (𝛼 )𝑑𝛼

If mean of x(t) is defined as mx(t)

𝑌 (𝑡)=∫−∞

∞

𝑋 (𝑡−𝛼 )h (𝛼 )𝑑𝛼

37

Expected Value of Y(t) (2/2)

If x(t) is wide sense stationary

𝑚𝑥 (𝑡−𝛼 )=𝑚𝑥 (𝑡 )=𝑚𝑥(𝑚𝑥𝑖𝑠𝑎𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡)

Alternative estimate:At 0 Hz the transfer function is equal to the DC gain

∫−∞

∞

h (𝛼 )𝑑𝛼=𝐻 (0)

Therefor: 𝑚𝑦=𝐸 [𝑌 (𝑡 ) ]=𝑚𝑥𝐻 (0)

38

Expected Mean square value (1/2)

𝐸 [𝑌 (𝑡 )2 ]=𝐸 [𝑌 (𝑡 )𝑌 (𝑡 ) ] 𝑌 (𝑡)=∫−∞

∞

𝑋 (𝑡−𝛼 )h (𝛼 )𝑑𝛼

𝐸 [𝑌 (𝑡 )2 ]=𝐸[(∫−∞∞

𝑋 (𝑡−𝛼1 )h (𝛼1 ) 𝑑𝛼1)(∫−∞

∞

𝑋 (𝑡−𝛼2 )h (𝛼2 )𝑑𝛼2) ]𝐸 [𝑌 (𝑡 )2 ]=𝐸[∫

−∞

∞

∫−∞

∞

𝑋 (𝑡−𝛼1 )𝑋 (𝑡−𝛼2 )h (𝛼1 )h (𝛼2 )𝑑𝛼1𝑑𝛼2 ]𝐸 [𝑌 (𝑡 )2 ]=∫

−∞

∞

∫−∞

∞

𝐸 [𝑋 (𝑡−𝛼1 ) 𝑋 (𝑡−𝛼2 ) ] h (𝛼1 )h (𝛼2 )𝑑𝛼1𝑑𝛼2

𝐸 [𝑌 (𝑡 )2 ]=∫−∞

∞

∫−∞

∞

𝑅𝑥𝑥 (𝑡−𝛼1, 𝑡−𝛼2)h (𝛼1 )h (𝛼2 )𝑑𝛼1𝑑𝛼2

𝐸 [𝑌 (𝑡 )2 ]=∫−∞

∞

∫−∞

∞

𝑅𝑥𝑥 (𝛼1 ,𝛼2)h (𝑡−𝛼1 )h (𝑡−𝛼2 )𝑑𝛼1𝑑𝛼2

39

Expected Mean square value (2/2)

𝐸 [𝑌 (𝑡 )2 ]=∫−∞

∞

∫−∞

∞

𝑅𝑥𝑥 (𝛼1 ,𝛼2)h (𝑡−𝛼1 )h (𝑡−𝛼2 )𝑑𝛼1𝑑𝛼2

𝐸 [𝑌 (𝑡 )2 ]=∫−∞

∞

∫−∞

∞

𝑅𝑥𝑥 (𝛼− 𝛽)h (𝛼 )h (𝛽 )𝑑𝛼1𝑑𝛼2

By substitution:

𝐸 [𝑌 (𝑡 )2 ]=∫−∞

∞

∫−∞

∞

𝑅𝑥𝑥 (𝑡−𝛼 ,𝑡− 𝛽)h (𝛼 )h ( 𝛽)𝑑𝛼1𝑑𝛼2

If X(t)is WSS

Thereby the Expected Mean square value is independent on time

40

Cross correlation function between input and output

• Can we estimate the Cross correlation between input and out if X(t) is wide sense stationary

𝑅𝑦𝑥 (𝑡+𝜏 , 𝑡 )=𝐸 [𝑌 (𝑡+𝜏 )𝑋∗(𝑡)]

𝑅𝑦𝑥 (𝑡+𝜏 , 𝑡 )=𝐸[(∫−∞∞

𝑋 (𝑡−𝛼+𝜏 )h (𝛼 ) 𝑑𝛼) 𝑋∗(𝑡)]𝑅𝑦𝑥 (𝑡+𝜏 , 𝑡 )=𝐸[∫

−∞

∞

𝑋 (𝑡−𝛼+𝜏 ) 𝑋∗(𝑡)h (𝛼 )𝑑𝛼 ]

𝑅𝑦 𝑥 (𝜏 )=∫−∞

∞

𝑅𝑥𝑥 (𝜏−𝛼 )h (𝛼 ) 𝑑𝛼=𝑅𝑥𝑥 (𝜏 )∗h (𝜏)  

-10 -5 0 5 10 15 20-3

-2

-1

0

1

2

3

t

x(t)

Input

-10 -5 0 5 10 15 20-3

-2

-1

0

1

2

3

t

y(t)

Output

𝑅𝑥𝑥 (𝜏 )=𝐸 [ 𝑋 (𝑡+𝜏 )𝑋 (𝑡)]

-30 -20 -10 0 10 20 30-1500

-1000

-500

0

500

1000

1500

(s)

Rxy

()

Cross-correlation between y(t) and x(t)

Thereby the cross-correlation is the convolution between the auto-correlation of x(t) and the impulse response

41

Autocorrelation of the output (1/2)

𝑅𝑦𝑦 (𝜏 )=𝑅𝑦𝑦 (𝑡+𝜏 , 𝑡 )=𝐸 [𝑌 (𝑡+𝜏 )𝑌 (𝑡) ]

𝑅𝑦𝑦 (𝜏 )=∫−∞

∞

∫−∞

∞

𝐸 [ 𝑋 (𝑡+𝜏−𝛼 ) 𝑋 (𝑡− 𝛽 )]h (𝛼 )h (𝛽 )𝑑𝛼 𝑑𝛽

𝑌 (𝑡+𝜏)=∫−∞

∞

𝑋 (𝑡+𝜏−𝛼 )h (𝛼 )𝑑𝛼

𝑌 (𝑡)=∫−∞

∞

𝑋 (𝑡− 𝛽)h (𝛽 )𝑑 𝛽

Y(t) and Y(t+τ) is :

𝑅𝑦𝑦 (𝜏 )=∫−∞

∞

∫−∞

∞

𝑅𝑥𝑥(𝜏−𝛼+𝛽)h (𝛼 )h (𝛽 )𝑑𝛼 𝑑𝛽

42

Autocorrelation of the output (2/2)

𝑅𝑦𝑦 (𝜏 )=∫−∞

∞

∫−∞

∞

𝐸 [ 𝑋 (𝑡+𝜏−𝛼 ) 𝑋 (𝑡− 𝛽 )]h (𝛼 )h (𝛽 )𝑑𝛼 𝑑𝛽

By substitution: α=-β

𝑅𝑦𝑦 (𝜏 )=∫−∞

∞

∫−∞

∞

𝐸 [ 𝑋 (𝑡+𝜏−𝛼 ) 𝑋 (𝑡+𝛼 )] h (𝛼 )h (−𝑎 )𝑑𝛼 𝑑𝛼

Remember:

-30 -20 -10 0 10 20 30-1000

-500

0

500

1000

(s)

Rxy

()

Autocorrelation of y(t)

𝑅𝑦𝑦 (𝜏 )=𝑅𝑦 𝑥 (𝜏 )∗h(−𝜏)

𝑅𝑦𝑦 (𝜏 )=𝑅𝑥 𝑥 (𝜏 )∗h (𝜏)∗h(−𝜏 )

43-2 -1 0 1 2

0

2

4

6

8

10

12x 10

5

f (Hz)

Syy

(f)

Spectrum of output

• Given:

• The power spectrum is

𝑅𝑦 𝑦 (𝜏 )=𝑅𝑥𝑥 (𝜏 )∗h (𝜏 )∗h(−𝜏 )

-2 -1 0 1 20

0.5

1

1.5

f (Hz)

|H(f

)|2

-2 -1 0 1 20

2

4

6

8

10x 10

6

f (Hz)

Sxx

(f)

x =

¿𝐻 ( 𝑓 )∨¿2=𝐻 ( 𝑓 )𝐻∗( 𝑓 )¿

44

Filter examples

45

Typical LIT filters

• FIR filters (Finite impulse response)• IIR filters (Infinite impulse response)

– Butterworth– Chebyshev– Elliptic

Ideal filters

• Highpass filter

• Band stop filter

• Bandpassfilter

47

Filter types and rippels

Analog lowpass Butterworth filter

• Is ”all pole” filter– Squared frequency transfer function

• N:filter order• fc: 3dB cut off frequency

• Estimate PSD from filter

NcfffH 2

2

/1

1)(

Nc

xxyyff

ffS 2/1

1)(S)(

Chebyshev filter type I

• Transfer function

• Where ε is relateret to ripples in the pass band

• Where TN is a N order polynomium

pN ffTfH

/1

122

2

1

1

)coshcosh(

)coscos()(

1

1

x

x

xN

xNxTN

Transformation of a low pass filter to other types (the s-domain)

Filter type Transformation New Cutoff frequency

Lowpas>Lowpas

Lowpas>Highpas

Lowpas>Highpas

Lowpas>Stopband

ssp

p

'

p'

ss pp '

p'

)(

2

lu

ulp s

ss

ul

lup s

ss

2

)(

ul ,

ul ,

Lowest Cutoff frequency

Highest Cutoff frequency:

:

u

l

p

New Cutoff frequencyp'

Old Cutoff frequency

51

Discrete time implantation of filters

• A discrete filter its Transfer function in the z-domain or Fourier domain

– Where bk and ak is the filter coefficients

• In the time domain:

Mm

Mm

zazazaa

zbzbzbbzH

zX

zY

.......´

.......´)(

)(

)(2

21

10

22

110

][......]2[]1[

][......]2[]1[][][

21

210

Mnyanyanya

Mnxanxbnxbnxbny

M

M

52

Filtering of a Gaussian process

• Gaussian process– X(t1),X(t2),X(t3),….X(tn) are jointly Gaussian for all t

and n values• Filtering of a Gaussian process

– Where w[n] are independent zero mean Gaussian random variables.

][......]2[]1[

][......]2[]1[][][

21

210

Mnyanyanya

Mnwanwbnwbnwbny

M

M

The Gaussian Process

• X(t1),X(t2),X(t3),….X(tn) are jointly Gaussian for all t and n values

• Example: randn() in Matlab

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000-4

-3

-2

-1

0

1

2

3

4

5Gaussian process

-4 -3 -2 -1 0 1 2 3 4 50

100

200

300

400

500

600

700Histogram of Gaussian process

The Gaussian Process and a linear time invariant systems

• Out put = convolution between input and impulse response

Gaussian input Gaussian output

Example

• x(t):

• h(t): Low pass filter• y(t):

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000-4

-3

-2

-1

0

1

2

3

4

5Gaussian process

-4 -3 -2 -1 0 1 2 3 4 50

100

200

300

400

500

600

700Histogram of Gaussian process

-1.5 -1 -0.5 0 0.5 1 1.50

100

200

300

400

500

600Histogram of y(t)

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000-1.5

-1

-0.5

0

0.5

1

1.5

56

Filtering of a Gaussian process example 2

0 100 200 300 400 500-1000

-500

0

Frequency (Hz)

Pha

se (

degr

ees)

0 100 200 300 400 500-100

-50

0

Frequency (Hz)

Mag

nitu

de (

dB) Transfere function of filter

0 100 200 300 400 500 600 700 800 900 1000-4

-2

0

2

4

t (ms)

x(t)

White noise

Band pass filter

0 100 200 300 400 500 600 700 800 900 1000-1

-0.5

0

0.5

1

t (ms)

y(t)

Output

57

Intro to system identification

• Modeling of signals using linear Gaussian models:

• Example: AR models

• The output is modeled by a linear combination of previous samples plus Gaussian noise.

][][......]2[]1[][ 21 nwMnyanyanyany M

58

Modeling example

• Estimated 3th order model

][]3[0.7299-]2[2.3903]1[-2.6397][ nwnynynyny

0 100 200 300 400 500 600 700 800 900 1000-1

-0.5

0

0.5

1

t (ms)

y(t)

Output

451 451.5 452 452.5 453 453.5 4540.25

0.3

0.35

0.4

t (ms)y(

t)

Output

signal

points used for predictionPrediction

True point

453.98 453.99 454 454.01 454.02

0.282

0.284

0.286

0.288

0.29

0.292

0.294

t (ms)

y(t)

Output

w[n]

59

Agenda (Lec. 7)

• Recap: Linear time invariant systems• Stochastic signals and LTI systems

– Mean Value function– Mean square value – Cross correlation function between input and output– Autocorrelation function and spectrum output

• Filter examples • Intro to system identification