1
Stochastic processes
Lecture 7Linear time invariant systems
2
Random process
3
1st order Distribution & density function
First-order distribution
First-order density function
4
2end order Distribution & density function
2end order distribution
2end order density function
5
EXPECTATIONS
• Expected value
• The autocorrelation
6
Some random processes
• Single pulse• Multiple pulses• Periodic Random Processes• The Gaussian Process• The Poisson Process• Bernoulli and Binomial Processes• The Random Walk Wiener Processes• The Markov Process
7
Recap: Power spectrum density
f
Sxx
(f)
8
Power spectrum density
• Since the integral of the squared absolute Fourier transform contains the full power of the signal it is a density function.
• So the power spectral density of a random process is:
• Due to absolute factor the PSD is always real
𝑆𝑥𝑥 ( 𝑓 )= 𝑙 𝑖𝑚𝑇→∞
𝐸 [|∫−𝑇𝑇 𝑠 (𝑡 )𝑒− 𝑗 2𝜋 𝑓𝑡𝑑𝑡|22𝑇 ]
9
Power spectrum density
• The PSD is a density function.– In the case of the random process the PSD is the density
function of the random process and not necessarily the frequency spectrum of a single realization.
• Example– A random process is defined as
– Where ωr is a unifom distributed random variable wiht a range from 0-π
– What is the PSD for the process and – The power sepctrum for a single realization
X (𝑡 )=sin (𝜔𝑟 𝑡)
10
Properties of the PSD
1. Sxx(f) is real and nonnegative
2. The average power in X(t) is given by:
3. If X(t) is real Rxx(τ) and Sxx(f) are also even
4. If X(t) has periodic components Sxx(f)has impulses
5. Independent on phase
11
Wiener-Khinchin 1
• If the X(t) is stationary in the wide-sense the PSD is the Fourier transform of the Autocorrelation
12
Wiener-Khinchin Two method for estimation of the PSD
X(t)
Fourier Transform
|X(f)|2
Sxx(f)
Autocorrelation
Fourier Transformt
X(t
)
f
X(f
)
Rxx
()
f
Sxx
(f)
13
The inverse Fourier Transform of the PSD
• Since the PSD is the Fourier transformed autocorrelation
• The inverse Fourier transform of the PSD is the autocorrelation
14
Cross spectral densities
• If X(t) and Y(t) are two jointly wide-sense stationary processes, is the Cross spectral densities
• Or
15
Properties of Cross spectral densities
1. Since is
2. Syx(f) is not necessary real
3. If X(t) and Y(t) are orthogonal Sxy(f)=0
4. If X(t) and Y(t) are independent Sxy(f)=E[X(t)] E[Y(t)] δ(f)
16
Cross spectral densities example
• 1 Hz Sinus curves in white noise
Where w(t) is Gaussian noise
0 5 10 15 20-10
0
10
X(t
)
t (s)
Signal X(t)
0 5 10 15 20-10
0
10
Y(t
)
t (s)
Signal Y(t)
𝑋 (𝑡 )=sin (2𝜋 𝑡 )+3𝑤 (𝑡)𝑌 (𝑡 )=sin(2𝜋𝑡+𝜋2 )+3𝑤(𝑡)
0 5 10 15 20 25-30
-25
-20
-15
-10
-5
0
5
Frequency (Hz)
Pow
er/f
requ
ency
(dB
/Hz)
Welch Cross Power Spectral Density Estimate
17
The periodogramThe estimate of the PSD
• The PSD can be estimate from the autocorrelation
• Or directly from the signal
𝑆 𝑥𝑥 [ω ]= ∑𝑚=−𝑁+1
𝑁− 1
𝑅𝑥𝑥 [𝑚]𝑒− 𝑗 ω𝑚
𝑆 𝑥𝑥 [ω ]= 1𝑁 |∑
𝑛=0
𝑁− 1
𝑥 [𝑛]𝑒− 𝑗ω𝑛 |2
18
Bias in the estimates of the autocorrelation
N=12𝑅𝑥𝑥 [𝑚 ]= ∑
𝑛=0
𝑁−|𝑚|− 1
𝑥 [𝑛 ] 𝑥 [𝑛+𝑚]
-10 -5 0 5 10 15 20-2
-1
0
1
2
n
-10 -5 0 5 10 15 20-2
-1
0
1
2
n+m-15 -10 -5 0 5 10 15
-6
-4
-2
0
2
4
6
8Autocorrelation
M=-10
-10 -5 0 5 10 15 20-2
-1
0
1
2
n
-10 -5 0 5 10 15 20-2
-1
0
1
2
n+m-15 -10 -5 0 5 10 15
-6
-4
-2
0
2
4
6
8Autocorrelation
M=0
-10 -5 0 5 10 15 20-2
-1
0
1
2
n
-10 -5 0 5 10 15 20-2
-1
0
1
2
n+m-15 -10 -5 0 5 10 15
-6
-4
-2
0
2
4
6
8Autocorrelation
M=4
19
Variance in the PSD
• The variance of the periodogram is estimated to the power of two of PSD
𝑉𝑎𝑟 (𝑆𝑥𝑥 [𝜔 ] )=𝑆 𝑥𝑥(𝜔) 2
0 5 10-5
0
5Realization 1
t (s)0 50 100 150 200
0
5
10PSD: Realization 1
f (Hz)
0 5 10-5
0
5
t (s)
Realization 2
0 50 100 150 2000
5
10
f (Hz)
PSD: Realization 2
0 5 10-5
0
5
t (s)
Realization 3
0 50 100 150 2000
5
10
f (Hz)
PSD: Realization 3 0 50 100 150 200
0
0.2
0.4
0.6
0.8
1
f (Hz)
Sxx
(f)
True PSD
20
Averaging
• Divide the signal into K segments of M length
• Calculate the periodogram of each segment
• Calculate the average periodogram
21
Illustrations of Averaging
0 1 2 3 4 5 6 7 8 9 10-4
-2
0
2X
(t)
0 100 2000
5
10
0 100 2000
2
4
0 100 2000
5
10
0 100 2000
2
4
6
0 50 100 150 2000
5
10
f (Hz)
22
PSD units
• Typical units:• Electrical measurements: V2/Hz or dB V/Hz• Sound: Pa2/Hz or dB/Hz
• How to calculate dB I a power spectrum:PSDdB(f) = 10 log10 { PSD(f) }
.
-100 -50 0 50 1000
1
2
3
4
5
6x 10
8
f (Hz)
Sxx
(f)
V2 /
Hz
-100 -50 0 50 10040
50
60
70
80
90
f (Hz)
Sxx
(f)
dB V
/H
z
23
Agenda (Lec. 7)
• Recap: Linear time invariant systems• Stochastic signals and LTI systems
– Mean Value function– Mean square value – Cross correlation function between input and output– Autocorrelation function and spectrum output
• Filter examples • Intro to system identification
24
Focus continuous signals and system
Continuous signal:
Discrete signal:
0 20 40 60 80 100-1
0
1
t (s)
x(t)
0 2 4 6 8 10-1
-0.5
0
0.5
1
n
x[n]
25
Systems
26
Recap: Linear time invariant systems (LTI)
• What is a Linear system:– The system applies to superposition
)()()()( 2121 txTbtxTatxbtxaT
0 1 2 3 4 50
2
4
6
8
10
12
14
16
18
20Linear system
x(t)
y(t)
0 1 2 3 4 5-20
-15
-10
-5
0
5
10
15
20
25Nonlinear systems
x(t)
y(t)
x[n]2
Öx[n]
20 log(x[n])
27
Recap: Linear time invariant systems (LTI)
• Time invariant:• A time invariant systems is independent on explicit time
– (The coefficient are independent on time)
• That means If: y2(t)=f[x1(t)]
Then: y2(t+t0)=f[x1(t+t0)]
The same to Day tomorrow and in 1000 years
70 years45 years20 yearsA non Time invariant
28
Examples
• A linear systemy(t)=3 x(t)
• A nonlinear systemy(t)=3 x(t)2
• A time invariant system y(t)=3 x(t)
• A time variant system y(t)=3t x(t)
The impulse response
T{∙}
)]([][ tfnh )(th)(t
The output of a system if Dirac delta is input
-10 -5 0 5 10 15 20
0
t
y(t)
Impuls response
-10 -5 0 5 10 15 200
inf
t
x(t)
Impuls
30
Convolution
• The output of LTI system can be determined by the convoluting the input with the impulse response
31
Fourier transform of the impulse response
• The Transfer function (System function) is the Fourier transformed impulse response
• The impulse response can be determined from the Transfer function with the invers Fourier transform
32
Fourier transform of LTI systems
• Convolution corresponds to multiplication in the frequency domain
-10 -5 0 5 10 15 20
0
t
y(t)
Impuls response
-2 -1 0 1 20
0.5
1
1.5
f (Hz)
|H(f
)|
-10 -5 0 5 10 15 20-3
-2
-1
0
1
2
3
t
x(t)
Input
-2 -1 0 1 20
500
1000
1500
2000
2500
3000
f (Hz)
|X(f
)|
-2 -1 0 1 20
200
400
600
800
1000
1200
f (Hz)
|Y(f
)|
-10 -5 0 5 10 15 20-3
-2
-1
0
1
2
3
t
y(t)
Output
Time domain
Frequency domain
* =
x =
33
Causal systems
• Independent on the future signal
-10 -5 0 5 10 15 20
0
t
y(t)
Impuls response
34
Stochastic signals and LTI systems
• Estimation of the output from a LTI system when the input is a stochastic process
Α is a delay factor like τ
35
Statistical estimates of output
• The specific distribution function fX(x,t) is difficult to estimate. Therefor we stick to– Mean – Autocorrelation – PSD – Mean square value.
36
Expected Value of Y(t) (1/2)
• How do we estimate the mean of the output?
𝐸 [𝑌 (𝑡 ) ]=𝐸[∫−∞
∞
𝑋 (𝑡−𝛼 )h (𝛼 )𝑑𝛼 ]𝐸 [𝑌 (𝑡 ) ]=∫
−∞
∞
𝐸 [ 𝑋 (𝑡−𝛼 ) ] h (𝛼 ) 𝑑𝛼
𝐸 [𝑌 (𝑡 ) ]=∫−∞
∞
𝑚𝑥 (𝑡−𝛼)h (𝛼 )𝑑𝛼
If mean of x(t) is defined as mx(t)
𝑌 (𝑡)=∫−∞
∞
𝑋 (𝑡−𝛼 )h (𝛼 )𝑑𝛼
37
Expected Value of Y(t) (2/2)
If x(t) is wide sense stationary
𝑚𝑥 (𝑡−𝛼 )=𝑚𝑥 (𝑡 )=𝑚𝑥(𝑚𝑥𝑖𝑠𝑎𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡)
Alternative estimate:At 0 Hz the transfer function is equal to the DC gain
∫−∞
∞
h (𝛼 )𝑑𝛼=𝐻 (0)
Therefor: 𝑚𝑦=𝐸 [𝑌 (𝑡 ) ]=𝑚𝑥𝐻 (0)
38
Expected Mean square value (1/2)
𝐸 [𝑌 (𝑡 )2 ]=𝐸 [𝑌 (𝑡 )𝑌 (𝑡 ) ] 𝑌 (𝑡)=∫−∞
∞
𝑋 (𝑡−𝛼 )h (𝛼 )𝑑𝛼
𝐸 [𝑌 (𝑡 )2 ]=𝐸[(∫−∞∞
𝑋 (𝑡−𝛼1 )h (𝛼1 ) 𝑑𝛼1)(∫−∞
∞
𝑋 (𝑡−𝛼2 )h (𝛼2 )𝑑𝛼2) ]𝐸 [𝑌 (𝑡 )2 ]=𝐸[∫
−∞
∞
∫−∞
∞
𝑋 (𝑡−𝛼1 )𝑋 (𝑡−𝛼2 )h (𝛼1 )h (𝛼2 )𝑑𝛼1𝑑𝛼2 ]𝐸 [𝑌 (𝑡 )2 ]=∫
−∞
∞
∫−∞
∞
𝐸 [𝑋 (𝑡−𝛼1 ) 𝑋 (𝑡−𝛼2 ) ] h (𝛼1 )h (𝛼2 )𝑑𝛼1𝑑𝛼2
𝐸 [𝑌 (𝑡 )2 ]=∫−∞
∞
∫−∞
∞
𝑅𝑥𝑥 (𝑡−𝛼1, 𝑡−𝛼2)h (𝛼1 )h (𝛼2 )𝑑𝛼1𝑑𝛼2
𝐸 [𝑌 (𝑡 )2 ]=∫−∞
∞
∫−∞
∞
𝑅𝑥𝑥 (𝛼1 ,𝛼2)h (𝑡−𝛼1 )h (𝑡−𝛼2 )𝑑𝛼1𝑑𝛼2
39
Expected Mean square value (2/2)
𝐸 [𝑌 (𝑡 )2 ]=∫−∞
∞
∫−∞
∞
𝑅𝑥𝑥 (𝛼1 ,𝛼2)h (𝑡−𝛼1 )h (𝑡−𝛼2 )𝑑𝛼1𝑑𝛼2
𝐸 [𝑌 (𝑡 )2 ]=∫−∞
∞
∫−∞
∞
𝑅𝑥𝑥 (𝛼− 𝛽)h (𝛼 )h (𝛽 )𝑑𝛼1𝑑𝛼2
By substitution:
𝐸 [𝑌 (𝑡 )2 ]=∫−∞
∞
∫−∞
∞
𝑅𝑥𝑥 (𝑡−𝛼 ,𝑡− 𝛽)h (𝛼 )h ( 𝛽)𝑑𝛼1𝑑𝛼2
If X(t)is WSS
Thereby the Expected Mean square value is independent on time
40
Cross correlation function between input and output
• Can we estimate the Cross correlation between input and out if X(t) is wide sense stationary
𝑅𝑦𝑥 (𝑡+𝜏 , 𝑡 )=𝐸 [𝑌 (𝑡+𝜏 )𝑋∗(𝑡)]
𝑅𝑦𝑥 (𝑡+𝜏 , 𝑡 )=𝐸[(∫−∞∞
𝑋 (𝑡−𝛼+𝜏 )h (𝛼 ) 𝑑𝛼) 𝑋∗(𝑡)]𝑅𝑦𝑥 (𝑡+𝜏 , 𝑡 )=𝐸[∫
−∞
∞
𝑋 (𝑡−𝛼+𝜏 ) 𝑋∗(𝑡)h (𝛼 )𝑑𝛼 ]
𝑅𝑦 𝑥 (𝜏 )=∫−∞
∞
𝑅𝑥𝑥 (𝜏−𝛼 )h (𝛼 ) 𝑑𝛼=𝑅𝑥𝑥 (𝜏 )∗h (𝜏)
-10 -5 0 5 10 15 20-3
-2
-1
0
1
2
3
t
x(t)
Input
-10 -5 0 5 10 15 20-3
-2
-1
0
1
2
3
t
y(t)
Output
𝑅𝑥𝑥 (𝜏 )=𝐸 [ 𝑋 (𝑡+𝜏 )𝑋 (𝑡)]
-30 -20 -10 0 10 20 30-1500
-1000
-500
0
500
1000
1500
(s)
Rxy
()
Cross-correlation between y(t) and x(t)
Thereby the cross-correlation is the convolution between the auto-correlation of x(t) and the impulse response
41
Autocorrelation of the output (1/2)
𝑅𝑦𝑦 (𝜏 )=𝑅𝑦𝑦 (𝑡+𝜏 , 𝑡 )=𝐸 [𝑌 (𝑡+𝜏 )𝑌 (𝑡) ]
𝑅𝑦𝑦 (𝜏 )=∫−∞
∞
∫−∞
∞
𝐸 [ 𝑋 (𝑡+𝜏−𝛼 ) 𝑋 (𝑡− 𝛽 )]h (𝛼 )h (𝛽 )𝑑𝛼 𝑑𝛽
𝑌 (𝑡+𝜏)=∫−∞
∞
𝑋 (𝑡+𝜏−𝛼 )h (𝛼 )𝑑𝛼
𝑌 (𝑡)=∫−∞
∞
𝑋 (𝑡− 𝛽)h (𝛽 )𝑑 𝛽
Y(t) and Y(t+τ) is :
𝑅𝑦𝑦 (𝜏 )=∫−∞
∞
∫−∞
∞
𝑅𝑥𝑥(𝜏−𝛼+𝛽)h (𝛼 )h (𝛽 )𝑑𝛼 𝑑𝛽
42
Autocorrelation of the output (2/2)
𝑅𝑦𝑦 (𝜏 )=∫−∞
∞
∫−∞
∞
𝐸 [ 𝑋 (𝑡+𝜏−𝛼 ) 𝑋 (𝑡− 𝛽 )]h (𝛼 )h (𝛽 )𝑑𝛼 𝑑𝛽
By substitution: α=-β
𝑅𝑦𝑦 (𝜏 )=∫−∞
∞
∫−∞
∞
𝐸 [ 𝑋 (𝑡+𝜏−𝛼 ) 𝑋 (𝑡+𝛼 )] h (𝛼 )h (−𝑎 )𝑑𝛼 𝑑𝛼
Remember:
-30 -20 -10 0 10 20 30-1000
-500
0
500
1000
(s)
Rxy
()
Autocorrelation of y(t)
𝑅𝑦𝑦 (𝜏 )=𝑅𝑦 𝑥 (𝜏 )∗h(−𝜏)
𝑅𝑦𝑦 (𝜏 )=𝑅𝑥 𝑥 (𝜏 )∗h (𝜏)∗h(−𝜏 )
43-2 -1 0 1 2
0
2
4
6
8
10
12x 10
5
f (Hz)
Syy
(f)
Spectrum of output
• Given:
• The power spectrum is
𝑅𝑦 𝑦 (𝜏 )=𝑅𝑥𝑥 (𝜏 )∗h (𝜏 )∗h(−𝜏 )
-2 -1 0 1 20
0.5
1
1.5
f (Hz)
|H(f
)|2
-2 -1 0 1 20
2
4
6
8
10x 10
6
f (Hz)
Sxx
(f)
x =
¿𝐻 ( 𝑓 )∨¿2=𝐻 ( 𝑓 )𝐻∗( 𝑓 )¿
44
Filter examples
45
Typical LIT filters
• FIR filters (Finite impulse response)• IIR filters (Infinite impulse response)
– Butterworth– Chebyshev– Elliptic
Ideal filters
• Highpass filter
• Band stop filter
• Bandpassfilter
47
Filter types and rippels
Analog lowpass Butterworth filter
• Is ”all pole” filter– Squared frequency transfer function
• N:filter order• fc: 3dB cut off frequency
• Estimate PSD from filter
NcfffH 2
2
/1
1)(
Nc
xxyyff
ffS 2/1
1)(S)(
Chebyshev filter type I
• Transfer function
• Where ε is relateret to ripples in the pass band
• Where TN is a N order polynomium
pN ffTfH
/1
122
2
1
1
)coshcosh(
)coscos()(
1
1
x
x
xN
xNxTN
Transformation of a low pass filter to other types (the s-domain)
Filter type Transformation New Cutoff frequency
Lowpas>Lowpas
Lowpas>Highpas
Lowpas>Highpas
Lowpas>Stopband
ssp
p
'
p'
ss pp '
p'
)(
2
lu
ulp s
ss
ul
lup s
ss
2
)(
ul ,
ul ,
Lowest Cutoff frequency
Highest Cutoff frequency:
:
u
l
p
New Cutoff frequencyp'
Old Cutoff frequency
51
Discrete time implantation of filters
• A discrete filter its Transfer function in the z-domain or Fourier domain
– Where bk and ak is the filter coefficients
• In the time domain:
Mm
Mm
zazazaa
zbzbzbbzH
zX
zY
.......´
.......´)(
)(
)(2
21
10
22
110
][......]2[]1[
][......]2[]1[][][
21
210
Mnyanyanya
Mnxanxbnxbnxbny
M
M
52
Filtering of a Gaussian process
• Gaussian process– X(t1),X(t2),X(t3),….X(tn) are jointly Gaussian for all t
and n values• Filtering of a Gaussian process
– Where w[n] are independent zero mean Gaussian random variables.
][......]2[]1[
][......]2[]1[][][
21
210
Mnyanyanya
Mnwanwbnwbnwbny
M
M
The Gaussian Process
• X(t1),X(t2),X(t3),….X(tn) are jointly Gaussian for all t and n values
• Example: randn() in Matlab
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000-4
-3
-2
-1
0
1
2
3
4
5Gaussian process
-4 -3 -2 -1 0 1 2 3 4 50
100
200
300
400
500
600
700Histogram of Gaussian process
The Gaussian Process and a linear time invariant systems
• Out put = convolution between input and impulse response
Gaussian input Gaussian output
Example
• x(t):
• h(t): Low pass filter• y(t):
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000-4
-3
-2
-1
0
1
2
3
4
5Gaussian process
-4 -3 -2 -1 0 1 2 3 4 50
100
200
300
400
500
600
700Histogram of Gaussian process
-1.5 -1 -0.5 0 0.5 1 1.50
100
200
300
400
500
600Histogram of y(t)
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000-1.5
-1
-0.5
0
0.5
1
1.5
56
Filtering of a Gaussian process example 2
0 100 200 300 400 500-1000
-500
0
Frequency (Hz)
Pha
se (
degr
ees)
0 100 200 300 400 500-100
-50
0
Frequency (Hz)
Mag
nitu
de (
dB) Transfere function of filter
0 100 200 300 400 500 600 700 800 900 1000-4
-2
0
2
4
t (ms)
x(t)
White noise
Band pass filter
0 100 200 300 400 500 600 700 800 900 1000-1
-0.5
0
0.5
1
t (ms)
y(t)
Output
57
Intro to system identification
• Modeling of signals using linear Gaussian models:
• Example: AR models
• The output is modeled by a linear combination of previous samples plus Gaussian noise.
][][......]2[]1[][ 21 nwMnyanyanyany M
58
Modeling example
• Estimated 3th order model
][]3[0.7299-]2[2.3903]1[-2.6397][ nwnynynyny
0 100 200 300 400 500 600 700 800 900 1000-1
-0.5
0
0.5
1
t (ms)
y(t)
Output
451 451.5 452 452.5 453 453.5 4540.25
0.3
0.35
0.4
t (ms)y(
t)
Output
signal
points used for predictionPrediction
True point
453.98 453.99 454 454.01 454.02
0.282
0.284
0.286
0.288
0.29
0.292
0.294
t (ms)
y(t)
Output
w[n]
59
Agenda (Lec. 7)
• Recap: Linear time invariant systems• Stochastic signals and LTI systems
– Mean Value function– Mean square value – Cross correlation function between input and output– Autocorrelation function and spectrum output
• Filter examples • Intro to system identification
Top Related