DCSP-15: Wiener Filter and Kalman Filter*

Jianfeng Feng

Department of Computer Science Warwick Univ., UK

Jianfeng.feng@warwick.ac.uk

http://www.dcs.warwick.ac.uk/~feng/dcsp.html

Father of cybernetics (control theory)

Norbert Wiener: Harvard awarded Wiener a PhD, when he was 17.

The RGB format stores three color values, R,

G and B, for each pixel.

RGB = imread(‘bush.png');

size(RGB)

ans =

1500 1200 3

imshow(RGB)

Example

Example

SetupRecorded signal x(n,m) = s(n,m) +x(n,m)

for example, an image of 1500 X1200

True Signal noise

RGB = imread(‘bush.png');

I = rgb2gray(RGB);

J =

imnoise(I,'gaussian',0,0.0

05);

figure, imshow(I),

figure, imshow(J)

Setup

To find a constant a(m,n) such that

E ( a(m,n) x(m,n) – s(m,n) ) 2

as small as possible

y(m,n) = a(m,n) x(m,n)

Different from the filter before, a(m,n) depends on (m,n) :

Adaptive filter

Setup

Minimum mean square error which is one of the most commonly used methods

The way to solve it is the gradient descent method (Newton method)

For a given error function c(a), we simply update a according to

a prop - c’(a)

C(a) is usually called cost function

Setup

C(a)

Luckly

Gradient Descent For Machine Learningb

C(a)

Gradient Descent For Machine Learningb

C(a)

Gradient Descent For Machine Learningb

Hard to reach the global minima

2 2 2 2

2 2 2

( ) ( 2 )

2

E ax s E a x axs s

a Ex aExs Es

The quanttity above is minimized if the derivative

of it with respect to a is zero

2aEx2 - 2Exs = 0

a =Exs

Ex2=E(s+x )s

Es2 + Ex 2=

Es2

Es2 + Ex 2=

s 2 - Ex 2

s 2

Setup

(optimal Wiener gain)

• Without loss of generality, we assume that

Es=0 Ex=0

for simplicity of formulation .

• S, x are independent

Setup

y(n1,n

2) = m(n

1,n

2)+

s 2(n1,n

2)-Ex 2

s 2(n1,n

2)

(x(n1,n

2)-m(n

1,n

2))

The filtered output is given by

Note that the coefficient a depends on the position

s is the local variance, Ex2 is the global variance (noise)

Algorithm

where summation is over an area of N and M

1 2

1 2

1 2

,

2 2 2

1 2

,

1( , )

1( , )

n n

n n

x n nNM

x n nNM

s

3X 3 area

(N=3, M=3)

Algorithm

Algorithm in Matlab

• wiener2 lowpass - filters an intensity

image that has been degraded by constant

power additive noise.

• wiener2 uses a pixelwise adaptive Wiener

method based on statistics estimated from

a local neighborhood of each pixel.

RGB = imread(‘bush.png');

I = rgb2gray(RGB);

J =

imnoise(I,'gaussian',0,

0.005);

K = wiener2(J,[5 5]);

figure, imshow(J),

figure, imshow(K)

Example

• Large M and N reduce noise, but blur the photo

• small M and N will not affect noise

• optimal size should be used

Further issues

N=M=50 N=M=2

•A more general version of Wiener: Kalman filter

•Kaman filter is widely used in many areas

(still a very hot research topic)

• As an introductory course, we only give you

a taste of its flavour

Further issues

Kalman Filter

Kalman Filter: find out xk

• Have two variables state variable x which is

not observable

xk = Fk xk-1 + Bkuk + wk

• Observable variable z which we can observe,

but is contaminated by noise

zk = Hk xk +vk

where wk ~ N(0,Qk) and vk ~ N(0,Rk) are noise, Fk and Hk are constants.

Xk-1Input uk

Bk

Output xk

Kalman FilterFor given xk-1|k-1 and pK-1|K-1

Predict

•Predicted (a priori) state estimate xk|k-1 = Fk-1 xk-1|k-1+Bkuk

•Predicted (a priori) estimate covariance pk|k-1 = Fk-1 pk-1 |k-1 FK-1 +QK

Update

•measurement residual yk = zk –Hk X k|k-1

•measurement residual covariance Sk = Hk Pk|k-1HK+Rk

•optimal Kalman gain Kk = P k|k-1 Hk (SK)-1

•updated (a posteriori) state estimate X k|k = x k|k-1 + Kk y k

•updated (a posteriori) estimate covariance pK|K = (I-K kH k) P k|k-1

Kalman Filter

• It is one of the most useful filters in the literature, as shown below

• We do not have time to go further, but do read around

clear all

close all

N=1000;

q=1;

r=1;

f=0.1;

h=0.2;

x(1)=0.5;

xsignal(1)=0.5;

xhat=zeros(1,N+1);

phat=zeros(1,N+1);

xhat(1)=x(1);

phat(1)=x(1);

p(1)=1;

z(1)=0.1;

y(1)=1;

for i=1:N

xsignal(i+1)=f*xsignal(i)+randn(1,1)*sqrt(q)+1*cos(2*pi*0.01*i);

z(i+1)=h*xsignal(i)+randn(1,1)*sqrt(r);

end

for i=1:N

x(i+1)=f*xhat(i)+1*cos(2*pi*0.01*i);

p(i+1)=f*phat(i)*f+q;

y(i+1)=z(i+1)-h*x(i+1);

s(i+1)=h*p(i+1)*h+r;

k(i+1)=p(i+1)*h/s(i+1);

atemp=x(i+1)+k*y(i+1);

xhat(1,i+1)=atemp(1,i+1);

atemp=(1-k*h)*p(i+1);

phat(i+1)=atemp(1,i+1);

clear atemp

end

plot(xsignal);

hold on

plot(xhat,'r')

plot(z,'g')

plot(xhat,'r')

Applications• Attitude and Heading Reference Systems

• Autopilot

• Battery state of charge (SoC) estimation [1][2]

• Brain-computer interface

• Chaotic signals

• Tracking and Vertex Fitting of charged particles in Particle Detectors [35]

• Tracking of objects in computer vision

• Dynamic positioning

• Economics, in particular macroeconomics, time series, and econometrics

• Inertial guidance system

• Orbit Determination

• Power system state estimation

• Radar tracker

• Satellite navigation systems

• Seismology [3]

• Sensorless control of AC motor variable-frequency drives

• Simultaneous localization and mapping

• Speech enhancement

• Weather forecasting

• Navigation system

• 3D modeling

• Structural health monitoring

• Human sensorimotor processing[36]

Flight control unit of an Airbus A340

modern weather forecasting.

DCSP-Revision

Jianfeng Feng

Department of Computer Science Warwick Univ., UK

Jianfeng.feng@warwick.ac.uk

http://www.dcs.warwick.ac.uk/~feng/dcsp.html

Next week

• Guest lecture next Monday:

Prof. Edmund Rolls

Digital Computation in Your Brain

323× 397Images may be subject to copyright

Next week

• Guest lecture next Monday:

Prof. Edmund Rolls

Digital Computation in Your Brain

• Assignment help Tuesday

(same time and venue as lecture)

B13. Rolls,E.T. (2018) The Brain, Emotion, and Depression. Oxford University Press: Oxford. (In press.)

B12. Rolls,E.T. (2016) Cerebral Cortex: Principles of Operation. Oxford University Press: Oxford. Paperback published November 2017.

B11. Rolls,E.T. (2014) Emotion and Decision-Making Explained. Oxford University Press: Oxford. Virtual Special Issue of Cortex on Emotion and

Decision-Making Explained.

B10.

Rolls,E.T. (2012) Neuroculture: On the Implications of Brain Science. Oxford University Press: Oxford.

B9.

Rolls,E.T. and Deco,G. (2010) The Noisy Brain: Stochastic Dynamics as a Principle of Brain Function. Oxford University Press: Oxford.

B8. Rolls,E.T. (2008) Memory, Attention, and Decision-Making: A Unifying Computational Neuroscience Approach. Oxford University Press:

Oxford.

B7. Rolls,E.T. (2005) Emotion Explained. Oxford University Press: Oxford.

B6. Rolls,E.T. and Deco,G. (2002) Computational Neuroscience of Vision. Oxford University Press: Oxford.

B5. Rolls,E.T. (1999) The Brain and Emotion. Oxford University Press: Oxford.

B4. McLeod,P., Plunkett,K. and Rolls,E.T. (1998) Introduction to Connectionist Modelling of Cognitive Processes. Oxford University Press:

Oxford.

B3. Rolls,E.T. and Treves,A. (1998) Neural Networks and Brain Function. Oxford University Press: Oxford.

B2. Rolls,B.J. and Rolls,E.T. (1982) Thirst. Cambridge University Press: Cambridge.

B1. Rolls, E.T. (1975) The Brain and Reward. Pergamon Press Oxford.

Next week

• Guest lecture next Monday:

Prof. Edmund Rolls

Digital Computation in Your Brai

• Assignment help on Tuesday

(same time and venue as lecture)

Revision class (second week, next term)

Fourier Thm.

Any signal x(t) of period T can be

represented as the sum of a set of

cosinusoidal and sinusoidal waves of

different frequencies and phases

x(t) = A0 + An cos(nwt)+n=1

¥

å Bn sin(nwt)n=1

¥

å

A0 =<1, x(t) >=1

Tx(t)dt

-T /2

T /2

ò

An =< cos(nwt), x(t) >=2

Tx(t)cos(nwt)dt

-T /2

T /2

ò

Bn =< sin(nwt), x(t) >=2

Tx(t)sin(nwt)dt

-T /2

T /2

ò

w =2p

T

ì

í

ïïïïï

î

ïïïïï

Frequency domain

Example I

2/p

|X(F)|

Time domain

=

If the periodic signal is replace with an aperiodic signal (general case)

FT is given by

FT in complex exponential

X(F) = FT (x) = x(t)exp(-2p jFt)dt-¥

¥

ò

x(t) = FT -1(X) = X(F)exp(2p jFt)dF-¥

¥

ò

ì

íï

îï

Is that deadly simple?

Defined the information contained in an outcome xi in x={x1, x2,…,xn}

I(xi) = - log2 p(xi)

H(X)= Si P(xi) I(xi) = - Si P(xi) log2 P(xi)

Information

Entropy

An instantaneous code can be found that

encodes a source of entropy H(X) with an

average number Ls (average length)

such that

Ls >= H(X)

Shannon's first theorem

• Ls = 0.729*1+0.081*3*3+0.009*5*3+0.001* 5

= 1.5980 (remember entropy is 1.4)

Huffman coding

code length

The DTFT gives the frequency representation of

a discrete time sequence with infinite length.

X(w): frequency domain x [n]: time domain

Definition and properties:

X(w) = DTFT (x) = x[n]exp(- jwn)n=-¥

¥

å

x[n] = IDTFT (X) =1

2pX(w)exp( jwn)dw

-p

p

ò

ì

í

ïï

î

ïï

DFT IIIDefinition: The discrete Fourier transform (DFT) and its own

inverse DFT (IDFT) associated with a vector

X = ( X[0], X[1], … , X[N-1] )

to a vector

x = ( x[0], x[1], …, x[N-1] )

of N data points, is given by

X[k] = DFT{x[n]} = x[n]exp - j2pk

Nn

æ

èç

ö

ø÷

n=0

N-1

å

x[n] = IDFT{X[k]} =1

NX[k]exp j

2pk

Nn

æ

èç

ö

ø÷

k=0

N-1

å

ì

í

ïï

î

ïï

clear all

close all

sampling_rate=100;%Hz

omega=30; %signal frequecy Hz

N=20000; %total number of samples

for i=1:N

x_sound(i)=cos(2*pi*omega*i/sampling_rate); %signal

x(i)=x_sound(i)+2*randn(1,1); %signal+noise

axis(i)=sampling_rate*i/N; % for psd

time(i)=i/sampling_rate; % for time trace

end

subplot(1,2,1)

plot(time,x); %signal + noise, time trace

xlabel('second')

ylabel('x')

subplot(1,2,2)

plot(axis,abs(fft(x)),'r'); % magnitude of signal

xlabel('Hz');

ylabel('Amplitude')

sound(x_sound)

Mysterious sound

Simple Matlab program is left on slides

Fig. 2

Spectrogram • t=0:0.001:2; % 2

secs @ 1kHz sample rate

• y=chirp(t,100,1,200,'q'); %

Start @ 100Hz, cross 200Hz at

t=1sec

•

spectrogram(y,128,120,128,1E3);

% Display the spectrogram

• title('Quadratic Chirp: start at

100Hz and cross 200Hz at

t=1sec');

• sound(y)

•• y=chirp(t,100,1,200,'q');

% Start @

• Fs=5000;

• filename = 'h.wav';

• audiowrite(filename,y,Fs);

Time (sec)

F

r

e

q

u

e

n

c

y

DFT for image

• DFT for x

• IDFT for X

X[k1,k2 ]=

n2=0

N2-1

å x[n1,n2 ]exp(-2p jk1N1

n1)n1=0

N1-1

å exp(-2p jk2

N2

n2 )

x[n1,n2 ]=1

N1N2 k2=0

N2-1

å X[k1,k2 ]exp(2p jk1N1

n1)k1=0

N1-1

å exp(2p jk2

N2

n2 )

Original picture Gray scale picture

clear all

close all

figure(1)

I = imread('peppers.png');

imshow(I);

K=rgb2gray(I);

figure(2)

F = fft2(K);

figure;

imagesc(100*log(1+abs(fftshift(F)))); colormap(gray);

title('magnitude spectrum');

figure;

imagesc(angle(F)); colormap(gray)

Example I

Example I

• Notice two facts:

• The signal has a frequency spectrum within the interval 0 to Fs/2 kHz.

• The disturbance is at frequency F0 (1.5) kHz.

• Design and implement a filter that rejects

the disturbance without affecting the signal excessively.

• We will follow three steps:

simple filter design

Step 1: Frequency domain specifications

• Reject the signal at the frequency of the disturbance.

• Ideally, we would like to have the following frequency response:

where w0=2p (F0/Fs) = p/4 radians, the digital frequency

of the disturbance.

Step 2: Determine poles and zeros

We need to place two zeros on the unit circle

z1 =exp(jw0)=exp(jp/4)

and

z2=exp(-jw0)=exp(-jp/4)

z2H (z) = k(z - z1)(z - z

2)

= k[z2 - (z1+ z

2)z + z

1z

2]

= k[z2 - (1.414)z +1]

p/4

Step 2: Determine poles and zeros

• We need to place two zeros on the unit circle

z1 =exp(jw0)=exp(jp/4)

and

z2=exp(-jw0)=exp(-jp/4)

• If we choose, say K=1, the frequency response is shown in the figure.

z2H (z) = k(z - z1)(z - z

2)

= k[z2 - (z1+ z

2)z + z

1z

2]

= k[z2 - (1.414)z +1]

PSD and Phase

• As we can see, it rejects the desired frequency

• As expected, but greatly distorts the signal

Result

• A better choice would be to select the poles close to the zeros, within the unit circle, for

stability.

• For example, let the poles be p1= r exp(jw0) and p2= r exp(-jw0).

• With r =0.95, for example we obtain the transfer function

Recursive Filter

H (z) = k(z- z1)(z- z2 )

(z- p1)(z- p2 )

Recursive Filter

H(z) = k(z- z1)(z- z2 )

(z- p1)(z- p2 )= k

(z- p1 +e)(z- z2 )

(z- p1)(z- p2 )= k 1+

e

(z- p1)

æ

èç

ö

ø÷

(z- z2 )

(z- p2 )

e= z1 - p1 = (1-r) z1 ~ 0. So we have H(w0) = 0, but close to 1 everywhere else

• We choose the overall gain k so that H(z)|z=1=1.

• This yields the frequency response function as shown in the figures below.

H(z) = k(z- z1)(z- z2 )

(z- p1)(z- p2 )= 0.9543

z2 -1.414z+1

z2 -1.343z+0.9025

Recursive Filter

Step 3: Determine the difference

equation in the time

domain

• From the transfer function, the difference

equation is determined by inspection:

y(n)= 0.954 x(n) -1.3495x(n-1)+0.9543 x(n-2)

+1.343 y(n-1)-0.9025 y(n-2)

• Easily implemented as a recursion in a high-level language.

• The final signal y(n) with the frequency spectrum shown in the

following Fig.,

• We notice the absence of the disturbance.

Step 3: difference equation

Step 3: Outcomes

• Define ai = sN-i (reversing the order)

y(n) = a0 x(n) + a1 x(n-1) + … + aN x(n-N)

So when the signal arrives we have

y(n) = SN x(n) + SN-1 x(n-1) + … + S1 x(n-N)

= SN SN + SN-1 SN-1 + … + S1 S1

Matched filter

y(n1,n

2) = m(n

1,n

2)+

s 2(n1,n

2)-Ex 2

s 2(n1,n

2)

(x(n1,n

2)-m(n

1,n

2))

The filtered output is given by

Note that the coefficient a depends on the position

s is the local variance, Ex2 is the global variance (noise)

Wiener Filter

Wiener Filter

past papers before the exam