Chapter-13: Least Mean Squares (LMS) Algorithm Marc Moonen€¦ · Chapter-11 Chapter-12 Chapter-13...

1

DSP-CIS

Chapter-13: Least Mean Squares (LMS) Algorithm

Marc Moonen Dept. E.E./ESAT-STADIUS, KU Leuven

[email protected] www.esat.kuleuven.be/stadius/

DSP-CIS / Chapter-13 : Least Mean Squares (LMS) Algorithm / Version 2014-2015 p. 2

Part-III : Optimal & Adaptive Filters : Optimal & Adaptive Filters - Intro

•  General Set-Up •  Applications •  Optimal (Wiener) Filters

–  : Least Squares & Recursive Least Squares Estimation •  Least Squares Estimation •  Recursive Least Squares (RLS) Estimation •  Square-Root Algorithms

–  : Least Means Squares (LMS) Algorithm •  LMS/NLMS : Stochastic Gradient Algorithms •  LMS analysis •  LMS Family

–  : Fast Recursive Least Squares Algorithms : Kalman Filtering

Chapter-11

Chapter-12

Chapter-13

Chapter-14

Chapter-15

2


Least Mean Squares (LMS) Algorithm



3



6 6 6 6

b

b

-w

a-bw

w

input signal

6

a

6

µ

6

- - - -

b b

aw

w+ab

u[k] u[k-1] u[k-2] u[k-3]

w0[k-1] w1[k-1] w2[k-1] w3[k-1]

w0[k] w1[k] w2[k] w3[k]

output signal

e[k] d[k]

desired signal

(Widrow 1965 !!)



Bernard Widrow https://www.youtube.com/watch?v=hc2Zj55j1zU

4





5



è large λ_max implies a small stepsize



h

minh

1<µh

max

-1

1µ=0

µ=2/h

µ

µ=1/h max

opt

max

6



error vector projected onto eigenvectors initial error vector projected onto eigenvectors

(=projection on i-th eigenvector)

è small λ_i implies slow convergence è  λ_min <<λ_max (hence small µ) implies *very* slow convergence



w(4)

w(0)

wopt

w1

w2

q2q1

7



w(4)

wopt

w(0)

w1

w2

q1



w(0)

w(4)

wopt

w1

w2

q1

8





9





10



Skip

this

slid

e



Skip

this

slid

e

11



Skip

this

slid

e



Skip

this

slid

e

12



Skip

this

slid

e



Skip

this

slid

e

13



66 6 6 6

b

wb

-w

aa-bw

6

6

6

6

6

6

6

W^1 W^2 W^3 W^4 W^5 W^6 W^7

W^2 W^4 W^6 W^8 W^10 W^12 W^14

6

W^6 W^9 W^12 W^15 W^18 W^21

6 6

W^3

00000000

W^0 W^0 W^0 W^0 W^0 W^0W^0W^0

W^0

W^0

W^0

W^0W^0

W^0

W^0

W^0

w0

- - -

w2 w3

b b

aw

w+ab

--

w1

- - -

w4 w5 w6 w7

desired signal

filter input

error

W^4 W^8 W^12 W^16 W^20 W^24 W^28

W^5

W^6

W^7

W^10 W^15 W^20 W^25 W^30 W^35

W^12 W^18 W^24 W^30 W^36 W^42

W^14 W^21 W^28 W^35 W^42 W^49

µ0 µ1 µ2 µ3 µ4 µ5 µ6 µ7

Skip

this

slid

e



+

+

+

++

+

+

+

+

+

+

+

+

++ +

+

+

+

+

+

+

+

+

6

6

-1

-1

-1

-1

6 6 6 66 6 6 6

6

b

wb

-w

a

6

6

6

6

a-bw

filter input

W^1

W^2

W^3

-1

-1

-1

-1

-1

-1

-1

-1

W^2

W^2

w0

- - -

w2 w3

b b

aw

w+ab

--

w1

- - -

w4 w5 w6 w7

desired signalerror

µ0 µ1 µ2 µ3 µ4 µ5 µ6 µ7

Skip

this

slid

e

14



+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

+

-1

-1

-1

-1

6 6 66 6 66 6

b

b

-w

aa-bw

6

w

6

6

6

6

6

66

6

6

filter input

-1

-1

-1

-1

W^2

W^2

w0

- - -

w2 w3

b b

aw

w+ab

--

w1

- - -

w4 w5 w6 w7

desired signalerror

µ0 µ1 µ2 µ3 µ4 µ5 µ6 µ7

W^1

W^2

W^3

-1

Skip

this

slid

e



+

+

+

+

+

+

+

++

+

++

+

+

-1

6

-1

66 66 6 66

b

-w

aa-bw

-1

-1

b

6

6

w

6

6

6

-1

6

-1

6

6

6

6

6

6

filter input

w0

- - -

w2 w3

b b

aw

w+ab

--

w1

- - -

w4 w5 w6 w7

desired signalerror

µ0 µ1 µ2 µ3 µ4 µ5 µ6 µ7

-1

W^2

W^2

W^1

W^1.W^2

Skip

this

slid

e

15





16





17





6 6

6 6 6 6

6 6

6 6 6 6

2 2

HP HP

2 2

LP

2

LP

2

filter input desired signal

xµ

6

w0

- - - -

w1 w2 w3

xµ

6

w0

- - - -

w1 w2 w3

error (HP)

LP

+error

error (LP)

HP

Chapter-13: Least Mean Squares (LMS) Algorithm Marc Moonen€¦ · Chapter-11 Chapter-12 Chapter-13...

Documents

Transcript of Chapter-13: Least Mean Squares (LMS) Algorithm Marc Moonen€¦ · Chapter-11 Chapter-12 Chapter-13...