Chapter 12: Iterative Methods

2

• Linear Equations– Gauss-Seidel– Jacobi

• Nonlinear Equations– Successive substitution– Newton-Raphson

3

Jacobi Iterative Method

• Goal: solve [A][x]=[b] iteratively. • Algorithm: let [A]=[A’] + [D] where

[A’]: the off-diagonal part of [A],[D]: the diagonal part of [A],then

[D][x]=[b]-[A’][x].The j-th iteration of x is computed by

[xj]=[D]-1([b]-[A’][xj-1])

4

Jacobi Iterative Method (cont.)

• Need initial guess of x.• Convergent not guaranteed.• In mathematical form

x ji

bin

m1aimx j1

m

aii, i1,2,...,nx j

i

bin

m1aimx j1

m

aii, i1,2,...,n

5

Gauss-Seidel Method

• Goal: solve [A][x]=[b] iteratively. • Algorithm: compute the j-th iteration of x

by

x ji

bii1

m1aimx j

m n

mi1aimx j1

m

aii, i1,2,...,nx j

i

bii1

m1aimx j

m n

mi1aimx j1

m

aii, i1,2,...,n

6

Gauss-Seidel Method (cont.)

• Use updated values of xij as soon as possible.

• Example: n=3

x j1

b1a12xj1

2 a13xj13

a11

x j2

b2a21xj

1a23xj1

3

a22

x j3

b3a31xj

1a32xj

2

a33

x j1

b1a12xj1

2 a13xj13

a11

x j2

b2a21xj

1a23xj1

3

a22

x j3

b3a31xj

1a32xj

2

a33

7

Example 12.1

• Solve the following by Gauss-Seidel method.

x17.850.1x20.2x3

3

x219.30.1x10.3x3

7

x371.40.3x10.2x2

10

x17.850.1x20.2x3

3

x219.30.1x10.3x3

7

x371.40.3x10.2x2

10

3x1 0.1x2 0.2x3 7.85

0.1x1 7x2 0.3x3 19.3

0.3x1 0.2x2 10x3 71.4

3x1 0.1x2 0.2x3 7.85

0.1x1 7x2 0.3x3 19.3

0.3x1 0.2x2 10x3 71.4

x1

x2

x3

32.5

7

x1

x2

x3

32.5

7Answer:

8

Comparison

• Gauss-Seidel usually but not always converges fasterthan Jacobi because the updated values of x are used as soon as possible.

9

Convergence Criteria

• Sufficient condition: diagonal dominant

• Proof:

|aii|> N

j1,ji|aij|, i1,2,...,N|aii|>

N

j1,ji|aij|, i1,2,...,N

x (n1)i

bi

aii

N

j1,ji

aij

aiix (n)

j

x (n1)i x (n)

i N

j1,ji

aij

aii(x (n)

j x (n1)j )

x (n1)i

bi

aii

N

j1,ji

aij

aiix (n)

j

x (n1)i x (n)

i N

j1,ji

aij

aii(x (n)

j x (n1)j )

10

Convergence Criteria (cont.)

• Proof (cont.):

Δx (n1)i

N

j1,ji

aij

aiiΔx (n)

j

N

j1,ji

aij

aii Δx (n)

j

Δx (n)max

N

j1,ji

aij

aii

Δx (n)max

Δx (n1)i

N

j1,ji

aij

aiiΔx (n)

j

N

j1,ji

aij

aii Δx (n)

j

Δx (n)max

N

j1,ji

aij

aii

Δx (n)max

11

Relaxation

• Purpose: to speed up converge by

• is problem specific, determined experimentally.

x newi λx new

i (1λ)x oldi , 0λ2x new

i λx newi (1λ)x old

i , 0λ2

12

Successive Substitution

f1(x1,x2,...,xn)0

f2(x1,x2,...,xn)0

fn(x1,x2,...,xn)0

x1g1(x1,x2,...,xn)

x2g2(x1,x2,...,xn)

xngn(x1,x2,...,xn)

f1(x1,x2,...,xn)0

f2(x1,x2,...,xn)0

fn(x1,x2,...,xn)0

x1g1(x1,x2,...,xn)

x2g2(x1,x2,...,xn)

xngn(x1,x2,...,xn)

x j11 g1(x

j1,x

j2,...,x

jn)

x j12 g2(x

j11 ,x j

2,...,xjn)

x j1n gn(x

j11 ,x j1

2 ,...,x jn)

x j11 g1(x

j1,x

j2,...,x

jn)

x j12 g2(x

j11 ,x j

2,...,xjn)

x j1n gn(x

j11 ,x j1

2 ,...,x jn)

13

Successive Substitution (cont.)

• Convergent Criteria (sufficient only): in an interval about the root, where initial guess is within

• May be difficult to find.

n

i1

gj

xi<1, j1,2,..n

n

i1

gj

xi<1, j1,2,..n

14

Successive Substitution (cont.)

• Example 12.2– Initial value: x1=1.5, x2=3.5 (correct answer: x1=2, x2=3)– Compare the following two cases

x 21x1x2100

x23x1x2257

x1

10x 21

x2

x2573x1x22

x 21x1x2100

x23x1x2257

x1

10x 21

x2

x2573x1x22

x 21x1x2100

x23x1x2257

x1 10x1x2

x257x2

3x1

x 21x1x2100

x23x1x2257

x1 10x1x2

x257x2

3x1

15

Newton-Raphson

• Let x1j, x2

j, …, xnj be close to the root, then

by Taylor’s expansion

f j11 f j

1n

i1(x j1

i x ji )f j

1

xi

f j12 f j

2n

i1(x j1

i x ji )f j

2

xi

f j1n f j

1n

i1(x j1

i x ji )f j

n

xi

f j11 f j

1n

i1(x j1

i x ji )f j

1

xi

f j12 f j

2n

i1(x j1

i x ji )f j

2

xi

f j1n f j

1n

i1(x j1

i x ji )f j

n

xi

16

Newton-Raphson (cont.)

• If x1j+1, x2

j+1, …, xnj+1 is the root, then

17

Newton-Raphson (cont.)

0f j1

n

i1(x j1

i x ji )f j

1

xi

0f j2

n

i1(x j1

i x ji )f j

2

xi

0f j1

n

i1(x j1

i x ji )f j

n

xi

0f j1

n

i1(x j1

i x ji )f j

1

xi

0f j2

n

i1(x j1

i x ji )f j

2

xi

0f j1

n

i1(x j1

i x ji )f j

n

xi

f j1

f j2

f jn

f j1

x1

f j1

x2

f j1

xn

f j2

x1

f j2

x2

f j2

xn

f jn

x1

f jn

x2

f jn

xn

x j11 x j

1

x j12 x j

2

x j1n x j

n

f j1

f j2

f jn

f j1

x1

f j1

x2

f j1

xn

f j2

x1

f j2

x2

f j2

xn

f jn

x1

f jn

x2

f jn

xn

x j11 x j

1

x j12 x j

2

x j1n x j

n

18

Newton-Raphson (cont.)

• To sum up: [xj+1]=[xj]- [J]-1 [f j], where

[f j]

f j1

f j2

f jn

, [J]

f j1

x1

f j1

x2

f j1

xn

f j2

x1

f j2

x2

f j2

xn

f jn

x1

f jn

x2

f jn

xn

, [x j]

x j1

x j2

x jn

[f j]

f j1

f j2

f jn

, [J]

f j1

x1

f j1

x2

f j1

xn

f j2

x1

f j2

x2

f j2

xn

f jn

x1

f jn

x2

f jn

xn

, [x j]

x j1

x j2

x jn

19

Newton-Raphson (cont.)

• Shortcomings– [J] is difficult to evaluate → use finite

difference approximation.– Good initial guess is required to ensure

converge.• Alternative: nonlinear optimization

F(x)n

i1fi(x1,x2,...,xn)

2F(x)n

i1fi(x1,x2,...,xn)

2

20

Newton-Raphson (cont.)

• Example 12.3: solve the following equations by Newton-Raphson method with initial guess x1=1.5, x2=3.5.

x 21x1x2100

x23x1x2257

f1x1

2x1x2f1x2

x1

f2x1

3x 22

f2x2

16x1x2

x 21x1x2100

x23x1x2257

f1x1

2x1x2f1x2

x1

f2x1

3x 22

f2x2

16x1x2