08.03.2016

SOLUTION METHODS for f(x)=0

Single Nonlinear Equation o The Bisection Method of Bolzano o Method of False Position (Regula Falsi Method) o Method of Linear Iteration (Fixed Point Iteration) o Newton's Method o Secant Method

System of nonlinear equations o Jacobi Method o Modified Jacobi Method o Newton's Method

Single Nonlinear Equation

f(x)=0 (1)

Example : ex-x ln(x)=0

Our aim is to find the value of x which satisfies Eq. (1). That value of x is called a root of Eq. (1). When f is linear function of x, f(x)=0 will be a linear equation (LE). When f is nonlinear (NL) function of x, f(x)=0 will be a nonlinear equation (NLE). LE’s or NLE's arise from physical problems.

Example for linear equation (LE):

f(x)=ax+b, f(x)=0 ax+b=0 x=-b/a

where x is the root of the LE. Unique solution!

Nonlinear equation (NLE)

If f(x) is NL function of x, it might have no roots or one root or more than one root.

Example

f(x)=xtan(x)-1 Nonlinear equation (NLE)

f(x)=0 xtan(x)-1=0.

To make a crude plot, xtan(x)=1, then tan(x)=1/x

Then graph is as follows:

From graph, function has many roots.

The Bisection Method of Bolzano A continuous function f(x)=0 is considered. Choose initial interval [a,b] where f(a) and f(b) have different signs.

2

bac

If f(a) and f(c) have opposite signs (i.e. f(a)*f(c)<0), then there is a root in [a,c] If f(c) and f(b) have opposite signs (i.e. f(c)*f(b)<0), then there is a root in [c,b] If f(c)=0, then c is a root.

Start with [a0,b0], find c0

2

bac 00

0

If f(a0) and f(c0) have opposite signs then a1=a0 and b1= c0. If f(c0) and f(b0) have opposite signs then a1=c0 and b1=b0, Therefore the new interval that contains root is 11 b,a

2211

111 b,a 2

bacb,a

3322

222 b,a2

bacb,a

2

bacb,a nn

nnn

When n goes to infinity or for very large value of n nn1n1n c,ab,a or

nn1n1n b,cb,a for all n. And nn b,ar or 1n1n b,ar .

012n1n1nn210 bbbbbraaaaa

Theorem: (Bisection method) Assume that b,aC)x(f and there exists a number b,ar such that f(r)=0. If f(a) and f(b)

have apposite signs and 0nnc represents the sequence midpoints generated by the bisection

process, and 1n

00n 2

abcr

for n=0,1,2,... then the sequence 0nnc converges to the root

r, that is rcLim nn

Proof :

nnn

nn

bca

bra

1nallfor2

abcr nn

n

2

abab 00

11

200

00

1122 2

ab

22

ab

2

abab

and so on, 22

abab

n00

nn

Combining (1) and (2)

22

ab

crn

00

n

, then 1n

00n 2

abcr

for all n.

1n

00n

n 2

abcr Lim

1n

00

nn

n 2

ab Limcr Lim

it is obvious that 02

ab Lim

1n

00

n

then left hand side 0rcLim nn

rcLim nn

or rcn nc is a root.

Therefore, proof of theorem also gives us error expression for Bisection method which is

nn cre or 1n

00n 2

abe

Example f(x)=xsin(x)-1 and f(x)=0 xsin(x)-1=0 sin(x)=1/x

From graph, initial interval can be taken as [0,2] Since f(0)= -1 and f(2)= 0.818595 There is a root in [0,2]

k left end point ak midpoint ck right end point bk f(ck)

0 1 2 3 4 5 6 7 8

0.000000 1.000000 1.000000 1.000000 1.000000 1.062500 1.093750 1.109375 1.109375

1.000000 1.500000 1.250000 1.125000 1.062500 1.093750 1.109375 1.117188 1.113281

2.000000 2.000000 1.500000 1.250000 1.125000 1.125000 1.125000 1.125000 1.117188

-0.158529 0.496224 0.186231 0.015051 -0.071827 -0.028362 -0.006643 0.004208 -0.001216

For two decimal place accuracy, root is 1.11 from table, 8cr . Then error is

1n

00n 2

abe

00390625.0

2

02e

188

for 31 iteration error will be 1013131 10x65.4

2

02e

Method of False Position(Regula Falsi Method)

ac

af0

ab

afbf

ac

af0m

ab

afbfm

afbf

abafac

(*)

cb

0bf

ab

afbf

cb

0bfm

ab

afbfm

afbf

abbfbc

(**)

The three possibilities are:

If f(a) and f(c) have opposite signs, (f(a)*f(c)<0) then root lies in [a,c] If f(c) and f(b) have opposite signs, (f(c)*f(b)<0) then root lies in [c,b] If f(c)=0, then c is the root.

For an iteration, Equation (**) is chosen. (Equation (*) may also be chosen)

nn

nnnnn afbf

abbfbc

Example

f(x)=xsin(x)-1

Take initial interval as [0,2]

ii

iiiii afbf

abbfbc

Starting with i=0

10 00 afa

81859485.02 00 bfb

09975017.1

181859485.0

0281859485.020

c

01010 02001921.0)1( bbcafcf

12124074.1

02001921.081859485.0

09975017.1281859485.021

c

12121 00983461.0 cbaacf

k ak ck bk f(ck)

0

1

2

3

0.0000000

1.09975017

1.09975017

1.09975017

1.09975017

1.12124074

1.11416120

1.11415714

2.0000000

2.0000000

1.12124074

1.11416120

-0.02001921

0.00983461

0.000000563

0.000000000

Method of Linear Iteration (LIM) (or Fixed Point Iteration)

f(x)=0 (1)

Put (1) into the form:

x=g(x) (2)

The representation of Eq. (2) of f(x) is not unique, such as

f(x)=xtan(x)-6

f(x)=0 xtan(x)=6 x=6/tan(x) then g(x)= 6/tan(x)

or

xx

xxxx

6tan

6tan6tan 1 then

xxg

6tan 1

or 6tan xx 06tan xx xxxx 6tan then x6xtanxxg

Derivation of formula for iteration:

Note that if r is a root of f(x)=0, then f(r)=0 and r = g(r).

Choose an initial approximation x0 to a root of f = 0

x=g(x)

x1=g(x0) , x1 is the next improved estimate

x2=g(x1)

x3=g(x2) . . . xi+1=g(xi) i=0,1,2,3... (2)

Claim: If the squence {x0,x1,x2,...} generated by (2) ( xi+1=g(xi) ) converges, let’s say to , then will be one of the roots of f=0

Proof

roottheis

0f

g

xglimxlimxgx

assumedisxlim

ii

1ii

i1i

ii

Therefore is a root.

When does {x0,x1,x2,...} converges ?

In fact it is easy to construct the sequence which will diverge.

f(x)=xtan(x)-1 Nonlinear equation (NLE)

f(x)=0 xtan(x)-1=0. To make a crude plot, xtan(x)=1 tan(x)=1/x

wish to find r1 by LIM. Note that there is a root in the interval 2

r0 1

1xtanxxf

xtan

1x0xf then xtan

1xg

79.04

xChoose 0

.1

4tan

1

xtan

1x

01

64.0.1tan

1

xtan

1x

12

i xi

0 0.79

1 1.

2 0.64

3 1.34

4 0.24

5 4.12

|xi+1-xi|

0.21

0.26

0.7

1.1

3.88

It diverges since relative simple error |xi+1-xi| is increasing while i is increasing.

Converging case:

1xtanxxf x

1xtan0xf then

x

1arctanxg

79.04

xChoose 0

91.0

4

1arctan

x

1arctanx

01

84.091.0

1arctan

x

1arctanx

12

i xi |xi+1-xi|

0 0.79

1 0.91 0.12

2 0.84 0.07

3 0.87 0.03

4 0.86

5 0.86

It converges since error |xi+1-xi| is decreasing while i is increasing.

1r86.0x which is correct to 2 rounded decimals

21 105.086.0r

CONVERGENCE CONDITION FOR LIM

{x0,x1,x2...} will converge if :

a) g & g' are continuous in I

b) 1xg in I

c) x0 is chosen to be in “I”

Theorem : Let x=r be a root. Let “I” be an interval containing this root

i. g(x) and g'(x) are continuous in I, ii. |g'(x)|<1 for all points in I,

iii. initial approximation x0 is chosen in I,

then the iteration converges to root r

x=g(x) equivalent form of f(x)=0 to iterate

r=g(r) (1)

On the other hand,

xi+1=g(xi) (2) recursive formula for LIM

Subtracting (2) from (1)

xi+1-r=g(xi)-g(r)

By mean value theorem, there exists a point i between xi and r. Then

g(xi)-g(r) = g'(ηi)(xi-r)

xi+1-r = g'(ηi)(xi-r)

let i=0

x1-r = g'(η0)(x0-r)

Taking absolute values of both sides

x

a b

I

r x0

f(x)

rxgrx 001 )(

rx 1 error in first iteration and rx 0 error in initial guess.

For convergence rxrx 01 implies that 1g 0

let i=1

x2-r = g'(η1)(x1-r)

Taking absolute values of both sides

rxgrx 112 )(

rx 2 error in 2nd iteration and rx 1 error in first iteration.

For convergence rxrx 12 implies that 11 g

when iteration converges, (ei+1 =|xi+1-r| and ei =|xi-r|)

ii

iii

1ii

erglimeglimelim

Since rx ii when i goes to infinity, in short

ei+1=constant*ei for i>>1

This result implies that; the proportion of the error at any iteration to the error at previous iteration is constant when "i" becomes very large. This is the reason why this iteration or method is called linear and named as “Linear Iteration Method”..

Proof of Convergence:

Start with i=0 in xi+1-r = g'(ηi)(xi-r)

It is assumed that 1g 0 and x1-r =

g'(η0)(x0-r) or rxgrx 001 and

rx0

Then rxrx 01 which means that x1 is in the interval I.

x

a b

I

r x0

f(x)

Next, take i=1,

rxrxgrx 1112

e1 =|x1-r|, e2 =|x2-r| errors in iterations. Then the error expression for LIM is:

ii1i ege for any i (ei+1 =|xi+1-r| and ei =|xi-r|)

Since power of ei is 1, it is said to be the rate of convergence is 1 or linear. Now let us introduce positive quantities k0, k1,…, ki defined by

ii1100 gkgkgk

and K, such that K=max(k0, k1,…, ki,…), since 0, 1, …, i,… are all in interval I . Due to assumption in the theorem k0, k1,…, ki,… are all less than one, hence K<1.

ii1i ege or i1i eKe

which is true for any i. It may be written recursively

1ii eKe , 2i1i eKe , 3i2i eKe

therefore, one gets

0rxlimelim i

ii

i

since K<1 and 0Ki , then xi = r.

Error Expression in General

i1i xgx (I)

rgr (II)

(II)-(I) rgxgrx i1i (III) Taylor expansion of g(xi) about r

T.O.Hrxrg!3

1rxrg

!2

1rxrgrgxg 3

i2

iii

0i

iii

2i2

1ii eKeKeKeKe

0xxlimrxlim

rxlimi1i

i1ii

ii

T.O.Hrxrg!3

1rxrg

!2

1rxrgrgxg 3

i2

iii (H.O.T stands for Higher Order Terms) And rewriting the Equation,

T.O.Hrxrg!3

1rxrg

!2

1rxrgrgxgrx 3

i2

iii1i

T.O.Hrxrg!3

1rxrg

!2

1rxrgrx 3

i2

ii1i

Where rxe 1i1i and rxe ii

Then error expression takes following form in general:

T.O.Herg!3

1erg

!2

1erge 3

i2ii1i

Where ierg is dominating term and rest of the terms may be assumed as higher order terms

and neglected, therefore error expression for LIM is as follows:

i1i erge Note that when i goes to infinity, in short rxx ii1i .

Example 0x3exf x

X f(x) - 0 1 1 -0.28 2 1.39 Wish to find r1 and r2 by LIM

x3lnelnx3e0x3e xxx x3lnxgx3lnx (*)

1x1when1xg

1x

1x1xwhen1xg

x

1xg

x

13

x3

1xg

(*) is suitable for finding r2 provided that x0 is choosen in (1,). (*) is not suitable for finding r1.

For finding r2 choose x0=2 x1=ln(3 x0)=ln(6)=1.79 x1=1.68 x2=1.62 x3=1.58 x8=1.52 x9=1.52 correct to two rounded decimals choose x0=1.50 (observe what is happening) x1=1.50 x2=1.51 The form x=ln(3x) is suitable for r2. To find r1, we should find a different g(x). Example x2xsin4.01xf , Roots of f(x)=?

1x2xsin4.00x2xsin4.010xf Wish to find root r, correct to 2 decimal places, by LIM 0xf

xsin4.01x20x2xsin4.01 xsin2.05.0x

xsin2.05.0xg and xcos2.0xg

xallfor12.0xcos2.0xg then g(x) is suitable for arbitrary x0

Choosing x0=0.00 x1= g(x0) =0.50 x2= g(x1) =0.60 x3=0.61 x4=0.61 then r0.61=x4 and

244 105.0rxe

x

y

-1r

y=2x-1

y=0.4sin(x)

Now we like to find the number of additional iterations which assumes for 4 decimal place accuracy.

4

m

4m4

4

2

4556

44445

ii1i

exge

exgege

exgege

ege

where m is the number of additional iterations. Wish to find m so that 4

m4 105.0e

2m

4m4 105.0xge and 4

m4 105.0e then 42m

4 105.0105.0xg

2m

4

42m

4

10xg

105.0105.0xg

1639.061.0cos2.0xgxcos2.0xg 4

10ln21639.0lnm101639.0 2m 3m54.2m

x4=0.61 x5=g(x4) x5= 0.6146 similarly x6=0.6153

x7=0.6154 x8=0.6154

Newton's Method (NM)

f(x)=0 xi : initial approach hi: correction so that xi+1 =xi+hi

For large values of i, r x 1i and 0 )f(xf(r) 1i f(xi+hi)=0, wish to find h so that the equation is satisfied. Taylor expansion of f(xi+hi) about xi:

T.O.H

i

2

iiiii xf!2

hxfhxfhxf

i

Assuming h is small, f(xi+hi)=0 and neglecting Higher Order Terms (H.O.T)

i

iiiii xf

xfh0xfhxf

xi+1 =xi+hi

i

ii1i

h

i

ii1i xf

xfxx

xf

xfxx

i

where xi+1 is the next approximate value for r. i starts from zero

1

112

0

001

xf

xfxx

xf

xfxx

i

ii1i xf

xfxx

(i=0,1,2...) Recursive formula for the Newton’s Method.

Claim: If {x0,x1,x2,...} generated by recursive formula of Newton's Method converges to , then is a root of f.

Proof:

For enough large values of i, αi

x1i

x

f

f

xf

xfxlimxlim

i

ii

i1i

i

0f0fIf rootais

Since i1i

i

ii1i xgx

xf

xfxx

is the same form as that of LIM. But in NM, g(xi) has

the specific form: i

iii xf

xfxxg

In NM

Axf

xfxxgx

xf

xfxxg

(A) can be derived in the following way too

xf

fx0

f

f0

f

f0f

. By indexing

i

ii1i xf

xfxx

Or geometrically from the graph by writing slope of straight line tangent to the function f(x) at ix

Red line is tangent to the curve at xi.

i1ii

i xfxx

xf)tan(

i

ii1i xf

xfxx

Convergence Condition for NM

2xf

xfxfxfxf1xg

xf

xfxxg

NMforxf

xfxfxg

xf

xfxf11xg 22

if r is a simple root and derivative of f at x=r is different than zero then,

0rf

rfrfrg 2

Since r is a simple root and 0rf . This implies that

Always there is an interval I containing the root r in which |g'|<1 NM will produce always convergent sequence provided that x0 is in I (that is

sufficiently close to r)

Pi

Pi-1

xi+1 xi

x

a b

I

r x0

f(x)

Example

f(x)=ex-3x=0

wish to find r1and r2 by NM

i

ii1i xf

xfxx

i=0,1,2,...

3exfx3exf xx

3

31

i

i

xi

x

ii e

xexx 00.00 x

3

0*30

3

30

00

01 0

0

e

e

e

xexx

x

x

=0.5,

3

5.0*30

3

35.0

5.01

12 1

1

e

e

e

xexx

x

x

=0.61

62.0

62.0

62.0

61.0

50.0

00.0

1

4

3

2

1

0

r

x

x

x

x

x

54.1r

54.1x

54.1x

68.1x

00.2x

2

3

2

1

0

Rate of Convergence of NM

i

iii1i xf

xfxxgx

(I)

rgr (II)

(I)-(II) rgxgrx i1i (III) Taylor expansion of g(xi) about r

T.O.Hrxrg!3

1rxrg

!2

1rxrgrgxg 3

i2

iii

T.O.Hrxrg!3

1rxrg

!2

1rxrgrgxg 3

i2

iii (H.O.T stands for Higher Order Terms) And rewriting the Equation,

rgxgrx i1i

T.O.Hrxrg!3

1rxrg

!2

1rxrgrgxgrx 3

i2

iii1i

or

T.O.Hrxrg!3

1rxrg

!2

1rxrgrx 3

i2

ii1i

Where rxe 1i1i and rxe ii

Then error expression takes following form in general:

T.O.Herg!3

1erg

!2

1erge 3

i2ii1i

Where rg is zero in NM and 2ierg

!2

1 is dominating term, and rest of the terms may be

assumed as higher order terms and neglected, therefore error expression for NM is as follows:

2i1i erg

!2

1e Note that when i goes to infinity, rxx ii1i .

or

i

2i

ii g!2

rxrgrxrgxg

2ii1i rxg!2

1rx or 21 !2

1rx grx iii

2ii1i eg

!2

1e where i is between xi and r. Since ei has power of 2, NM converges

quadratically. So it is two times faster than LIM (since in linear iteration method the first term ierg is dominating term and the power of ei is 1 than means that the rate of convergence is

1 or linear.)

Now, it is shown that

Convergence of LIM is linear Convergence of NM is quadratic

Example

f(x)=ex-3x=0

LIM

3

ex

x

suitable for finding r1 in LIM

61.0

61.0

60.0

59.0

57.0

53.0

46.0

33.0

00.0 8

7

6

5

4

3

2

1

0

x

x

x

x

x

x

x

x

x

The root is found in 8 interations using LIM.

NM

62.0

62.061.0

50.0

00.0

4

3

2

1

0

x

xx

x

x

So 62.0xr 41 for two decimal place accuracy. The root is found in 4 iterations using NM.

Advantages of NM

xf

xfxxg

is given in NM

NM is suitable for finding all roots of f(x)=0

Example

2

2m

cr L

EIrP

rm is the absolutely smallest root of xtan(x)-6=0

x

6xtan06xtanxxf

20 mr

6tan xxxf x

xxxf

2costan and

i

iii xf

xfxx

1

Choose x0=1.2

x1=1.449

x2=1.394

x3=1.350 rm

x4=1.350

Convergence criteria

It may be shown that the sequence {xi}={x0,x1,x2...} generated by NM converges to r provided that x0 is chosen sufficiently close to r.

If ab)a(f

)a(f

and ab

)b(f

)b(f

and x0 is chosen

in [a,b], then the sequence {xi} generated by NM would converge to r

Example

f(x)=sin(x)-(x/2)2 . Chosen initial interval is [a,b]=[1.5,3]. 5.1ab64.0)5.1(f

)5.1(f

and

5.1ab84.0)3(f

)3(f

. Then, if x0 is chosen in [1.5,3], NM will converge to the root r.

Claim: The convergence of NM to a double root, if converges, would be linear.

Verification: Assume that r is a double root. Double root means that 0rf , and 0rf

but 0rf

NM xi+1 =xi+hi and i

ii xf

xfh

,

i

ii1i xf

xfxx

xf

xfxxg

And error expression is T.O.Herg!3

1erg

!2

1erge 3

i2ii1i

2xf

xfxfxfxf1xg

xf

xfxxg

b

a

b-a

|f(a)/f’(a)|

|f(b)/f’(b)|

r

y=f(x)

22 xf

xfxfxg

xf

xfxf11xg

rgxgLimrx

,

2rx2rx xf

xfLimrf

xf

xfxfLimrg

indefinite. Then by using

L'Hôpital's rule (LR)

xf*xf*2

xfLimrfrg

rx

= 2

1

rf2

1rf

T.O.Herg!3

1erg

!2

1erge 3

i2ii1i

i1i e2

1e errors at two successive iterations are related linearly, it means that the

convergence of NM to a double root is linear.

Claim: The convergence of the formula i

ii1i xf

xf*2xx

which is called modified NM-1

(MNM-1) to a double root would be quadratic; but the convergence of this formula to a simple root would be linear. Verify it!

Claim: But the convergence of the formula ii1i hxx with i

ii xu

xuh

and

i

ii xf

xfxu

which is called modified NM-2 (MNM-2) to a double root or simple root

would be quadratic. Summary:

i

ii1i xf

xfxx

Regular NM

i

ii1i xf

xf*2xx

Modified NM-1

i

ii1i xu

xuxx

,

i

ii xf

xfxu

Modified NM-2 Simple Root

Quadratic Linear Quadratic

Double Root

Linear Quadratic Quadratic

In general, for multiple roots: f(x)=0 and assume that r is a multiple root with the multiplicity M, there are M roots at x=r, i.e 0rf , 0rf , ..., 0rf 1M and 0rf M .

It can be written that )x(h)rx(xf M , where 0rh .

Claim: Then the convergence of the formula i

ii1i xf

xf*Mxx

to a multiple root r with

the multiplicity M would be quadratic but to a simple root it would be linear. And the error

relation is i1i eM

1Me

. Verify!

SECANT METHOD: Iterative formula could be derived geometrically by using following graph

Slope of straight line between ix and 1ix is

i1i

i1i

xx

xfxf

And between 1ix and ix is

1ii

i

xx

xf

.

Then by equating those slopes:

i1i

i1i

xx

xfxf

=

1ii

i

xx

xf

Then 1ii

1iii1i1i xfxf

xfxxfxx

.

To use the formula, two initial approximations or guesses which are 0x and 1x should be

considered . Iteration starts with 0x and 1x and result of the first iteration will be

01

01102 xfxf

xfxxfxx

And then second, third, fourth etc...

12

12213 xfxf

xfxxfxx

.

.

f(xi-1)

xi+1 xi xi-1

f(xi)

Example

f(x)=1+0.4sin(x)-2x=0

0.4sin(x)=2x-1

choose x0=0, x1=1 and 01

01102 xfxf

xfxxfxx

x2=0.60

x3=0.62

x4=0.62 r