CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the...
Transcript of CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the...
![Page 1: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/1.jpg)
CS321
Numerical Analysisand Computing
Lecture 2
Locating Roots of Equations
Professor Jun Zhang
Department of Computer ScienceUniversity of Kentucky
Lexington, KY 40506-0633
![Page 2: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/2.jpg)
2
What is the Root
Many physical system can be written in the form of an equation
This equation represents the relationship between the dependent variables and independent variables
The root of a nonlinear equation is the solution of that equation
It can also be said to be the solution of a nonlinear system
Most nonlinear equations are too complicated to have an analytical solution
In practice, we are more interested in finding some numerical solutions
explicit numbers, approximate solutions
![Page 3: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/3.jpg)
Roots (Zeros)of a Function
3
![Page 4: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/4.jpg)
Roots (Zeros)of a Function
4
![Page 5: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/5.jpg)
Roots of a Functions
5
Let f(x) be a function that has values of opposite signs at the two ends of a given interval [a,b] with a < b, i.e., f(a)•f(b) < 0. If f(x) is continuous on [a,b], then there exists a number c in [a,b] such that f(c) = 0
c is called a root of function f(x) = 0
Example. The function
has a root in the interval [0,2]. It has two roots in the interval [-5,2]
Remark: roots are not necessarily unique in a given interval
Need some root finding algorithms for general functions
032)( 2 xxxf
![Page 6: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/6.jpg)
A Root of a Function
6
![Page 7: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/7.jpg)
A Function with Four Roots
7
![Page 8: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/8.jpg)
Bisection Method
8
Given an interval [a,b] and a continuous function f(x), if f(a)•f(b) < 0, then f(x) must have a root in [a,b]. How to find it?
We suppose f(a) > 0 and f(b) < 0
Step 1. Compute the midpoint stop if is small, and take c as the root
Step 2. Evaluate f(c), if f(c) = 0, a root is found
Step 3. If f(c) ≠ 0, then either f(c) > 0 or f(c) < 0
Step 4. If f(c) < 0, a root must be in [a,c]
Step 5. Let go to Step 1.
,2
abc
2
ab
),()(, cfbfcb
![Page 9: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/9.jpg)
Bisection Process
9
![Page 10: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/10.jpg)
Bisection Process
10
Find a root of the function ଶ
![Page 11: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/11.jpg)
Bisection Process
11
![Page 12: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/12.jpg)
Convergence Analysis
12
Let r be a root of f(x) in the interval [a0, b0]. Let c0 = be the midpoint, then
If we use the bisection algorithm, we compute and have a0, b0, c0, a1, b1, c1, …, then
Since the interval length is halved at each step, we have
Hence
which is the maximum error if we take cn as an approximate to the root r
200 ba
200
0ab
cr
)2(2
nab
cr nnn
nnn
nnabab
ab22
0011
100
2
nn
abcr
![Page 13: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/13.jpg)
Linear Convergence
13
A sequence {xn} has linear convergence to a limit x if there exists a constant C in the interval [0,1) such that
By recursion, we have
Or equivalently, a linear convergence satisfies
For some positive number A
The bisection algorithm has a linear convergence rate with C = ½ andA = (b0 - a0)/2
)1(1 nxxCxx nn
xxC
xxCxxCxxn
nnn
1
12
1
)10(1 CACxx nn
![Page 14: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/14.jpg)
Stopping Criterion
14
What is our goal? When to stop? How many iterations?
Our goal is to find r Є [a,b] such that f(r) = 0
With the bisection algorithm, we generate a sequence such that
for some prescribed number
i.e., we find a point cn inside the interval [a,b] that is very close to the root r. We then use cn as an approximate to r
It is not guaranteed, however, that f(cn) is very close to 0
|| ncr
0
![Page 15: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/15.jpg)
How Many Iterations
15
If we want the approximate root cn is close to the true root r, i.e., we want
Then the number of bisection steps n satisfies
Or
Example. Find a root in [16,17] up to machine single precision
a = (10 000.0)2, b = (10 001.0)2 so r must have a binary formr = (10 000.***…)2. We have a total of 24 bits, 5 is already fixed. The accuracy will be up to another 19 bits, which is between 2-19 and 2-20. We choose є = 2-20. Since b – a = 1, we need 2n+1 > 220, yielding n ≥ 20
,cn r
12n
ab
2log)2log()log(
ab
n
![Page 16: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/16.jpg)
Newton’s Method
16
Given a function f(x) and a point x0, if we know the derivative of f(x) at x0, we can construct a linear function that passes through (x0, f(x0)) with a slope f’(x0) ≠ 0 as
Since l(x) is close to f(x) at x0, if x0 is close to r, we can use the root of l(x) as an approximate to r, the root of f(x)
x1 may not be close to r enough, we repeat the procedure to find
Under certain conditions, {xn} converges to r
)())((')( 000 xfxxxfxl
)('
)(
0
001 xf
xfxx
,)('
)(
1
112 xf
xfxx
![Page 17: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/17.jpg)
Newton’s Method
17
![Page 18: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/18.jpg)
From Taylor Series
18
If f(x0) ≠ 0, but x0 is close to r, we may assume that they differ by h, i.e., x0 + h = r, or
Using Taylor series expansions
Ignoring the higher order terms, we have
Or
Since h does not satisfy f(x0 + h) = 0, we use
as an approximate to r, and repeat the process
0)()( 0 rfhxf
0)(')( 00 xhfxf
0)("2
)(')( 0
2
00 xfh
xhfxf
)('
)(
0
0
xf
xfh
)('
)(
0
0001 xf
xfxhxx
![Page 19: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/19.jpg)
First Few Approximations
19
![Page 20: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/20.jpg)
Fast Convergence
20
Find a root for the following function f(x), starting at x0 = 4
143)('
32)(2
23
xxxf
xxxxf
Each iteration gains double digits of accuracy and f(xn) decreases quadratically to 0
![Page 21: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/21.jpg)
Example
21
![Page 22: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/22.jpg)
Convergence Analysis
22
Let the function f have continuous first and second derivatives f’ and f”, and r be a simple root of f with f’(r) ≠ 0. If x0 is sufficiently close to r, then Newton’s method converges to r quadratically.
If xn differs from r by at most one unit in the kth decimal place, i.e.,
Then, for c = 1, we have
The number of correct decimal digits doubles after another iteration
2
1 rxcrx nn
kn rx 10
kn rx 2
1 10
![Page 23: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/23.jpg)
23
Convergence Proof
Let en = r – xn. Newton’s method generates a sequence {xn} such that
Using Taylor’s expansion, there exists a point between xn and r for which
It follows that
)('
)()('
)('
)(
)('
)(11
n
nnn
n
nn
n
nnnn
xf
xfxfe
xf
xfe
xf
xfxrxre
)("21
)(')(
)()(02
nnnnn
nn
fexfexf
exfrf
)("2
1)()(' 2
nnnnn fexfxfe
![Page 24: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/24.jpg)
24
Convergence Proof Cont.
We thus have
Define an upper bound
We can choose δ small so that
This is to guarantee that xn is close to r within a distance of δ
21 )('
)("
21
nn
nn e
xf
fe
)0(,)('min
)("max
2
1)(
||
||
xf
xfc
rx
rx
randδrxe nnn
![Page 25: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/25.jpg)
25
Convergence Proof Cont.
For very small δ > 0, we have
With ρ = δc(δ) < 1 if δ is small enough, therefore
xn+1 is also close to r within a distance of δ. By recursion, if x0 is close to r, then
Since ρ < 1, this is to say
nn
nnn
nn
eec
ecexf
fe
)(
)()('
)("
21 22
1
nnnn eeerx 11
022
1 eeee nnnn
nasenn
0lim
![Page 26: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/26.jpg)
26
Weakness of Newton’s Method
Newton’s method converges fast, only when x0 is chosen close to r. In practice, there might also be a number of problems
1.) needs derivative value and availability
2.) starting point must be close to r
3.) lose quadratic convergence if multiple root
4.) iterates may run away (not in convergence domain)
5.) flat spot with f ’(xn) = 0
6.) cycling iterates around r
![Page 27: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/27.jpg)
Problems of Newton’s Method
27
![Page 28: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/28.jpg)
Newton’s Method Cycling
28
![Page 29: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/29.jpg)
29
Systems of Nonlinear Equations
Newton’s method is really useful for finding zero of a system of nonlinear equations
Written in vector form asf(x) = 0
Where
We have
f’(x(k)) = J(x(k)) is the Jacobian matrix
0),,,(
0),,,(
0),,,(
21
212
211
nn
n
n
xxxf
xxxf
xxxf
Tn
Tn
xxx
fff
),,,(x
),,,(f
21
21
)x(f)]x('f[xx )(1)()()1( kkkk
![Page 30: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/30.jpg)
30
A 3 Equation Example
Using Taylor expansion
Let be an approximate solution and the computed correction be . Hence
0),,(
0),,(
0),,(
3213
3212
3211
xxxf
xxxf
xxxf
33
22
11
321
332211
),,(
),,(
x
fh
x
fh
x
fh
xxxf
hxhxhxf
iii
i
i
Txxxx ),,( )0(3
)0(2
)0(1
)0( Thhhh ),,( 321
h)x('f)x(f)hx(f0 )0()0()0(
![Page 31: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/31.jpg)
31
Example Cont.
The Jacobian matrix is
It follows that
Hence, the new iterate is
In practice, we solve the Jacobian matrix in
So that
3
3
2
3
1
3
3
2
2
2
1
2
3
1
2
1
1
1
J
x
f
x
f
x
f
x
f
x
f
x
f
x
f
x
f
x
f
)x(f)](x-[f'h )0(1(0)
)x(f)]x('f[xx )0(1)0((0)(1)
)x(fh)]x(J[ )()()( kkk
)()()1( hxx kkk
![Page 32: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/32.jpg)
32
Secant Method
In Newton’s method
We need to evaluate f(xn) and f’(xn) at each iteration
We can approximate the derivative at x = xn by
Thus, the secant method generates iterates
Only one functional evaluation at each iteration
)('
)(1
n
nnn xf
xfxx
nn
nnn xx
xfxfxf
1
1 )()()('
)()()( 1
11 n
nn
nnnn xf
xfxf
xxxx
![Page 33: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/33.jpg)
Secant Method
33
![Page 34: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/34.jpg)
34
Comments
Secant method needs two iterates to start with, we can use bisection method to generate the second iterate
Secant method does not need to know the derivative of f(x)
If | f(xn) – f(xn-1)| is small, the computation may lose significant digits and becomes unstable
The convergence rate of the secant method is superlinear
With Its convergence rate is between that of the bisection method and the Newton’s method
nn eCe 1
.62.1)51(2
1
![Page 35: CS321 Numerical Analysis and Computingjzhang/CS321/lecture2.pdfConvergence Analysis 22 Let the function f have continuous first and second derivatives f’ and f”, and r be a simple](https://reader034.fdocuments.net/reader034/viewer/2022050400/5f7d9199ba7a5540f05fd1f4/html5/thumbnails/35.jpg)
35
Hybrid Approaches
In practice, hybrid methods are usually used
For example, we can use Bisection Method to generate initial iterates that are close to the root, so that Newton’s method can be applied
When the evaluation of derivative is expensive, the Secant Method should be used to replace the Newton’s method
The trade-offs between Bisection Method and Newton’s Method are robustness and fast convergence
The Secant Method is supposed to come between these two methods, to take the advantages of both, and avoid the disadvantages of either