Post on 04-Jan-2016
description
Nonlinear Equations
Jyun-Ming Chen
2
Contents
• Bisection
• False Position
• Newton
• Quasi-Newton
• Inverse Interpolation
• Method Comparison
Fall 2015
3
Solve the Problem Numerically
• Consider the problem in the following general form:
f(x) = 0
• Many methods to choose from:– Interval Bisection
Method
– Newton
– Secant
– …
Fall 2015
4
Interval Bisection Method
• Recall the following theorem from calculus
• Intermediate Value Theorem ( 中間值定理 )– If f(x) is continuous on [a,b]
and k is a constant, lies between f(a) and f(b), then there is a value x[a,b] such that
f(x) = kFall 2015
5
Bisection Method (cont)
• Simply setting k = 0
• Observe:– if sign( f(a) ) ≠ sign( f(b) )– then there is a point x [a, b] such that f(x) = 0
Fall 2015
6
Definition
• non-trivial interval [a,b]:f(a) ≠ 0, f(b) ≠ 0
and
sign( f(a) ) ≠ sign( f(b) )
sign(-2) = -1
sign(+5) = 1
Fall 2015
7
Idea
• Start with a non-trivial interval [a,b]
• Set c(a+b)/2
• Three possible cases:
⑴ f(c) ＝ 0, solution found
⑵ f(c) ≠ 0, [c,b] nontrivial
⑶ f(c) ≠ 0, [a,c] nontrivial
• Keep shrinking the interval until convergence
• → ⑴ problem solved• → ⑵⑶ a new smaller
nontrivial interval ½ size_______
Fall 2015
8
Algorithm
What’s wrong with this code?
Fall 2015
9
Remarks
• Convergence– Guaranteed once a nontrivial interval is found
• Convergence Rate– A quantitative measure of how fast the
algorithm is– An important characteristics for comparing
algorithms
Fall 2015
10
Convergence Rate of Bisection
• Let: – Length of initial
interval L0
– After k iterations, length of interval is Lk
– Lk=L0/2k
– Algorithm stops when Lk eps
• Plug in some values…
93.1910
1log
10
1Let
62
6
k
eps
L
This is quite slow, compared to other
methods…Meaning of
eps Fall 2015
11
How to get initial (nontrivial) interval [a,b] ?
• Hint from the physical problem
• For polynomial equation, the following theorem is applicable:
roots (real and complex) of the polynomial
f(x) = anxn + an-1xn-1 +…+ a1x + aο
satisfy the bound:
) , , , (1
1 10 nn
aaaMaxa
x ) , , , (1
1 10 nn
aaaMaxa
x
Fall 2015
12
Example
• Roots are bounded by
• Hence, real roots are in [-10,10]
• Roots are
–1.5251,
2.2626 ± 0.8844i
093 23 xxx109) 1, 3, ,1( max
1
11 x
complex
Fall 2015
13
Other Theorems for Polynomial Equations
• Sturm theorem: – The number of real roots of an algebraic
equation with real coefficients whose real roots are simple over an interval, the endpoints of which are not roots, is equal to the difference between the number of sign changes of the Sturm chains formed for the interval ends.
Fall 2015
14
Sturm Chain
Fall 2015
15
Example
38879.1 ,32836.10802951.0 ,334734.0 ,21465.1
13)( 5
ix
xxxf
Fall 2015
16
Sturm Theorem (cont)
• For roots with multiplicity:– The theorem does not apply, but …– The new equation : f(x)/gcd(f(x),f’(x))
• All roots are simple
• All roots are same as f(x)
Fall 2015
17
Sturm Chain by Maxima
Fall 2015
18
Maxima (cont)
Fall 2015
19
Descarte’s Sign Rule• A method of determining the
maximum number of positive and negative real roots of a polynomial.
• For positive roots, start with the sign of the coefficient of the lowest power. Count the number of sign changes n as you proceed from the lowest to the highest power (ignoring powers which do not appear). Then n is the maximum number of positive roots.
• For negative roots, starting with a polynomial f(x), write a new polynomial f(-x) with the signs of all odd powers reversed, while leaving the signs of the even powers unchanged. Then proceed as before to count the number of sign changes n. Then n is the maximum number of
negative roots.
3 positive roots
4 negative roots Fall 2015
20
False Position Method
• x2 defined as the intersection of x axis and x0f0-x1f1
• Choose [x0,x2] or [x2,x1], whichever is non-trivial
• Continue in the same way as bisection
• Compared to bisection:x2=(x1+x0)/2
Fall 2015
21
False Position (cont)
Determine intersection point
• Using similar triangles:
)(
)11
(
1
1
0
0
01
102
1
1
0
0
102
1
21
0
02
f
x
f
x
ff
ffx
f
x
f
x
ffx
f
xx
f
xx
)(1
011001
2 fxfxff
x
)(1
011001
2 fxfxff
x
Fall 2015
22
False Position (cont)
Alternatively, the straight line passing thru (x0,f0) and (x1,f1)
Intersection: simply set y=0 to get x
)( 001
010 xx
xx
fffy
0001
01
001
010 )(0
xxfff
xx
xxxx
fff
Fall 2015
23
Example
0 ,1 ,0
0sin3)(
1010
ffxx
exxxf x
k xk (Bisection) fk xk (False
Position)
fk
1 0.5 0.471
2 0.25 0.372
3 0.375 0.362
4 0.3125 0.360
5 0.34315 -0.042 0.360 2.93×10-5
Fall 2015
24
False Position
• Always better than bisection?
Fall 2015
25
(x0, f0)
Newton’s Method
tangent line thru (x0 , f0)
00 )( slope fxf
)(
)0 (
axis-on with intersecti
)(
0100
000
xxff
yset
x
xxffy
,...3,2,1 ,1
kf
fxx
k
kkk
,...3,2,1 ,1
kf
fxx
k
kkk
Graphical Derivation
Also known as Newton-Raphson method
Fall 2015
26
Newton’s Method (cont)
• Derived using Taylor’s expansion
f(x)
))(x-x(xf)f(x(x)f
)(xfxx
xxxf
xxxfxfxf
ofion approximat good a is
ˆthen
large not too and near is if
)(2
)())(()()(
000
00
20
0000
Fall 2015
27
Taylor’s Expansion (cont)
0)(ˆ
0)(for iteratenext theas
0)(ˆ ofroot theTake
xf
xf
xf
,...3,2,1 ,1
kf
fxx
k
kkk
,...3,2,1 ,1
kf
fxx
k
kkk
Fall 2015
28
Example
• Old Barbarians used the following formula to compute the square root of a number a
explain why this works:
)(2
11
kkk x
axx
8-84
4-025.11
3
2-2
1-1
0
104.7 107.41
103 1.00030 )025.1(2
1
102.5 025.1)25.1
125.1(
2
1
102.5 25.1)2
12(
2
1
1 2
:Error
x
x
x
x
x
Finding square root of 1 (a=1)with x0 = 2
Fall 2015
29
Newton’s Method
)(2
1 ...
)22
( 2
2)(
)(
)(
)(
0)( solve toMethod sNewton' Use
0)( of roots theof one is
22
1
2
1
2
kk
kk
kk
k
kkk
k
kkk
x
ax
x
a
x
xx
x
axxx
xxf
axxf
xf
xfxx
xf
axxfa
01)( 2 xxf
Fall 2015
30
Definition
• Error of the ith iterate
• Order of a method m, satisfies
where Ek is an error bound of k
)lim (i.e.
valueconverged theis where
αx
x
ii
ii
constantlim 1
k
mk
k
Ε
Ε constantlim 1
k
mk
k
Ε
Ε
Fall 2015
31
Linear Convergence of Bisection
root
a0
L2
L1
L0
a1 b1
a2 b2
b0
2
2
1222
0111
0
LabL
LabL
abL
22
is bounderror The
2 isroot of approx. reasonable a
,],[With
000
00
00
Lab
ba
ba
22
is bounderror The
2 isroot of approx. reasonable a
,],[With
000
00
00
Lab
ba
ba
Fall 2015
32
/2
2
1limor
2
1 /2
/2
22
11
1
211
00
L
E
E
Ε
ΕL
L
k
k
k
Linear Convergence of Bisection (cont)
• We say the order of bisection method is one, or the method has linear convergence
Fall 2015
33
Quadratic Convergence of Newton
• Let x* be the converged solution
• Recall
)()()(
)(2
1)()()( 2
xfxfxf
xfxfxfxf
)(
)(1
k
kkk xf
xfxx
Fall 2015
34
Quadratic Convergence of Newton (cont)
• Subtracting x*:
21
2
2
1
1
)(
)(
2
1
)(
)(
2
1
)(
)(21
)()(
)(
)(
kk
kkk
kk
kk
k
kkk
xf
xf
xf
xf
xf
xfxfxf
xf
xfxxxx
Or we say Newton’s method has quadratic convergenceFall 2015
35
Example: Newton’s Method
• f(x)= x3–3x2 – x+9=0
10)9131max(1
1
9,1,3,1
]10,10[ thmRecall
0123
,,,x
aaaa
xk xk
0 0
1 9
2 6.41
46 -1.5250
163)(
93)(
0 choose
2
23
0
xxxf
xxxxf
x
Worse than bisection !?Fall 2015
36
Why?
• plot f(x) • Plot xk vs. k
-1.525k
xk 60
5
30
10
-1.525
-10 25 35 40
Fall 2015
37
Newton Iteration
-20
0
20
40
60
80
100
0 10 20 30 40 50 60 70
k
xk 1數列
Fall 2015
38
Case 1:
Fall 2015
39
Case 2:
Diverge to
Fall 2015
40
Recall Quadratic Convergence of Newton’s
• The previous example showed the importance of initial guess x0
• If you have a good x0, will you always get quadratic convergence?– The problem of multiple-root
21 )(
)(
2
1kk xf
xf
Fall 2015
41
Example• f(x)=(x+1)3=0• Convergence is linear near multiple roots
Prove this!!
Fall 2015
42
Multiple Root
• If x* is a root of f(x)=0, then (x-x*) can be factored out of f(x)– f(x) = (x-x*) g(x)
• For multiple roots:– f(x) = (x-x*)k g(x) – k>1 and g(x) has no factor of (x-x*)
Fall 2015
43
Multiple Root (cont)
0][*)*(*)(:1
0*)(*)(
)(*)()()(:1
)](*)()([*)(
)(*)()(*)()(
1
1
1
k
k
kk
xxxfk
xgxf
xgxxxgxfk
xgxxxkgxx
xgxxxgxxkxf
Implication:Fall 2015
44
• where k is the multiplicity of the root
• Get quadratic convergence!
• Problem: do not know k in advance!
Remedies for Multiple Roots
)(
)(1
n
nnn xf
xfkxx
Fall 2015
45
Modified Newton’s Method
0(x) ofroot thealso is 0(x) ofroot the
:Check
0)( ofroot thefind tomethod sNewton' use
)(
)()(function new a Define
)(*)()(*)()(
)(*)()( ofty multiplici1
fF
xF
xf
xfxF
xgxxxgxxkxf
xgxxxfkkk
k
Fall 2015
46
Modified Newton’s Method (cont)
)(*)()(
)(*)(
)(*)()(*)(
)(*)(
)(
)()(
) converge alwaysNewton (hence,
roots multiple no has 0)(
1
xgxxxkg
xgxx
xgxxxgxxk
xgxx
xf
xfxF
llyquadratica
xF
kk
k
Fall 2015
47
Examplef(x)=(x–1)3sin((x – 1)2)
Fall 2015
48
Quasi-Newton’s Method
• Recall Newton:
• The denominator requires derivation and extra coding
• The derivative might not explicitly available (e.g., tabulated data)
• May be too time-consuming to compute in higher dimensions
)(
)(1
k
kkk xf
xfxx
Fall 2015
49
Quasi-Newton (cont)
• Quasi:
• where gk is a good and easily computed approx. to f’(xk)
• The convergence rate is usually inferior to that of Newton’s
k
kkk g
xfxx
)(1
Fall 2015
50
Secant Method
– Use the slope of secant to replace the slope of tangent
– Need two points to start
)()()(
)(
Or,
)()(
11
1
1
1
kkkk
kkk
kk
kkk
xxxfxf
xfxx
xx
xfxfg
Order: 1.62
Fall 2015
51
Idea:
• x2: Intersection of x-axis and a line interpolating x0 f0 & x1 f1
• x3: Intersection of x-axis and a line interpolating x1 f1 & x2 f2
• xk+1: Intersection of x-axis and a line interpolating xk-1fk-1 & xkfk
x0x1
‧
‧
‧‧
x2
Fall 2015
52
Comparison
• Newton’s method • False Position
(Newton)
xkxk+1
‧
‧‧‧
f ’(xk)
‧
Fall 2015
53
Secant vs. False Position
False PositionSecantFall 2015
54
Beyond Linear Approximations
• Both secant and Newton use linear approximations• Higher order approximation yields better accuracy?• Try to fit a quadratic polynomial g(x) thru the
following three points:g(xi) = f(xi), i = k, k–1, k – 2
• Let xk+1 be the root of g(x) = 0– Could have two roots; choose the one near xk
• This is called the Muller's Method
Fall 2015
55
• See Textbook
• g(x) 通過 (xk-2, fk-2), (xk-1, fk-1), (xk,fk)
Muller's MethodOrder: 1.84
Fall 2015
56
Finding the Interpolating Quadratic Polynomial g(x)
kkkk
kkkk
kkkk
faxaxaxg
faxaxaxg
faxaxaxg
axaxaxg
012
2
10112
121
20212
222
012
2
)(
)(
)(
)(
3 eqns to solveunknowns ： a2 , a1 , a0
))((1
)()( 121
21
1
1
21
1
kkkk
kk
kk
kk
kkk
kk
kkk xxxx
xx
ff
xx
ff
xxxx
xx
fffxg ))((
1)()( 1
21
21
1
1
21
1
kkkk
kk
kk
kk
kkk
kk
kkk xxxx
xx
ff
xx
ff
xxxx
xx
fffxg
Or,
Double-check !
Fall 2015
57
SummaryIterative Methods for Solving f(x)=0• Basic Idea:
– Local approximation + iterative computation
– At kth step, construct a polynomial p(x) of degree n, then solve p(x) = 0; take one of the roots as the next iterate, xk+1
• In other words,– construct p(x)
– solve p(x) = 0; find the intersection between y=p(x) and x-axis
– choose one root
Fall 2015
58
Revisit Newton
‧
‧xk+1 xk
))(()()(
ofroot theis
)(
)(
1
1
kkk
k
k
kkk
xxxfxfxp
x
xf
xfxx
p(x): is a linear approximation passing thru(xk,fk) with the slope fk
‘
Fall 2015
59
Revisit Secant
p(x)
)()(1
1k
kk
kkk xx
xx
fffxp
p(x): is a linear approximation
passing thru (xk-1,fk-1) and (xk,fk) with the secant slope
xk-1xk
‧
‧
‧‧
Fall 2015
60
Revisit Muller
• p(x) is a parabola (2nd degree approximation) passing thru three points
• Heuristic: choose the root that is closer to the previous iterate
• Potential problem:– No solution (parabola and x-axis do not
intersect!)
Fall 2015
61
Categorize by Starting Condition
• Bisection and False Position– Require non-trivial
interval [a,b]
– Convergence guaranteed
• Newton: one point – x0 → x1 →…
• Secant: two points– x0 x1 → x2 → …
• Muller: three points – x0 x1 x2 → x3→ …
• These methods converge faster but can diverge …
Fall 2015
62
A Slightly Different Method:Inverse Interpolation
• Basic Idea (still the same)– Local approximation + iterative computation
• Method:– At kth step, construct a polynomial g(y) of
degree n; then compute the next iterate by setting g(y = 0):
)0(1 ygxk
Fall 2015
63
Inverse Linear Interpolation
• Secant: Inverse linear Interpolation
‧
‧‧
(xk-1, fk-1)
(xk, fk)
(xk+1, fk+1)x
y
)(
)(
1
11
1
1
1
1
kkk
kkkk
kkk
kkk
kk
kk
k
k
fff
xxxx
fyff
xxxx
xx
ff
xx
fy
x = g(y), xk+1=g(0)
Fall 2015
64
Inverse Quadratic Interpolation
• Find another parabola: x = g(y)
• Set the next iteratexk+1 = g(0)
))((1
)()( 121
21
1
1
21
1
kkkk
kk
kk
kk
kkk
kk
kkk fyfy
ff
xx
ff
xx
fffy
ff
xxxyg ))((
1)()( 1
21
21
1
1
21
1
kkkk
kk
kk
kk
kkk
kk
kkk fyfy
ff
xx
ff
xx
fffy
ff
xxxyg
Fall 2015
65
Example (IQI)
• Solve f(x)=x3–x=0– x0 = 2, x1 = 1.2, x2 = 0.5
k xk+1 xk-2 fk-2 xk-1 fk-1 xk fk
1 0.8102335181319 2 6 1.2 0.528 0.5 -0.3752 1.3884057934643 1.2 0.528 0.5 -0.375 0.810234 -0.2783333 1.5252259894989 0.5 -0.375 0.810234 -0.27833 1.388406 1.2879834 0.9414762279683 0.810234 -0.27833 1.388406 1.287983 1.525226 2.0229295 0.9844320642825 1.388406 1.287983 1.525226 2.022929 0.941476 -0.1069736 1.0010409722070 1.525226 2.022929 0.941476 -0.10697 0.984432 -0.0304137 1.0000043435326 0.941476 -0.10697 0.984432 -0.03041 1.001041 0.0020858 0.9999999997110 0.984432 -0.03041 1.001041 0.002085 1.000004 8.69E-069 1.0000000000000 1.001041 0.002085 1.000004 8.69E-06 1 -5.78E-10
Fall 2015
66
Professional Root Finder
• Need guaranteed convergence and high convergence rate
• Combine bisection and Newton (or inverse quadratic interpolation)– Perform Newton step whenever possible
(convergence is achieved)– If diverge, switch to bisection
Fall 2015
67
Brent’s Method
• Guaranteed to converge• Combine root bracketing, bisection and
inverse quadratic interpolation– van Wijngaarden-Dekker-Brent method– Zbrent in NR
• Brent uses the similar idea in one-dimensional optimization problem– Brent in NR
Fall 2015
GSL Library
68Fall 2015
69
Fast Inverse Square Root
To understand the magic number 0x5f3759df, read: Chris Lomont or Paul Hsieh
Fall 2015