Finding solutions by approximationvipul/studenttalks/approximatingsolutions.pdf · Finding...

Finding solutionsby approximation

Vipul Naik

The parallelbetween equationsand differentialequations

What it means tosolve an equation

We can showexistence evenwithout solving

Easy methods ofsolution

Bisection search

Interpolation

Qualitativecharacteristics of aroot-findingalgorithm

What is anumber-findingalgorithm?

Convergence issues

Issues of when to stop

Concept of aniterating map

Comparison ofbisection and linearinterpolation

A broad overview ofthe comparison

Value (one-point)iterationalgorithms

The idea

Finding candidates forthe fixed-pointfunction

Linear interpolationand theNewton-Raphsonmethod

Solving differentialequations

The problemstatement

Interval iteration doesnot adapt to thissetup

Adapting valueiteration

The contractionprinciple

First-degree first-orderdifferential equations

Finding solutions by approximation

Vipul Naik

March 26, 2007


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Outline

The parallel between equations and differential equationsWhat it means to solve an equationWe can show existence even without solving

Easy methods of solutionBisection searchInterpolation

Qualitative characteristics of a root-finding algorithmWhat is a number-finding algorithm?Convergence issuesIssues of when to stopConcept of an iterating map

Comparison of bisection and linear interpolationA broad overview of the comparison

Value (one-point) iteration algorithmsThe ideaFinding candidates for the fixed-point functionLinear interpolation and the Newton-Raphson method

Solving differential equationsThe problem statementInterval iteration does not adapt to this setupAdapting value iterationThe contraction principleFirst-degree first-order differential equations


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Finding zeroes of a function

Finding solutions of an “equation” boils down to finding thezeroes of a function. In other words, let f : R → R be a(reasonably well-behaved) function. Then our problem is todetermine x ∈ R for which:

f (x) = 0

Note that we thus have a map from:Functions from reals to reals → subsets of realswhich sends a function to its set of roots


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









The function could be on any space

Instead of looking at functions from R to R, we could lookat functions from Rn to R, or more generally from anymanifold to R (though we aren’t going to do this right now).With some suitable assumptions of good behaviour from thefunction, we may need to determine the set of roots of thefunction.

Even more generally, we could look at two spaces X and Y ,and a function f : X → Y , and we are interested indetermining f −1(y) for some y ∈ Y .


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









The function could be on any space

Instead of looking at functions from R to R, we could lookat functions from Rn to R, or more generally from anymanifold to R (though we aren’t going to do this right now).With some suitable assumptions of good behaviour from thefunction, we may need to determine the set of roots of thefunction.Even more generally, we could look at two spaces X and Y ,and a function f : X → Y , and we are interested indetermining f −1(y) for some y ∈ Y .


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Solving differential equations

We have an equation:

F (f ) ≡ 0

here f is a (not known) function from R to R and F is afunction involving f and its instantaneous derivatives. The≡ means that F (f ), evaluated at any point x , gives zero.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Solution in the sense of an explicit formula

For some, solving an equation (usual or differential) meansobtaining a “closed form” for the solutions. For instance,solving the ordinary equation:

x2 − 2x − 1 = 0

Means writing the solution explicitly as 1±√

2.

Of course, here the definition of “explicit” must be madeexplicit – what are the known functions in terms of whichthe solution is explicit? Do explicit solutions really makethings clear? For instance, is sin−1(234 + log π) really allthat explicit a solution? Are we allowed to write inversefunctions while being “explicit”? If we are, then we’ve foundexplicit solutions to any equation!


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Solution in the sense of an explicit formula

For some, solving an equation (usual or differential) meansobtaining a “closed form” for the solutions. For instance,solving the ordinary equation:

x2 − 2x − 1 = 0

Means writing the solution explicitly as 1±√

2.Of course, here the definition of “explicit” must be madeexplicit – what are the known functions in terms of whichthe solution is explicit? Do explicit solutions really makethings clear? For instance, is sin−1(234 + log π) really allthat explicit a solution? Are we allowed to write inversefunctions while being “explicit”? If we are, then we’ve foundexplicit solutions to any equation!


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Solution in the sense of a converging sequence

A more reliable approach towards describing the solutions isby giving a converging sequence of rational numbers (withthe denominators not too large) that converges reasonablyfast to the solution.

Rational numbers are fairly explicit; they are also easy tocompare and compute with, so such a solution may be mostconvenient computationally.Upshot: Approximate solutions may actually be better thanexact ones!


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










A more reliable approach towards describing the solutions isby giving a converging sequence of rational numbers (withthe denominators not too large) that converges reasonablyfast to the solution.Rational numbers are fairly explicit; they are also easy tocompare and compute with, so such a solution may be mostconvenient computationally.

Upshot: Approximate solutions may actually be better thanexact ones!


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










A more reliable approach towards describing the solutions isby giving a converging sequence of rational numbers (withthe denominators not too large) that converges reasonablyfast to the solution.Rational numbers are fairly explicit; they are also easy tocompare and compute with, so such a solution may be mostconvenient computationally.Upshot: Approximate solutions may actually be better thanexact ones!


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Functions with intermediate value property

A function f : R → R is said to have the intermediate valueproperty if for any a < b ∈ R, and any d that lies betweenf (a) and f (b), there exists c ∈ (a , b) such that f (c) = d .

Continuous functions satisfy the intermediate value property.For functions that satisfy the intermediate value property, wehave the following:If a < b and f (a) and f (b) have opposite signs, then there isa zero of f somewhere between a and b.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










A function f : R → R is said to have the intermediate valueproperty if for any a < b ∈ R, and any d that lies betweenf (a) and f (b), there exists c ∈ (a , b) such that f (c) = d .Continuous functions satisfy the intermediate value property.

For functions that satisfy the intermediate value property, wehave the following:If a < b and f (a) and f (b) have opposite signs, then there isa zero of f somewhere between a and b.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










A function f : R → R is said to have the intermediate valueproperty if for any a < b ∈ R, and any d that lies betweenf (a) and f (b), there exists c ∈ (a , b) such that f (c) = d .Continuous functions satisfy the intermediate value property.For functions that satisfy the intermediate value property, wehave the following:If a < b and f (a) and f (b) have opposite signs, then there isa zero of f somewhere between a and b.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Outline








Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









A precise problem formulation

We are given a function f : R → R in the form of a blackbox that takes any x and outputs f (x).

We may also be told some of the properties of f .We need to find a sequence of values an such thatlimn→∞ an is a zero of f .We would like to get a sequence with a high convergencerate and our algorithm should not make too many calls to f(ideally the number of calls to compute the sequence to nterms should be O(n)).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










We are given a function f : R → R in the form of a blackbox that takes any x and outputs f (x).We may also be told some of the properties of f .

We need to find a sequence of values an such thatlimn→∞ an is a zero of f .We would like to get a sequence with a high convergencerate and our algorithm should not make too many calls to f(ideally the number of calls to compute the sequence to nterms should be O(n)).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










We are given a function f : R → R in the form of a blackbox that takes any x and outputs f (x).We may also be told some of the properties of f .We need to find a sequence of values an such thatlimn→∞ an is a zero of f .

We would like to get a sequence with a high convergencerate and our algorithm should not make too many calls to f(ideally the number of calls to compute the sequence to nterms should be O(n)).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










We are given a function f : R → R in the form of a blackbox that takes any x and outputs f (x).We may also be told some of the properties of f .We need to find a sequence of values an such thatlimn→∞ an is a zero of f .We would like to get a sequence with a high convergencerate and our algorithm should not make too many calls to f(ideally the number of calls to compute the sequence to nterms should be O(n)).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Bisection search

Bisection search is a technique used to find a root of acontinuous function once we have got hold of values a and bat which the function takes opposite values. It is an iterativeprocess that outputs a sequence a1, a2, . . . such thatlimn→∞ an is a root.

Another interpretation is that the algorithm returns adecreasing sequence of intervals C0, C1, . . . , where each Ci

is half in size of Ci−1 such that the intersection of all the Ci sis a zero of the function. At each stage, the current Ci istermed the interval of uncertainty – it measures the extentto which we are uncertain about the exact value of the root.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Bisection search

Bisection search is a technique used to find a root of acontinuous function once we have got hold of values a and bat which the function takes opposite values. It is an iterativeprocess that outputs a sequence a1, a2, . . . such thatlimn→∞ an is a root.Another interpretation is that the algorithm returns adecreasing sequence of intervals C0, C1, . . . , where each Ci

is half in size of Ci−1 such that the intersection of all the Ci sis a zero of the function. At each stage, the current Ci istermed the interval of uncertainty – it measures the extentto which we are uncertain about the exact value of the root.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Algorithm for bisection search

At each iteration of the algorithm, we have two endpoints aand b such that f takes opposite values at a and b.Let c = (a + b)/2. There are three possibilities for f (c):

I f (c) = 0: In this case we have found a root

I f (c) has the same sign as f (a). In this case, we canconsider the interval (c , b)

I f (c) has the same sign as f (b). In this case, we canconsider the interval (a , c).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Convergence rate

The democratic nature of bisection search forces that theconvergence in the domain side to the root is fairly fast.This is because the size of the interval of uncertaintydecreases exponentially.

However, this does not guarantee that in the image,convergence will be fast. That is because without knowinganything about the function f , we cannot comment on howfast f (x) approaches f (x0).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Convergence rate

The democratic nature of bisection search forces that theconvergence in the domain side to the root is fairly fast.This is because the size of the interval of uncertaintydecreases exponentially.However, this does not guarantee that in the image,convergence will be fast. That is because without knowinganything about the function f , we cannot comment on howfast f (x) approaches f (x0).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









The secant method

This method tries to approximate the given function by alinear function. Like bisection search, it outputs adescending sequence of closed intervals whose intersection isa zero. However, while bisection search tries the midpoint,the secant method draws a line between (a, f (a)) and(b, f (b)) and determines the x-coordinate of the point wherethis line intersects the x-axis.

For the secant method we are not guaranteed a good rate ofconvergence in the domain; on the other hand, for niceenough functions, it’ll converge faster than bisection search.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









The secant method

This method tries to approximate the given function by alinear function. Like bisection search, it outputs adescending sequence of closed intervals whose intersection isa zero. However, while bisection search tries the midpoint,the secant method draws a line between (a, f (a)) and(b, f (b)) and determines the x-coordinate of the point wherethis line intersects the x-axis.For the secant method we are not guaranteed a good rate ofconvergence in the domain; on the other hand, for niceenough functions, it’ll converge faster than bisection search.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









A more general idea of interpolation

Let A be a class of functions from R to R , with theproperty that any function in A is uniquely determined by itsvalues at any n points (where n is a fixed number).

Now given any f : R → R, we can interpolate f by a functionfrom A as follows: pick n points of R, and find a function inA that takes the same value as f at these n points.For instance, polynomial functions of degree at most n areuniquely determined by their value at any n + 1 given points.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










Let A be a class of functions from R to R , with theproperty that any function in A is uniquely determined by itsvalues at any n points (where n is a fixed number).Now given any f : R → R, we can interpolate f by a functionfrom A as follows: pick n points of R, and find a function inA that takes the same value as f at these n points.

For instance, polynomial functions of degree at most n areuniquely determined by their value at any n + 1 given points.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










Let A be a class of functions from R to R , with theproperty that any function in A is uniquely determined by itsvalues at any n points (where n is a fixed number).Now given any f : R → R, we can interpolate f by a functionfrom A as follows: pick n points of R, and find a function inA that takes the same value as f at these n points.For instance, polynomial functions of degree at most n areuniquely determined by their value at any n + 1 given points.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Using interpolation to find roots

The idea behind interpolation is roughly as follows:

I Start off by taking the value of the function at n points

I Interpolate it to a polynomial of degree n − 1

I Find a zero of this polynomial (which is closest to the npoints)

I Now throw away one of the original n points (say theone at which the value was largest) and add the newpoint obtained as the zero of the polynomial.

I Iterate the procedure


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Outline








Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









What is a real number?

A real number is computationally described by a sequence ofrationals. The faster the sequence of rationals converges tothe real number, the better.

Thus, an algorithm that aims to output a real number(which is in this case a root of a give function) shouldproceed to do so by outputting a sequence of rationalnumbers that, in the limit, give the real number.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









What is a real number?

A real number is computationally described by a sequence ofrationals. The faster the sequence of rationals converges tothe real number, the better.Thus, an algorithm that aims to output a real number(which is in this case a root of a give function) shouldproceed to do so by outputting a sequence of rationalnumbers that, in the limit, give the real number.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Convergence rate of a sequence

Given a sequence of numbers a1, a2, . . . , converging to a, wedefine the order of convergence(defined) of the sequence asthe largest nonnegative p for which:

limn→∞

∣∣∣∣ an+1 − an

(an − an−1)p

∣∣∣∣ = β <∞

The value β is termed the order of convergence(defined).When p = 1 and β < 1, we say that we have linearconvergence(defined) or geometric convergence(defined).

When p = 1 and β = 0, or when p > 1, we say we havesuperlinear convergence(defined).When p = 2, we say we have quadratic convergence(defined).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea











limn→∞

∣∣∣∣ an+1 − an

(an − an−1)p

∣∣∣∣ = β <∞

The value β is termed the order of convergence(defined).When p = 1 and β < 1, we say that we have linearconvergence(defined) or geometric convergence(defined).When p = 1 and β = 0, or when p > 1, we say we havesuperlinear convergence(defined).

When p = 2, we say we have quadratic convergence(defined).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea











limn→∞

∣∣∣∣ an+1 − an

(an − an−1)p

∣∣∣∣ = β <∞

The value β is termed the order of convergence(defined).When p = 1 and β < 1, we say that we have linearconvergence(defined) or geometric convergence(defined).When p = 1 and β = 0, or when p > 1, we say we havesuperlinear convergence(defined).When p = 2, we say we have quadratic convergence(defined).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Convergence of an algorithm

An algorithm that intends to output a real number does soby outputting a sequence of rationals that converges to thatreal number. The convergence of the algorithm is measuredby the following parameters:

I The time taken to output the nth member of thesequence

I The complexity (in terms of size of numerator anddenominator) of the nth member

I The rate of convergence of the algorithm


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Caveats with convergence rate

The convergence rate as described here is an asymptoticconvergence rate. That is, the sequence will converge atthat rate once it reaches sufficiently close to the limit.

Thus, if we start very far off, the initial rate of convergencemay be too bad even if the asymptotic rate is good.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Caveats with convergence rate

The convergence rate as described here is an asymptoticconvergence rate. That is, the sequence will converge atthat rate once it reaches sufficiently close to the limit.Thus, if we start very far off, the initial rate of convergencemay be too bad even if the asymptotic rate is good.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Stop rules

Suppose we want to “compute” a real number, and for thepurpose, we have an algorithm that outputs a sequence thatconverges to the real number.A stop rule (or termination rule) for this algorithm is a rulethat terminates the computation of terms of the sequenceafter finitely many steps, in such a way that the term atwhich we stop is sufficiently close to the final limit.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Deciding when to stop without using the limit

Recall that for reals, a sequence is convergent if and only ifit is Cauchy, that is, if and only if the differences of termsbeyond a certain point go to zero.Thus, here are some possible stop rules:

I Stop when the difference of successive terms comesbelow a certain threshold ε.

I Stop when the ratio∣∣∣ xn−xn−1

xn−1−xn−2

∣∣∣ comes below a certain

threshold ε (this works well when we have superlinearconvergence).

I Do either of the above two replacing xn with f (xn)where f may be the function whose zero we are huntingfor.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Deciding when to stop using the limit

Suppose we need to find a zero of a given function f , andwe have an algorithm that outputs a sequence thatconverges to a zero. Then, a possible stop rule for thealgorithm is to stop when |f (x)| < ε.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Computing the sequence

Given a function f , we want to find a setup of this kind:

I A one-point iteration setup: Here, we start with ax ∈ R and g : R → R such that the sequence(x , g(x), . . . , ) converges to a zero of f .

I An interval iteration setup: Here we start with a closedinterval [a , b] ⊆ R and at each stage, we output asmaller interval contained in the given interval, with theproperty that at each stage, the interval contains theroot.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Generic iteration algorithm

I Generic one-point iteration setup: The input is afunction f . The algorithm computes a functiong = G (f ) (here G is independent of f ) and a startingvalue x0 (determined by trial and error) such that theiteration of g on x0 converges to a zero.

I Generic interval iteration setup: The input is a functionf . The algorithm computes a function g = G (f ) (hereG is independent of f ) and a starting closed interval[a , b] such that the sequence of intervals obtained byiteratively applying g , converges to the zero of f .


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Dependence on initial conditions

An important way of measuring a generic iteration algorithmis its dependence on the initial values we start with.

For instance, for a function f , and a starting interval [a , b]let n(f , ε, [a , b]) denote the number of steps we need toiterate the algorithm till we find an interval for which oneendpoint x satisfies |f (x)| < ε.Fixing f and ε, we would like to compute n as a function of[a , b]. In particular, we would like to know how n depends asa function of the length of a and b.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










An important way of measuring a generic iteration algorithmis its dependence on the initial values we start with.For instance, for a function f , and a starting interval [a , b]let n(f , ε, [a , b]) denote the number of steps we need toiterate the algorithm till we find an interval for which oneendpoint x satisfies |f (x)| < ε.

Fixing f and ε, we would like to compute n as a function of[a , b]. In particular, we would like to know how n depends asa function of the length of a and b.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










An important way of measuring a generic iteration algorithmis its dependence on the initial values we start with.For instance, for a function f , and a starting interval [a , b]let n(f , ε, [a , b]) denote the number of steps we need toiterate the algorithm till we find an interval for which oneendpoint x satisfies |f (x)| < ε.Fixing f and ε, we would like to compute n as a function of[a , b]. In particular, we would like to know how n depends asa function of the length of a and b.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Outline








Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Both are interval iteration algorithms

Both algorithms work as follows:

I At each stage of the iteration, we have a closed intervalsuch that the function has opposite signs at theendpoints of the closed interval.

I We pick a value in this interval and check the sign ofthe function at this value.

I We then find the endpoint such that the sign differs andwe pick the interval from this mid-value to thatendpoint.

The difference between bisection and linear interpolation isin the way the value inside is chosen.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









The performance of bisection search

Suppose that a function f has only one root and that thisroot lies in [0 , 1]. Then, the number of steps for [0 , 2n] is nplus the number of steps for [0 , 1].

Thus, for bisection search, starting off with a bigger intervalalters the time taken by a factor logarithmic in the size ofthe interval. While this is not too bad, this is not too goodeither.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









The performance of bisection search

Suppose that a function f has only one root and that thisroot lies in [0 , 1]. Then, the number of steps for [0 , 2n] is nplus the number of steps for [0 , 1].Thus, for bisection search, starting off with a bigger intervalalters the time taken by a factor logarithmic in the size ofthe interval. While this is not too bad, this is not too goodeither.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Performance of linear interpolation

If we have a function of the form:

f (x) = ax + o(x)

Then linear interpolation works very well to cut down the sizeof the interval to nearly a constant size in the very first step.

For instance, consider:

f (x) = x − cos x

The number of steps taken for this for a starting interval[0 , a] is almost constant for large values of a. For instancewhen ε = .001, then the number of steps taken is 5 for alla ≥ 10.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Performance of linear interpolation

If we have a function of the form:

f (x) = ax + o(x)

Then linear interpolation works very well to cut down the sizeof the interval to nearly a constant size in the very first step.For instance, consider:

f (x) = x − cos x

The number of steps taken for this for a starting interval[0 , a] is almost constant for large values of a. For instancewhen ε = .001, then the number of steps taken is 5 for alla ≥ 10.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Performance of linear interpolation (contd)

Linear interpolation can perform very badly if the function inquestion is superlinear. In fact, in this case, the number ofsteps taken can be polynomially dependent on the size of thenumber. For instance, consider:

f (x) = xd − 1

and the behaviour of this over the closed interval [0 , a] as avaries over positive reals greater than 1.

Since the function rises much faster than a linear function,the linear interpolation of the function gives a value veryclose to 0. Thus, at each stage of the interpolation, thevalue grows by a very small amount.In fact, at every stage, it grows by an amount roughly equalto a1−d .


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea











f (x) = xd − 1

and the behaviour of this over the closed interval [0 , a] as avaries over positive reals greater than 1.Since the function rises much faster than a linear function,the linear interpolation of the function gives a value veryclose to 0. Thus, at each stage of the interpolation, thevalue grows by a very small amount.

In fact, at every stage, it grows by an amount roughly equalto a1−d .


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea











f (x) = xd − 1

and the behaviour of this over the closed interval [0 , a] as avaries over positive reals greater than 1.Since the function rises much faster than a linear function,the linear interpolation of the function gives a value veryclose to 0. Thus, at each stage of the interpolation, thevalue grows by a very small amount.In fact, at every stage, it grows by an amount roughly equalto a1−d .


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Outline








Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









The basic problem

Given a function f , construct a continuous function g suchthat f (x) = 0 ⇐⇒ g(x) = x . Further, find a value x0 suchthat the sequence of iterates of g at x0 is a zero.

More generally, if we want to locate all roots, we should findinitial values corresponding to each root.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









The basic problem

Given a function f , construct a continuous function g suchthat f (x) = 0 ⇐⇒ g(x) = x . Further, find a value x0 suchthat the sequence of iterates of g at x0 is a zero.More generally, if we want to locate all roots, we should findinitial values corresponding to each root.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Terminology for fixed points

Let g be a function R → R. Denote:

g∞(x) := limn→∞

g (n)(x)

A fixed point x0 of g is termed:

I stable(defined) if there is an open neighbourhood U of x0

such that g∞(x) = x0 ∀ x ∈ U.

I unstable(defined) if there is an open neighbourhood U ofx0 such that g∞(x) = x0, x ∈ U =⇒ x = x0.

I neutral(defined) if it is neither stable nor unstable.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Basins of attractions

Given g : R → R, the basin of attraction of a fixed point x0

of g is the set of x ∈ R such that g∞(x) = x0.

To find a fixed point, we thus need to find a point which liesin the basin of attraction of that fixed point.There may be points x for which the sequence of g -iteratesdoes not converge; for instance it may be oscillatory, chaoticor divergent.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea











of g is the set of x ∈ R such that g∞(x) = x0.To find a fixed point, we thus need to find a point which liesin the basin of attraction of that fixed point.

There may be points x for which the sequence of g -iteratesdoes not converge; for instance it may be oscillatory, chaoticor divergent.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea











of g is the set of x ∈ R such that g∞(x) = x0.To find a fixed point, we thus need to find a point which liesin the basin of attraction of that fixed point.There may be points x for which the sequence of g -iteratesdoes not converge; for instance it may be oscillatory, chaoticor divergent.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Some obvious functions

Given the function f , an obvious candidate for g is:

g(x) = x + λf (x)

where λ is a suitably chosen real number.

Let’s try to seehow this performs for some simple cases of f .


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Some obvious functions

Given the function f , an obvious candidate for g is:

g(x) = x + λf (x)

where λ is a suitably chosen real number. Let’s try to seehow this performs for some simple cases of f .


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Conditions for stability

We have the following:

I If λf ′(x0) > 0 then x0 is an unstable fixed point of g .

I If λf ′(x0) < 0 then the fixed point may be stable,unstable or neutral. We can hope that if we choose λwisely, we can actually make it stable with a large basinof attraction.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









A squareroot of 2

Consider the polynomial

f (x) = x2 − 2

Trying λ = 1 or λ = −1 is hopeless – both the roots areunstable fixed points.

In fact, when λ = −1, even if we take a value fairly close to√2 the iterates give a periodic behaviour,.

However, if we try λ = −0.1, then −√

2 remains unstablebut

√2 becomes stable, with the basin of attraction as(

−√

2 , α)

where, numerically α ' 11.41.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









A squareroot of 2


f (x) = x2 − 2

Trying λ = 1 or λ = −1 is hopeless – both the roots areunstable fixed points.In fact, when λ = −1, even if we take a value fairly close to√

2 the iterates give a periodic behaviour,.

However, if we try λ = −0.1, then −√

2 remains unstablebut

√2 becomes stable, with the basin of attraction as(

−√

2 , α)



Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









A squareroot of 2


f (x) = x2 − 2

Trying λ = 1 or λ = −1 is hopeless – both the roots areunstable fixed points.In fact, when λ = −1, even if we take a value fairly close to√

2 the iterates give a periodic behaviour,.However, if we try λ = −0.1, then −

√2 remains unstable

but√

2 becomes stable, with the basin of attraction as(−√

2 , α)



Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Relation between λ and the derivative of f

Suppose x0 is a zero of f and x is a value fairly close to x0.Then, a good choice of λ is one that takes x closer to x0. Itshould not be too slow (otherwise we will not reach x) and itshould not be too fast (otherwise we run the risk ofovershooting x).

In other words, a good algorithm would be one where we canchoose the λ adaptively depending on the current value of x ,rather than having a fixed λ.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Relation between λ and the derivative of f

Suppose x0 is a zero of f and x is a value fairly close to x0.Then, a good choice of λ is one that takes x closer to x0. Itshould not be too slow (otherwise we will not reach x) and itshould not be too fast (otherwise we run the risk ofovershooting x).In other words, a good algorithm would be one where we canchoose the λ adaptively depending on the current value of x ,rather than having a fixed λ.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Linear interpolation

The Newton-Raphson method is an adaptation of thepreceding method where λ is chosen depending on x , in sucha way that it takes one closer to the solution if one is alreadyclose.

The idea is to consider a linear function with the same valueand the same first derivative at the current x , and find thezero of the linear function.In other words:

g(x) = x − f (x)

f ′(x)

Or equivalently, λ = −1/f ′(x).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










The Newton-Raphson method is an adaptation of thepreceding method where λ is chosen depending on x , in sucha way that it takes one closer to the solution if one is alreadyclose.The idea is to consider a linear function with the same valueand the same first derivative at the current x , and find thezero of the linear function.

In other words:

g(x) = x − f (x)

f ′(x)



Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










The Newton-Raphson method is an adaptation of thepreceding method where λ is chosen depending on x , in sucha way that it takes one closer to the solution if one is alreadyclose.The idea is to consider a linear function with the same valueand the same first derivative at the current x , and find thezero of the linear function.In other words:

g(x) = x − f (x)

f ′(x)



Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Intuitive explanation of this choice

There are many ways of intuitively justifyingNewton-Raphson. One is using a Taylor-like expansion for f ,viz:

f (x0 + h) = f (x0) + hf ′(x0) + O(h2)

Since f (x0) = 0, we get:

f (x0 + h) = hf ′(x0) + O(h2)

Thus, for x = x0 + h, we get:

g(x) = x0 + O(h2)

In other words, the Newton-Raphson seems to showquadratic convergence.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









The contraction mapping principle

While the above intuitive reasoning can be rigourized, thereis another approach that is somewhat more general. Thisuses the fact that for a C 2 differentiable function, the map gis a contraction in a sufficiently small neighbourhood of anyroot.

A contraction with contraction factor β < 1 is a mappingwith the property g such that:

|g(x)− g(y)| ≤ β |x − y |

The contraction mapping principle states that anycontraction mapping has a unique fixed point and moreoverthat this fixed point is obtained as the limit of iterates of thefunction taken at any point.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









The contraction mapping principle

While the above intuitive reasoning can be rigourized, thereis another approach that is somewhat more general. Thisuses the fact that for a C 2 differentiable function, the map gis a contraction in a sufficiently small neighbourhood of anyroot.A contraction with contraction factor β < 1 is a mappingwith the property g such that:

|g(x)− g(y)| ≤ β |x − y |

The contraction mapping principle states that anycontraction mapping has a unique fixed point and moreoverthat this fixed point is obtained as the limit of iterates of thefunction taken at any point.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Why the Newton-Raphson map is a contraction

We claim that if x0 is a root and f ′(x0) 6= 0, theNewton-Raphson map is a contraction in a suitableneighbourhood of x0.

The proof is as follows. We need to upper-bound|g(x2)− g(x1)| where x1, x2 are sufficiently close to x0. Forthe purpose, it suffices to upper-bound g ′(x) in a sufficientlysmall neighbourhood of x0, by a number strictly less than 1.

g ′(x) =f (x)f ′′(x)

(f ′(x))2

But since f (x)→ 0 as x → x0 and since f ′(x0) 6= 0, we canfind an interval around x0 such that g ′(x) is sufficientlysmall throughout the neighbourhood.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Why the Newton-Raphson map is a contraction

We claim that if x0 is a root and f ′(x0) 6= 0, theNewton-Raphson map is a contraction in a suitableneighbourhood of x0.The proof is as follows. We need to upper-bound|g(x2)− g(x1)| where x1, x2 are sufficiently close to x0. Forthe purpose, it suffices to upper-bound g ′(x) in a sufficientlysmall neighbourhood of x0, by a number strictly less than 1.

g ′(x) =f (x)f ′′(x)

(f ′(x))2

But since f (x)→ 0 as x → x0 and since f ′(x0) 6= 0, we canfind an interval around x0 such that g ′(x) is sufficientlysmall throughout the neighbourhood.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Outline








Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









General statement

Let F be a function of n + 2 variables (viz a mapU ∈ Rn+2 → R). Then the differential equation of order ndefined by F is the equation for a function f such that:

F (x , f (x), f ′(x), . . . , f (n)(x)) = 0

We need to solve this for f , that is, find an f that is a rootof this equation.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Why this parallels solving an ordinary equation

Solving an ordinary differential equation parallels solving anordinary equation, because we can consider the problem asfinding a zero of an equation.

Let’s try to make this analogy precise.The set of all functions from R to R forms a vector spaceover R under pointwise addition. Further, the set of Cn

functions (for any n) forms a vector subspace. Note that themap sending f to f ′ (the differentiation) is a well-definedmap from the space of C 1 functions to the space ofcontinuous functions.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










Solving an ordinary differential equation parallels solving anordinary equation, because we can consider the problem asfinding a zero of an equation.Let’s try to make this analogy precise.

The set of all functions from R to R forms a vector spaceover R under pointwise addition. Further, the set of Cn



Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










Solving an ordinary differential equation parallels solving anordinary equation, because we can consider the problem asfinding a zero of an equation.Let’s try to make this analogy precise.The set of all functions from R to R forms a vector spaceover R under pointwise addition. Further, the set of Cn



Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Why interval iteration is hard to imitate

Recall that for functions from R to R, we could start offwith a closed interval, then compute the function value atsome internal point, and thus get a smaller interval. By thisprocess, we obtained a nested sequence of intervals whoseintersection is the required zero.

However, this is not easy to imitate in the higherdimensional case because we cannot use a single point tobisect a closed region of possible functions.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Why interval iteration is hard to imitate

Recall that for functions from R to R, we could start offwith a closed interval, then compute the function value atsome internal point, and thus get a smaller interval. By thisprocess, we obtained a nested sequence of intervals whoseintersection is the required zero.However, this is not easy to imitate in the higherdimensional case because we cannot use a single point tobisect a closed region of possible functions.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









The naive adaptation of value-iteration

The naive adaptation is to consider:

F1 = f 7→(x 7→ F (x , f (x), f ′(x), . . . , f (n)(x))

)

and then look at:

G = f 7→ (x 7→ f (x) + F1(f )(x))

G = f 7→ (x 7→ f (x) + F (x , f (x), f ′(x), . . . , f (n)(x)))

Then, starting with an initial function f , we consider theiterates G (f ), G 2(f ) and so on and hope that theseconverge to somewhere.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









The naive adaptation of value-iteration

The naive adaptation is to consider:

F1 = f 7→(x 7→ F (x , f (x), f ′(x), . . . , f (n)(x))

)and then look at:

G = f 7→ (x 7→ f (x) + F1(f )(x))

G = f 7→ (x 7→ f (x) + F (x , f (x), f ′(x), . . . , f (n)(x)))

Then, starting with an initial function f , we consider theiterates G (f ), G 2(f ) and so on and hope that theseconverge to somewhere.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Why this is bad

The naive adaptation is bad because differentiation is ingeneral a destructive process, and it removes all the niceproperties of the original function. For instance, even if westart with a continuous function f we may not be able toapply G to f .

A better approach is to convert the equation to an integralequation. Integrations are more constructive, more holistic,and they step up differentiability.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Why this is bad

The naive adaptation is bad because differentiation is ingeneral a destructive process, and it removes all the niceproperties of the original function. For instance, even if westart with a continuous function f we may not be able toapply G to f .A better approach is to convert the equation to an integralequation. Integrations are more constructive, more holistic,and they step up differentiability.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









The corresponding integral equation

Consider a first-order differential equation:

F (x , f (x), f ′(x)) = 0

with initial value f (x0) = y0. Our aim is to find a solutionfor f in a neighbourhood of x0.

First observe that for any equation:∫ x1

x0

F (x , f (x), f ′(x)) dx = 0

For any x1. This suggests that we look at the function:

G = f 7→(

x1 7→ f (x1) +

∫ x1

x0

F (x , f (x), f ′(x)) dx

)and look at the iterates of f0 under G where f0 is theconstant function mapping everything to y0.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









The corresponding integral equation

Consider a first-order differential equation:

F (x , f (x), f ′(x)) = 0

with initial value f (x0) = y0. Our aim is to find a solutionfor f in a neighbourhood of x0.First observe that for any equation:∫ x1

x0

F (x , f (x), f ′(x)) dx = 0

For any x1. This suggests that we look at the function:

G = f 7→(

x1 7→ f (x1) +

∫ x1

x0

F (x , f (x), f ′(x)) dx

)and look at the iterates of f0 under G where f0 is theconstant function mapping everything to y0.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Intuitive justification

We claim the following:

I Any function fixed by G must be a solution to thedifferential equation

I If f is a function satisfying the initial value conditions,so is G (f )

Given these, it’s clear that if the sequence of iteratesconverges, then the thing it converges to is a solution of theinitial value problem.The question is: (when) does it converge?


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Recall of the contraction principle

Let (M, d) be a metric space. A function f : M → M istermed a contraction(defined) with contraction factor β < 1 iffor any x , y ∈ M:

d(f (x), f (y)) ≤ βd(x , y)

Clearly any contraction mapping is continuous, andmoreover, any contraction mapping in a complete metricspace has a unique fixed point.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Giving a metric space structure to functions

We’d ideally like conditions under which G is a contractionmapping.

In order to make sense of this statement, we first need toequip the space of functions with the structure of a completemetric space.We use the sup-norm for the purpose. For functions f and gfrom R to R, define d(f , g) = supx∈R d(f (x), g(x)).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










We’d ideally like conditions under which G is a contractionmapping.In order to make sense of this statement, we first need toequip the space of functions with the structure of a completemetric space.

We use the sup-norm for the purpose. For functions f and gfrom R to R, define d(f , g) = supx∈R d(f (x), g(x)).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










We’d ideally like conditions under which G is a contractionmapping.In order to make sense of this statement, we first need toequip the space of functions with the structure of a completemetric space.We use the sup-norm for the purpose. For functions f and gfrom R to R, define d(f , g) = supx∈R d(f (x), g(x)).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Conditions for the map to be a contraction

We want to show that:

supx∈R

d(G (f ),G (g)) ≤ βd(f , g)

where β < 1 is some suitable number.

In order to show this, we need to assume some condition onF . It turns out that a Lipschitz-like condition on F sufficesto ensure that the map is a contraction.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Conditions for the map to be a contraction

We want to show that:

supx∈R

d(G (f ),G (g)) ≤ βd(f , g)

where β < 1 is some suitable number.In order to show this, we need to assume some condition onF . It turns out that a Lipschitz-like condition on F sufficesto ensure that the map is a contraction.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Statement of the problem

Earlier we had looked at the general first-order differentialequation. We now look at a particular case, where we can“solve” for the first derivative in terms of the the point andits image under the function. That is, we look at adifferential equation of the form:

f ′(x) = H(x , f (x))

where H is a function of two variables.

Note that this is a special case of the general first-orderdifferential equation, viz where:

F (x , y , z) = H(x , y)− z


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Statement of the problem

Earlier we had looked at the general first-order differentialequation. We now look at a particular case, where we can“solve” for the first derivative in terms of the the point andits image under the function. That is, we look at adifferential equation of the form:

f ′(x) = H(x , f (x))

where H is a function of two variables.Note that this is a special case of the general first-orderdifferential equation, viz where:

F (x , y , z) = H(x , y)− z


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Picard iteration

Let’s look at the general iteration for the special case of afirst-degree differential equation:

G = f 7→(

x1 7→ f (x1) +

∫ x1

x0

F (x , f (x), f ′(x)) dx

)Writing F in terms of H, we get:

G = f 7→(

x1 7→ f (x1) +

∫ x1

x0

H(x , f (x))− f ′(x) dx

)This simplifies to:

G = f 7→(

x1 7→ f (x0) +

∫ x1

x+0H(x , f (x)) dx

)The latter is termed Picard iteration(defined).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Why Picard iteration works

Clearly, any function fixed by G is a solution to thefirst-degree first-order differential equation.

We need to justify conditions under which G has a fixedpoint, and the conditions under which starting with anygiven function, the iterates of G on that give the solution.The idea is to show that G is a contraction mapping (undersuitable assumptions on F ).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










Clearly, any function fixed by G is a solution to thefirst-degree first-order differential equation.We need to justify conditions under which G has a fixedpoint, and the conditions under which starting with anygiven function, the iterates of G on that give the solution.

The idea is to show that G is a contraction mapping (undersuitable assumptions on F ).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea










Clearly, any function fixed by G is a solution to thefirst-degree first-order differential equation.We need to justify conditions under which G has a fixedpoint, and the conditions under which starting with anygiven function, the iterates of G on that give the solution.The idea is to show that G is a contraction mapping (undersuitable assumptions on F ).


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Existence, uniqueness theorem for differentialequations

The existence and uniqueness theorem for differentialequations, also called the Cauchy-Picard theorem, Picard’sexistence theorem, and Picard-Lindelof theorem, is thefollowing:

Suppose F : R2 → R is a function and there is a compactrectangular region on which F is continuous in bothvariables, and is Lipschitz continuous in the second variable.Then the initial value problem:

f ′(x) = F (x , f (x)), f (x0) = y0

has a unique solution in a sufficiently small neighbourhood of(x0, y0) and further, the process of Picard iteration startingwith the constant function f0 = x 7→ y0 converges to this.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Existence, uniqueness theorem for differentialequations

The existence and uniqueness theorem for differentialequations, also called the Cauchy-Picard theorem, Picard’sexistence theorem, and Picard-Lindelof theorem, is thefollowing:Suppose F : R2 → R is a function and there is a compactrectangular region on which F is continuous in bothvariables, and is Lipschitz continuous in the second variable.Then the initial value problem:

f ′(x) = F (x , f (x)), f (x0) = y0

has a unique solution in a sufficiently small neighbourhood of(x0, y0) and further, the process of Picard iteration startingwith the constant function f0 = x 7→ y0 converges to this.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Importance of an existence and uniquenesstheorem

Geometrically, the existence and uniqueness theorem fordifferential equations tells us that if we have a “law” thatdescribes the instantaneous rate of change of a quantity interms of the time and the quantity, the law actually uniquelydetermines the global flow of the quantity given an initialvalue.

The iteration steps correspond to the fact that we can findthe correct flow by iterating a correction procedure.


Vipul Naik





Bisection search

Interpolation



Convergence issues






The idea









Importance of an existence and uniquenesstheorem

Geometrically, the existence and uniqueness theorem fordifferential equations tells us that if we have a “law” thatdescribes the instantaneous rate of change of a quantity interms of the time and the quantity, the law actually uniquelydetermines the global flow of the quantity given an initialvalue.The iteration steps correspond to the fact that we can findthe correct flow by iterating a correction procedure.

Finding solutions by approximationvipul/studenttalks/approximatingsolutions.pdf · Finding...

Documents

Transcript of Finding solutions by approximationvipul/studenttalks/approximatingsolutions.pdf · Finding...