Zeroing in on the Implicit Function Theorem

The Implicit Function Theorem for

several equations in several unknowns.

So where do we stand?

Solving a system of m equations in n unknowns is equivalent to finding the “zeros” of a vector-valued function from

n → m. When n > m, such a system will “typically”

have infinitely many solutions. In “nice” cases, the solution will be a function from

n-m → m.

So where do we stand?

Solving linear systems is easy; we are interested in the non-linear case.

We will ordinarily not be able to solve a system “globally.” But under reasonable conditions, there will be a solution function in a (possibly small!) region around a single “known” solution.

xx

yy((aa,,bb))

((aa,,bb))

yy = = gg((xx))

For a Single Equation in Two UnknownsIn a small box around (a,b), we hope to find g(x).

And we can, provided that the y-partials at (a,b) are continuous and non-zero.

xx

yy

0),(

bayf

D

((aa,,bb))

((aa,,bb))

yy = = gg((xx))

Start with a point (a,b) on the contour line, where the partial with respect to y is not 0:

Make the box around (a,b) small enough so that all of the y-partials in this box are “close” to D.

xx

yy

0),(

bayf

D

((aa,,bb))

((aa,,bb))

yy = = gg((xx))

Start with a point (a,b) on the contour line, where the partial with respect to y is not 0:

x

D

yxfyyx

,

Fix x.

For this x, construct function

Iterate x(y). What happens?

A Whole Family of Quasi-Newton Functions

D

yxfyyx

,

( , )f a bD

y

Remember that (a,b) is

a “known” solution.and

There are a whole bunch of these functions:

There is one for each x value.

A Whole Family of Quasi-Newton Functions

( , )f a bD

y

Remember that (a,b) is a “known” solution to f(x,y)=0,

and

,a

f a yy y

D

If x = a, then we have

The “best of all possible worlds ”

Leibniz method!

If x a, then we have

,x

f x yy y

D

The “pretty good ”Quasi-Newton

method!

What are the issues?

We have to make sure the iterated maps converge---how do we do this?

“Pretty good” quasi-Newton’s method

Dxf

xxQ)(

)(

If we choose D near enough to f’(p)

so that |Q’(p)| < ½, iterated maps will converge in a neighborhood of p.

How does that work in our case?

,x

f x yy y

D

If we make sure that the partials of f with respect to y are all near enough to D to guarantee that | x’(y) | < ½ for all (x,y) in the square, then the iterated maps will converge.

The Role of the DerivativeIf we have a function f: [a,b] → which is differentiable on (a,b) and continuous on [a,b], the Mean Value Theorem says that there is some c in [a,b] such that

( ) ( ) ( ) ( )'( ) '( ) .

f b f a f b f af c f c

b a b a

If |f (x)| < k < 1 for all x in [a,b] ( ) ( )

'( ) ( ) ( )f b f a

k f c f b f a k b ab a

Likewise, if |f (x)| > k > 1 for all x in [a,b]

( ) ( )'( ) ( ) ( )

f b f ak f c f b f a k b a

b a

Distances contract by a factor of k!

Distances expand by a factor of k!

The Role of the DerivativeIf p is a fixed point of f in [a,b], and |f (x)| < k < 1 for all x in [a,b], then

( ) ( ) .f x f p k x p

Likewise, if |f (x)| > k > 1 for all x in [a,b], f moves other points farther and farther away from p. (Repelling fixed point!)

f moves other points closer to p by a

factor of k!But f (p) = p, so ( ) .f x p k x p

Each time we re-apply the next iterate is even closer to p! (Attracting fixed point!)

2 2( ) ( ( )) ( ) ( ) | | .f x p f f x f p k f x p k x p

What are the issues?

Not so obvious . . . we have to work a bit to make sure we get the right fixed point. (We don’t leave the box!)

((aa,,bb))

x

yy

x

Systems of Equations and DifferentiabilityA vector valued function f of several variables is differentiable at a vector p if in some small neighborhood of p, the graph of f “looks a lot like” an affine function.

( ) ( )( ) ( ) f z Df p z p f p

That is, there is a linear transformation Df(p) so that for all z “close” to p,

Suppose that f (p) = 0. When can we solve f(z) = 0 in some neighborhood of p?

Where Df(p) is the Jacobian matrix made up of all the partial derivatives of f.

Systems of Equations and Differentiability

( ) ( )( ) ( ) f z Df p z p f p

For z “close” to p,

Since f (p) = 0, ( ) ( )( ) f z Df p z p

When can we solve f(z) = 0 in some neighborhood of p?

Answer: Whenever we can solve

( )( ) Df p z p 0

Because the existence of a solution depends on the geometry of the function.

When can we do it?

is a linear system.We understand linear systems extremely well.

We can solve for the variables y1, y2, and y3 in terms of x1 and x2 if and only if the sub-matrix . . .is invertible.

11 12 13 14 15

21 22 23 24 25

31 32 33 34 35

a a a a a

a a a a a

a a a a a

( )( ) Df p z p 011 1 12 2 13 3 14 1 15 2

21 1 22 2 23 3 24 1 25 2

31 1 32 2 33 3 34 1 35 2

0

0

0

a y a y a y a x a x

a y a y a y a x a x

a y a y a y a x a x

A bit of notation

To simplify things, we will write our vector valued function F : n+m →m.

We will write our “input” variables as concatenations of n-vectors and m-vectors.

e.g. (y,x)=(y1, y2,. . ., yn, x1, x2,. . ., xm)

So when we solve F(y,x)=0 we will be solving for the y-variables in terms of the x-variables.

The System

1 1 1 1 1

1 2 1

32 2 2 2

1 2 1

3 3 3 3 3

1 2 1

1 2 1

( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( )

n m

n n

n n

m m m m m

n m

F F F F F

y y y x x

FF F F F

y y y x x

F F F F F

y y y x x

F F F F F

y y y x x

p p p p p

p p p p p

p p p p p

p p p p p

1 1

2 2

1 1

n n

n

m n m

y p

y p

y p

x p

x p

0

( )( ) Df p z p 0

We can solve for the variables y1, y2, . . . yn in terms of x1, x2 ,. . ., xm if and only if the sub-matrix . . .is invertible.

So the Condition We Need is. . .

1 1 1

1 2

32 2

1 2

3 3 3

1 2

1 2

( ) ( ) ( )

( ) ( ) ( )

.( ) ( ) ( )

( ) ( ) ( )

n

n

n

m m m

n

F F F

y y y

FF F

y y y

F F F

y y y

F F F

y y y

p p p

p p p

Dp p p

p p p

. . . the invertibility of the matrix

We will refer to the inverse of the matrix D as D-1.

The Implicit Function Theorem

F : n+m →m has continuous partials. Suppose b n and a m with F(b,a)=0. The n x n matrix that corresponds to the y-

partials of F (denoted by D) is invertible.

Then “near” a there exists a unique function g(x) such that F(g(x),x)=0; moreover g(x) is continuous.

What Function Do We Iterate?

D

yxfyyx

,

The 2-variable case. Fix x.

For this x, construct function

Iterate x(y).

Where ( , )

.f a b

Dy

What Function Do We Iterate?

D

yxfyyx

,

The 2-variable case. Fix x.

For this x, construct function

Iterate x(y).

, -1x y y D F y x

The n-variable case. Fix x.

For this x, construct function

Iterate x(y).

Multi-variable Parallel?

n+m

t

a xg(x)

We want g continuous & unique

F(g(x),x) = 0

m

(b,a)

r

b

r

n

tr

Partials of F in the ball are all close to the partials at (b,a)

(b,a)

r

n+m

Notation: Let dF(y,x) denote the n x n submatrix made up of all the y partial derivatives of F at (y,x). Step I: Since D is invertible, D-1 0.

We choose r as follows:

We choose r as follows: use continuity of the partials of F to choose r small enough so that for all (y,x) Br(b,a)

1

1D ( , )

2 Dd

F y x

Then 1 1( ) I D ( , ) D ( , ) .n d D d x y F y x F y x

So1

1

1

( ) D D ( , )

1 1D

22 D

d

x y F y x

!1 2 1 2

1( ) ( )

2 x xy y y yMultivariable Mean

Value Theorem

Notation: Let dF(y,x) denote the n x n submatrix made up of all the y partial derivatives of F at (y,x). Step I: Since D is invertible, D-1 0.

There are two (highly non-trivial) steps in the proof:

In Step 1 we choose the radius r of the ball in n+m so that the partials in the ball are all “close” to the (known) partials at (b,a). This makes x contract distances on the ball, forcing convergence to a fixed point. (Uses continuity of the partial derivatives.)

In Step 2 we choose the radius t around a “small” so as to guarantee that our iterates stay in the ball. In other words, our “initial guess” for the quasi-Newton’s method is good enough to guarantee convergence to the “correct” root.

The two together guarantee that we can find value a g(x) which solves the equation. We then map x to g(x).

Since this is the central idea of the proof, I will reiterate it:

In Step 1 we make sure that iterated maps on our “pretty good” quasi-Newton function actually converge.

In Step 2, by making sure that they didn’t “leave the box,” we made sure the iterated maps converged to the fixed point we were aiming for. That is, that they didn’t march off to some other fixed point outside the region.

((aa,,bb))

x

yy

x

Final thoughtsThe Implicit Function Theorem is a DEEP theorem of

Mathematics.

Fixed point methods show up in many other contexts: For instance: They underlie many numerical approximation techniques. They are used in other theoretical contexts, such as in

Picard’s Iteration proof of the fundamental theorem of differential equations.

The iteration functions frequently take the form of a “quasi-Newton’s method.”