Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity...

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

Chapter 1Convex AnalysisNonlinear Multiscale Methods for Image and Signal AnalysisSS 2015

Michael MoellerComputer Vision

Department of Computer ScienceTU Munchen

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

Variational Problems

Example: Inpainting

u = arg minu∈Rn

∥∥∥∥√(Dxu)2 + (Dy u)2

∥∥∥∥1, such that ui = fi ∀i ∈ I

with index set I of uncorrupted pixels.

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

Convexity

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Let us repeat some basics things to talk about

u = arg minu∈Rn

E(u).

Definition

• For E : Rn → R ∪ {∞}, we call

dom(E) := {u ∈ Rn | E(u) <∞}

the domain of E .• We call E proper if dom(E) 6= ∅.

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Definition: Convex Function

We call E : Rn → R ∪ {∞} a convex function if

1 dom(E) is a convex set, i.e. for all u, v ∈ dom(E) and allθ ∈ [0,1] it holds that θu + (1− θ)v ∈ dom(E).

2 For all u, v ∈ dom(E) and all θ ∈ [0,1] it holds that

E(θu + (1− θ)v) ≤ θE(u) + (1− θ)E(v)

We call E strictly convex, if the inequality in 2 is strict for allθ ∈]0,1[, and v 6= u.

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Example: Inpainting

u = arg minu∈Rn

∥∥∥∥√(Dxu)2 + (Dy u)2

∥∥∥∥1, such that ui = fi ∀i ∈ I

with index set I of uncorrupted pixels.→ Discuss convexity.

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

Existence

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


When doesu = arg min

u∈RnE(u)

exist?

• E is lower semi-continuous, i.e. for all u

lim infv→u

E(v) ≥ E(u)

holds.• There exists an α such that

{u | E(u) ≤ α}

is non-empty and bounded.

Proof: Board.

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Fundamental Theorem of Optimization

If E : Rn → R ∪ {∞} is lower semi-continuous and has anonempty bounded sublevelset, then there exists

u = arg minu∈Rn

E(u)

Remark: For a proper convex function, lower semi-continuity isthe same as the closedness of the sublevelsets.

Examples on the board:• A convex continuous function that does not have a

minimizer• A convex function with bounded sublevelsets that does not

have a minimizer

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Continuity of Convex Functions

If E : Rn → R ∪ {∞} is convex, then E is locally Lipschitz (andhence continuous) on int(dom(E)).

Proof: Exercise (in 1d)

Board: Considering the interior is important!

Conclusion

If E : Rn → R is convex, then E is continuous.

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Definition

We call E : Rn → R ∪ {∞} coercive, if all sequences (un)n with‖un‖ → ∞ meet E(un)→∞.

Theorem

If E : Rn → R is convex and coercive, then there exists

u = arg minu∈Rn

E(u).

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


When isu = arg min

u∈RnE(u)

unique?

Theorem

If E : Rn → R ∪ {∞} is convex, then any local minimum is aglobal minimum. If E is strictly convex, the global minimum isunique.

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

Subdifferential Calculus

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

Variational ProblemsWhat is an optimality condition for

u = arg minu∈Rn

E(u)?

Definition: Subdifferential

We call

∂E(u) = {p ∈ Rn | E(v)− E(u)− 〈p, v − u〉 ≥ 0, ∀v ∈ Rn}

the subdifferential of E at u.• Elements of ∂E(u) are called subgradients.• If ∂E(u) 6= ∅, we call E subdifferentiable at E .• By convention, ∂E(u) = ∅ for u 6= dom(E).

Theorem: Optimality condition

Let 0 ∈ ∂E(u). Then u ∈ arg minu E(u).

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Examples for non-differentiable functions:• The `1 norm.• Functional

E(u) ={

0 if u ≥ 0∞ else.

Subdifferential and derivatives

Let the convex function E : Rn → R ∪ {∞} be differentiable atx ∈ dom(E). Then

∂E(x) = {∇E(x)}.

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Is any convex E subdifferentiable at x ∈ dom(E)?

Answer: Almost...

Definition: Relative Interior

The relative interior of a convex set M is defined as

ri(M) := {x ∈ M | ∀y ∈ M, ∃λ > 1, s.t. λx + (1− λ)y ∈ M}

Theorem: Subdifferentiability1

If E is a proper convex function and u ∈ ri(dom(E)), then∂E(u) is non-empty and bounded.

1Rockafellar, Convex Analysis, Theorem 23.4

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Theorem: Sum rule2

Let E1, E2 be convex functions such that

ri(dom(E1)) ∩ ri(dom(E2)) 6= ∅,

then it holds that

∂(E1 + E2)(u) = ∂E1(u) + ∂E2(u).

Example: Minimize (u − f )2 + ιu≥0(u).

Example: Minimize 0.5(u − f )2 + α|u|.


Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Theorem: Chain rule3

If A ∈ Rm×n, E : Rm → R ∪ {∞} is convex, andri(dom(E)) ∩ range(A) 6= ∅, then

∂(E ◦ A)(u) = A∗∂E(Au)

Example: Minimize ‖Au − f‖22.


Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Summary (without assumptions):

• ∂E(u) = {p ∈ Rn | E(v)− E(u)− 〈p, v − u〉 ≥ 0,∀v ∈ Rn}

• If E differentiable: ∂E(x) = {∇E(x)}

• Sum rule ∂(E1 + E2)(x) = ∂E1(x) + ∂E2(x)

• Cain rule ∂(E ◦ A)(u) = A∗∂E(Au)

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

TV minimization

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

What is TV again?

For u ∈ Rm×n let us consider the anisotropic total variation

TVa(u) =m∑

i=2

n∑j=1

|ui,j − ui−1,j |+m∑

i=1

n∑j=2

|ui,j − ui,j−1|

For doing math, it is often easier to consider ~ui+m(j−1) = u(i , j)and write

TVa(u) = ‖K~u‖1

for a suitable matrix K that discretizes the gradient.

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

TV Minimization

Our problem becomes

u(α) = arg minu∈Rnm

12‖u − f‖2

2 + α‖Ku‖1.

Let us try to apply all the learned theory. The minimizer isobtained at

0 ∈ u(α)− f + αK T q

with q ∈ ∂‖Ku(α)‖1, i.e.

qi

= 1 if (Ku(α))i > 0= −1 if (Ku(α))i < 0∈ [−1,1] if (Ku(α))i = 0

Seems extremely difficult to find...

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

TV Minimization

Crazy idea:

minu

12‖u − f‖2

2 + α‖Ku‖1 =minu

12‖u − f‖2

2 + α sup‖q‖∞≤1

〈Ku,q〉

=minu

sup‖q‖∞≤1

12‖u − f‖2

2 + α〈Ku,q〉

Can we exchange min and sup?

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

TV Minimization

Saddle point problems4

Let C and D be non-empty closed convex sets in Rn and Rm,respectively, and let S be a continuous finite concave-convexfunction on C × D. If either C or D is bounded, one has

infv∈D

supq∈C

S(v ,q) = supq∈C

infv∈D

S(v ,q).

We can therefore compute

minu

12‖u − f‖2

2 + α‖Ku‖1 =minu

sup‖q‖∞≤1

12‖u − f‖2

2 + α〈Ku,q〉

= sup‖q‖∞≤1

minu

12‖u − f‖2

2 + α〈Ku,q〉.

4Rockafellar, Convex Analysis, Corollary 37.3.2

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

TV Minimization

Now the inner minimization problem obtains its optimum at

0 = u − f + αK T q,

⇒u = f − αK T q.

The remaining problem in q becomes

sup‖q‖∞≤1

12‖f − αK T q − f‖2

2 + α〈K (f − αK T q),q〉

= sup‖q‖∞≤1

12‖αK T q‖2

2 + α〈Kf ,q〉 − ‖αK T q‖22

= sup‖q‖∞≤1

−12‖αK T q − f‖2

2

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

TV Minimization

Since we prefer minimizations over maximizations, we write

q =arg max‖q‖∞≤1

−12‖αK T q − f‖2

2

= arg min‖q‖∞≤1

12

∥∥∥∥K T q − fα

∥∥∥∥2

2

Idea: Gradient descent + project onto feasible set.

qk+1 = π‖q‖∞≤1

(qk − τK

(K T qk − f

α

))

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

TV Minimization

Gradient projection algorithm5

The algorithm

qk+1 = π‖·‖∞≤1

(qk − τK

(K T qk − f

α

))with uk = f − αqk , for TV minimization converges for τ < 1

4 .

Remark: The 1/4 is two over the Lipschitz constant of thegradient of the smooth objective.

5Levitin, Polyak, Constrained minimization problems, 1966. Goldstein,Convex programming in Hilbert space, 1964.

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

TV Minimization

1 img = im2double(imread('cameraman.tif'));2 [m,n] = size(img);3

4 e = ones(m,1);5 yDerMat = spdiags([-e e], 0:1, m, m);6 yDerMat(end) =0;7

8 e = ones(n,1);9 xDerMat = spdiags([-e e], 0:1, n, n)';

10 xDerMat(end)=0;11

12 yDer = yDerMat*img;13 xDer = img*xDerMat;14

15 gradientMatrixForVectorizedImage = ...[kron(xDerMat',speye(m,m)); kron(speye(m,m), ...yDerMat)];

16

17 imgGradient = gradientMatrixForVectorizedImage*img(:);18 figure, imagesc(reshape(imgGradient(1:n*m), ...

[n,m])), colorbar

Note that AXB = C ⇔ kron(B′,A)~X = ~C!

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

Duality

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

We replaced ‖Ku‖1 (not differentiable) with sup‖q‖∞≤1〈q,Ku〉.

Numerics became easier.

Is there a systematic concept behind this idea?

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Very important concept: Duality!

Definition: Convex Conjugate

We define the convex conjugate of the functionE : Rn → R ∪ {∞} to be

E∗(p) = supu∈Rn

(〈u,p〉 − E(u)) .

Board: E∗ is convex.

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Examples:

• E(u) = u2 leads to E∗(p) = p2

• E(u) = ‖u‖2 leads to E∗(p) ={

0 if ‖p‖2 ≤ 1,∞ else.

• E(u) = ‖u‖1 leads to E∗(p) ={

0 if ‖p‖∞ ≤ 1,∞ else.

• E(u) = ‖u‖∞ leads to E∗(p) ={

0 if ‖p‖1 ≤ 1,∞ else.

• E(u) ={

0 if ‖u‖2 ≤ 1,∞ else. leads to E∗(p) = ‖p‖2.

• E(u) ={

0 if ‖u‖∞ ≤ 1,∞ else. leads to E∗(p) = ‖p‖1.

• E(u) ={

0 if ‖u‖1 ≤ 1,∞ else. leads to E∗(p) = ‖p‖∞.

Suspicion: E∗∗ = E?

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Fenchel-Young Inequality6

Let E be proper, convex and lower semi-continuous,u ∈ dom(E) ⊂ Rn, and p ∈ Rn, then

E(u) + E∗(p) ≥ 〈u,p〉.

Equality holds if and only if p ∈ ∂E(u).

Theorem: Biconjugate7

Let E be proper, convex and lower semi-continuous, thenE∗∗ = E .

Now we understand what we did for TV minimization...

6Borwein, Zhu Techniques of variational analysis, Proposition 4.4.17Rockafellar, Convex Analysis, Theorem 12.2

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


Theorem: Subgradient of convex conjugate8

Let E be proper, convex and lower semi-continuous, then thefollowing two conditions are equivalent:

• p ∈ ∂E(u)• u ∈ ∂E∗(p)

Board: Example with proximity operator.


Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015


What is this good for? Easier analysis and optimization!

Fenchel’s Duality Theorem9

The following problems yield the same result:

infu H(u) + R(Ku) ”Primal”

infu supq H(u) + 〈q,Ku〉 − R∗(q)”Saddle point”

supq infu H(u) + 〈q,Ku〉 − R∗(q)

supq −H∗(−K ∗q)− R∗(q) ”Dual”

Assuming that H and R are proper, lower semi-continuous,convex functions with ri(dom(H)) ∩ ri(dom(R ◦ K )) 6= ∅.

9C.f. Rockafellar, Convex Analysis, Section 31, or Borwein, Zhu Techniquesof variational analysis, Theorem 4.4.3

Convex Analysis

Michael Moeller

BasicsConvexity

Existence

Uniqueness

The Subdifferential

TV minimization

Duality

updated 07.05.2015

Variational ProblemsConsider the isotropic total variation

TV (u) =m∑

i=2

n∑j=2

√(ui,j − ui−1,j)2 + (ui,j − ui,j−1)2 = ‖Ku‖2,1

How do we minimize

12‖u − f‖2

2 + α‖Ku‖2,1 ?

Dual problem:

12

∥∥∥∥K T q − fα

∥∥∥∥2

2+ (‖ · ‖2,1)

∗(q)

Similar to previous examples:

(‖ · ‖2,1)∗(q) =

{0 if ‖q‖2,∞ ≤ 1 ∀i ,∞ else.

→ Gradient projection algorithm!

Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity...

Documents

Transcript of Convex Analysis Michael Moeller Chapter 1 · Convex Analysis Michael Moeller Basics Convexity...