1

Fast Trust Region for SegmentationLena Gorelick

Joint work with Frank Schmidt and Yuri Boykov

Rochester Institute of Technology, Center of Imaging Science January 2013

2

Sf,E(S) B(S)Sf,E(S)

Image segmentation Basics

I

Fg)|Pr(I Bg)|Pr(I

Sx

f(x)s(x)E(S)

bg)|Pr(I(x)fg)|Pr(I(x)lnf(x)

S

3

Standard Segmentation Energy

Fg

Bg

Intensity

ProbabilityDistribution

Target AppearanceResulting Appearance

4

Minimize Distance to Target Appearance Model

KL( R(S) Bha( R(S)

)||

),

2L||-|| R(S)

Sp p

p

bg)|Pr(Ifg)|Pr(I

ln

Non-linear harder to optimizeregional term

TS

TS

TS

5

Non-linear Energies with High- Order Terms

complex appearance models shape

non-linear regional

term

B(S)R(S)E(S)

S

6

Related Work

Can be optimized with gradient descent First order (linear) approximation models

We use more accurate non-linear approximation models based on trust region

Ben Ayed et al. Image Processing 2008,Foulonneau et al., PAMI 2006Foulonneau et al., IJCV 2009

7

Our contributions

General class of non-linear regional functionals

Optimization algorithm based on trust region framework – Fast Trust Region

)S,f,,S,fF(R(S) k1

8

Outline

Non-linear Regional Functionals Overview of Trust Region

Framework Trust region sub-problem

Lagrangian Formulation for the sub-problem

Fast Trust Region method Results

Regional FunctionalExamples

Volume Constraint

1f(x) ibin for (x)fi

S,fiS1,|S|

20 )VS1,(R(S)

9

10

Regional FunctionalExamples

Bin Count Constraint

1f(x) ibin for (x)fi

2ii

k

1i)VS,f(ΣR(S)

S1,|S| S,fi

11

Regional FunctionalExamples Histogram Constraint

1f(x) ibin for 1(x)fi

S1,S,f

(S)P ii

S1,|S| S,fi

12

Regional FunctionalExamples Histogram Constraint

VS)PΣR(S) 2ii

k

1i

(

2L||-|| R(S) TS

S1,S,f

(S)P ii

13

Regional FunctionalExamples Histogram Constraint

V(S)P(S)logPΣR(S)

i

ii

k

1i

KL( R(S) )|| TS

S1,S,f

(S)P ii

14

Regional FunctionalExamples Histogram Constraint

ii

k

1iV(S)PΣlogR(S)

Bha( R(S) ), TS

S1,S,f

(S)P ii

15

Shape Prior

Volume Constraint is a very crude shape prior

Can be encoded using a set of shape moments mpq

p+q is the order

16

Volume Constraint is a very crude shape prior

Shape Prior

01pq yxy)(x,f 22pq yxy)(x,f 32pq yxy)(x,f

qppq

pqpq

yxy)(x,f

Sf(S)m

,

17

Shape Prior using Shape Moments mpq

Volumem00

...RatioAspect

nOrientatio Principalmmmm

0211

1120

Mass OfCenter )m,(m 0110

18

Shape Prior Constraint

Shape Prior using Shape moments

Dist( ),R(S)

kqp

2pqpq (T))m(S)(mR(S)

qppq

pqpq

yxy)(x,f

f(S)m

S,

S T

19

Optimization of Energies with Regional Functional

B(S)R(S)E(S)

20

Gradient Descent with Level Sets

Gradient Descent First Order Taylor Approximation for

R(S) First Order approximation for B(S)

(“curvature flow”) Only robust with tiny steps

Slow Sensitive to initialization

B(S)R(S)E(S)

http://en.wikipedia.org/wiki/File:Level_set_method.jpg

Ben Ayed et al. CVPR 2010,Freedman et al. tPAMI 2004

21

Energy Specific vs. General

Speedup via energy- specific methods Bhattacharyya Distance Volume Constraint

In contrast: Fast optimization algorithm

for general high-order energies

Based on more accurate non-linear approximation models

Ben Ayed et al. CVPR 2010,Werner, CVPR2008Woodford, ICCV2009

22

B(S)(S)U(S)E 0d||SS|| 0

~min

General Trust Region ApproachAn overview

The goal is to optimize

Trust regio

nTrust

Region Sub-

Problem

B(S)R(S)E(S)

B(S)(S)U(S)E 0 ~

• First Order Taylor for R(S)

• Keep quadratic B(S)

0Sd

23

General Trust Region ApproachAn overview

The goal is to optimize B(S)R(S)E(S)

B(S)(S)U(S)E 0 ~

Trust Region Sub-

Problem

B(S)(S)U(S)E 0d||SS|| 0

~min

0Sd

24

||SS||λB(S)(S)U(S)L 00λ

How to Solve Trust Region Sub-Problem Constrained optimization

minimize

Unconstrained Lagrangian Formulationminimize

Can be optimized globally using graph-cut

d||SS||s.t.B(S)(S)U(S)E

0

0

~

||SS||λ(S)E(S)L 0λ ~

Can be approximated with

unary termsBoykov et al. ECCV

2006

25

Spectrum of Solutions for different λ or d

• Newton step• “Gradient Descent”• Exact Line Search

(ECCV12)

28

General Trust Region

Repeat Solve Trust Region Sub-problem

around S0 with radius d Update solution S0 Update Trust Region Size d

Until Convergence

29

Fast Trust Region

General Trust Region Control of

the distance constraint d

Lagrangian Formulation Control of

the Lagrange multiplier λ λd1λ

31

Comparison Simulated Gradient Descent Exact Line-Search (ECCV 12) Newton step Fast Trust Region (CVPR 13)

Volume Constraint for Vertebrae segmentation

Log-Lik. + length + volumeFast Trust Region

InitializationsLog-Lik. + length

35

maxVminV0

)(SR

|| S

36

Shape Prior with Geometric moments for liver segmentation

Fast Trust Region

Log-LikelihoodsNo Shape Prior

Second order geometric moments computed for the user provided initial ellipse

37

Appearance model with KL Divergence Constraint

Init

Fast Trust Region

“Gradient Descent”

Exact Line Search

Appearance model is obtained from the ground truth

38

Appearance Model with Bhattacharyya Distance Constraint

“ “

Fast Trust Region

“Gradient Descent”

Exact Line Search

Appearance model is obtained from the ground truth

39

Shape prior with Tchebyshev moments for spine segmentation

Log-Lik. + length + Shape PriorFast Trust Region

Second order Tchebyshev moments computed for the user scribble

40

Appearance Model with Bhattacharrya Distance Constraint

BHA. + length Fast Trust Region

Ground Truth

Appearance model is obtained from the ground truth

41

Future Directions

Multi-label Fast Trust Region Binary shape prior:

affine-invariant Legendre/Tchebyshev moments

Learning class specific distribution of moments

Multi-label shape prior moments of multi-label atlas map

Experimental evaluation and comparison between level-sets and FTR.

42

Thank you