DESIGN IN IMAGING FROM COMPRESSIVE TO COMPREHENSIVE

SENSING 

Lior HoreshJoint work with E. Haber, L. Tenorio,

A. Conn, U. Mello, J. Fohring

IBM TJ Watson Research CenterSIAM Imaging Science 2012, Philadelphia, PA - May 2012

INTRODUCTION

• Aim: infer model

• Given • Experimental design y• Measurements• Observation model

• Ill - posedness Naïve inversion ...

Fails...

• Need to fill in the missing information

EXPOSITION - ILL - POSED INVERSE PROBLEMS

Tm

( );T

d m y

( ) ( ); ;d m y F m y» +ò

Tm

( ) ( )2

mis

ˆ argmin ;

data fit

m F m y d y= -14444444244444443

2ˆ (1)

Tm m- = O

HOW TO IMPROVE MODEL RECOVERY ?

• How can we ...

• Improve observation model ?

• Incorporate more meaningful a-priori information ?

• Extract more information in the measurement procedure ?

• Provide more efficient optimization schemes ?

• Define appropriate distance measures / noise model ?

• Forward problem (simulation, description)

• Given: model m & observation model

• Simulate: data d

• Inverse problem (estimation, prediction)

• Given: data d & observation model

• Infer: model m (and uncertainties)

• Design (prescription)

• Given: inversion scheme & observation model

• Find: the ‘best’ experimental settings y, regularization S, …

EVOLUTION OF INVERSIONFROM SIMULATION TO DESIGN

WHY DESIGN ?

WHICH EXPERIMENTAL DESIGN IS BEST ?

Design 1 Design 3Design 2

• Where light sources and optodes should be placed ?

• In what sequence they should be activated ?

• What laser source intensities should be used ?

DESIGN QUESTIONS – DIFFUSE OPTICAL TOMOGRAPHY

Arridge 1999

• What frequencies should be used ?

• What trajectory of acquisition should be considered ?

DESIGN QUESTIONS –ELECTROMAGNETIC INVERSION

Newman 1996, Haber & Ascher 2000

DESIGN QUESTIONS – SEISMIC INVERSION

Clerbout 2000

• How simultaneous sources can be used ?

• How many projections should we collect and at what angles ?

• How accurate should the projection data be ?

DESIGN QUESTIONS –LIMITED ANGLE TOMOGRAPHY

WHAT CAN WE DESIGN ?

DESIGN EXPERIMENTAL LAYOUT

Stonehenge 2500 B.C.

DESIGN EXPERIMENTAL PROCESS

Galileo Galilei 1564-1642

RESPECT EXPERIMENTAL CONSTRAINTS…

French nuclear test, Mururoa, 1970

HOW TO DESIGN ?

• Previous work

• Well-posed problems - well established [Fedorov 1997, Pukelsheim 2006 ]

• Ill-posed problems - under-researched [Curtis 1999, Bardow 2008 ]

• Many practical problems in engineering and sciences are ill-posed

What makes ill-posed problems so special ?

ILL VS. WELL - POSED OPTIMAL EXPERIMENTAL DESIGN

OPTIMALITY CRITERIA IN WELL-POSED PROBLEMS

• For linear inversion, employ Tikhonov regularized least squares solution

• Bias - variance decomposition

• For over-determined problems

• A-optimal design problem

0a =

OPTIMALITY CRITERIA IN WELL-POSED PROBLEMS• Optimality criteria of the information matrix

• A-optimal design average variance

• D-optimality uncertainty ellipsoid

• E-optimality minimax

• Almost a complete alphabet…

( )1

C y-

THE PROBLEMS...

• Ill-posedness controlling variance alone reduces the error mildly [Johansen 1996 ]

• Non-linearity bias-variance decomposition is impossible

What strategy can be used ?

Proposition 1 - Common practice so far

Trial and Error…

EXPERIMENTAL DESIGN BY TRIAL AND ERROR

• Pick a model

• Run observation model of different experimental designs, and get data

• Invert and compare recovered models

• Choose the experimental design that provides the best model recovery

THE PROBLEMS...

• Ill-posedness controlling variance alone reduces the error mildly [Johansen 1996 ]

• Non-linearity bias-variance decomposition is impossible

What other strategy can be used ?

Proposition 2 - Minimize bias and variance altogether

How to define the optimality criterion ?

OPTIMALITY CRITERIA FOR DESIGN

• Loss

• Mean Square Error

• Bayes risk

Depends on the noise Depends on unknown modelT

m

Depends on unknown model

Computationally infeasible

Tm

OPTIMALITY CRITERIA FOR DESIGN

• Bayes empirical risk

• Assume a set of authentic model examples is available

• Discrepancy between training and recovered models [Vapnik 1998 ]

How can y and S be regularized ?

• Regularized Bayesian empirical risk

{ }1,...,

mnm m

OTHER DESIGNERS / KEY PLAYERS

• This is one doctrine

• Other interesting choices were developed by

• Y. Marzouk et. al, MIT (2011, 2012)

• A. Curtis et. al, University of Edinburgh (1999, 2010)

• D. Coles & M. Prange, Schlumberger (2008, 2012)

• S. Körkel et. al, Heidelberg University (2011)

• A. Bardow, RWTH Aachen University (2008, 2009)

DIFFERENTIABLE OBSERVATIONSPARSITY CONTROLLED DESIGN

• Assume: fixed number of observations

• Design preference: small number of sources / receivers is activated

• The observation model

• Regularized risk

Horesh, Haber & Tenorio 2011

1y

2y

( )2

12

ˆ, : ( , )m

i i p

n

i

y m y ySm m b=

= - +åEò

R

DIFFERENTIABLE OBSERVATION SPARSITY CONTROLLED DESIGN

• Total number of observations may be large

• Derivatives of the forward operator w.r.t. y

• Effective when activation of each source and receiver is expensive

Horesh, Haber & Tenorio 2011

Difficult…

WEIGHTS FORMULATION SPARSITY CONTROLLED DESIGN

• Assume: a predefined set of candidate experimental setups is given

• Design preference: small number of observations

( )1;F m y( )2;F m y( )999;F m y ( )1

;F m y( )2;F m y( )3;F m y( )1000

;F m y

Haber, Horesh & Tenorio 2010

WEIGHTS FORMULATION SPARSITY CONTROLLED DESIGN

• Let be discretization of the space [Pukelsheim 1994]

• Let

• The (candidates) observation operator is weighted

• w inverse of standard deviation

• - reasonable standard deviation - conduct the experiment

• - infinite standard deviation - do not conduct the experiment

( )( ) ( ) ( )1 1 1

2 2 2F m dW W Wm i g wd a+ = =ò

{ }1, ,

ky yK Y

( )( )

( )

1;

;k

F m y

F m

F m y

ì üï ïï ïï ïï ï= í ýï ïï ïï ïï ïî þ

M

Haber, Horesh & Tenorio 2010

WEIGHTS FORMULATION SPARSITY CONTROLLED DESIGN

• Solution - more observations give better recovery

• Desired solution many w‘s are 0

• Add penalty to promote sparsity in observations selection

• Less degrees of freedom

• No explicit access to the observation operator needed

( )2

1

2ˆ, : ( , ) , 0

1

m

reg im

p

n

ii

m S w wn

w wm bm=

= - + ³åEò

R

THE OPTIMIZATION PROBLEMS

• Leads to a stochastic bi-level optimization problem

• Direct formulation

• Weights formulation

Haber, Horesh & Tenorio 2010, 2011

THE OPTIMIZATION PROBLEM(S)

• In general (multi-level) optimization problem

• Difficult to solve (although not impossible) [Alexaderov &

Denis 1994 ]

• To simplify make assumptions on

• F - linear, nonlinear

• S - quadratic, matrix form

• y - discrete, continuous

• Important - sufficient improvement can be “good enough”

HOW MANY SAMPLES ARE NEEDED ?

• The optimist - choose a single m and design for it

• The pessimist - assume that m is the worst it can be

• The realist - choose an ensemble of m’s and average

• A Bayesian estimate of the frequentist risk [Rubin 1984 ]

OPTIMAL DESIGN IN PRACTICE

LINEAR DESIGN

LINEAR DESIGN

• Assume

• Then

LINEAR DESIGN

• Assume m has second moment

• Then

• Best linear design

CASE STUDY ILINEAR EXPERIMENTAL DESIGN

THE BEST (LINEAR) EXPERIMENT

• Considering discretization of the design space

• We obtain

BOREHOLE RAY TOMOGRAPHY -OBSERVATION MODEL

• Use 322 × 3 = 3072 rays

• Goal: choose 500 optimal observations

BOREHOLE RAY TOMOGRAPHY EXPERIMENTAL DESIGN ASSESSMENT

• Assess design performance upon unseen models (cross validation)

Test model Optimal designNon-optimal design

NON - LINEAR DESIGN

NON - LINEAR DESIGN

• Use Stochastic Optimization - approximate by sampling [Shapiro, Dentcheva & Ruszczynski 2009 ]

NON - LINEAR DESIGN

• Solution through sensitivity calculation

and therefore

• The sensitivities

NON - LINEAR DESIGN

• The reduced gradient

• Can use steepest descent/LBFGS/Truncated Gauss-Newton to solve the problem

CASE STUDY IINON-LINEAR EXPERIMENTAL DESIGN

• Governing equations

• Following Finite Element discretization

• Given model and design settings

• Find data , n < k

• Design: find optimal source-receiver configuration

IMPEDANCE TOMOGRAPHY – OBSERVATION MODEL

IMPEDANCE TOMOGRAPHY – DESIGNS COMPARISON

True modelNaive design Optimized design

Horesh, Haber & Tenorio 2011

MAGNETO - TULLERICS TOMOGRAPHY – OBSERVATION MODEL

• Governing equations

• Following Finite Volume discretization

• Given: model and design settings (frequency )

• Find: data ,

• Design: find an optimal set of frequencies

( ) {1; ( ; )

y

d m V A m i sw w

w w w-= +• ò

n k<nd Î ¡

km Î ¡ w

1 in

ˆ 0 on r

E i mE i s

E n

m w w-Ñ ´ Ñ ´ - = W

Ñ ´ ´ = ¶W

r r rr

y

MAGNETO - TELLURICS TOMOGRAPHY – DESIGNS COMPARISON

Test model Naive design

Optimized non-linear designOptimal linearized design

Haber, Horesh & Tenorio 2008, 2010

THE PARETO CURVE – A DECISION MAKING TOOL

Haber, Horesh & Tenorio 2010

10

6

8

0 20

2

4

40 60 80 1000

Ris

k

1w

• To drill or not to drill ?

Shakespeare 1564-1616

CASE STUDY IIIREGULARIZATION DICTIONARY

DESIGN

OPTIMAL OVER-COMPLETE DICTIONARY DESIGN

• Use popular L1 to get a sparse solution - but what dictionary should be used ?

• Accounts for the model space, observation operator and noise characteristics of the problem

• Requires more sophisticated algorithms to compute

• Non-smooth optimization framework Modified L-BFGS [Overton 2003 ] Horesh & Haber 2009

OPTIMAL DICTIONARY DESIGN - NUMERICAL RESULTS

Tm ( )T

F m d+ =ò ( )0 0D u D

Horesh & Haber 2009

TVm( )ˆ ˆDu D

OPTIMAL DICTIONARY DESIGN - ASSESSMENT WITH NOISE

( )TF m d+ =ò ( )0 0

D u D

Horesh & Haber 2009

( )ˆ ˆDu D

0.1%

1%

5%

Tm

TVm

OPTIMAL DICTIONARY DESIGN GEOPHYSICAL ASSESSMENT

( )TF m d+ =ò

Horesh & Haber 2009

Tm ( )0 0

D u D ( )ˆ ˆDu D

SUMMARY

THE QUOTE CORNER

• What I see in nature is a grand design that we can comprehend only imperfectly, and that must fill a thinking person with a feeling of humility — A. Einstein

• Don’t design for everyone. It’s impossible. All you end up doing is designing something that makes everyone unhappy — L. Reichelt

• It’s art if can’t be explainedIt’s fashion if no one asks for an explanationIt’s design if it doesn’t need explanation — W. Stokkel

TAKE HOME MESSAGES

• Only two (important) elements in the big (inversion) puzzle...

• Experimental design

• Regularization design

• Design in ill-posed inverse problems is an important topic which requires more attention

• New frontiers in inverse problems and optimization

EPILOGUE

DESIGN IN INVERSION – OPEN COLLABORATIVE RESEARCH

• IBM Research

• MITACS

• University of British Columbia

www.wix.com/ocrdesign/home

Andrew Conn

Eldad Haber

Raya Horesh

Jim Nagy

Luis Tenorio

Andreas Wächter

.

.

.

ACKNOWLEDGMENTS

Michele Benzi

Jennifer Fohring

Hui Haung

Ulisses Mello

David Nahamoo

Alessandro Veneziani

DOE NSF

.

.

.

Questions?Thank you