Bayesian Experimental Design for Large Scale Signal ...
Transcript of Bayesian Experimental Design for Large Scale Signal ...
Bayesian Experimental Design forLarge Scale Signal Acquisition Optimization
Matthias Seeger
Laboratory for Probabilistic Machine LearningEcole Polytechnique Fédérale de Lausanne
http://lapmal.epfl.ch/
9/12/2013
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 1 / 20
Motivation
Magnetic Resonance Imaging
⊕ Extremely versatile⊕ Noninvasive, no ionizing radiation Very expensive Long scan times: Major limiting factor
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 2 / 20
Motivation
Magnetic Resonance Imaging
Faster scans by undersampledreconstructionWhich fast designs give best images?
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 2 / 20
Motivation
Magnetic Resonance Imaging
Faster scans by undersampledreconstructionWhich fast designs give best images?
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 2 / 20
Motivation
Image Reconstruction
Ideal Image u
Reconstruction
Measurement
Design
y X u
Data y
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 3 / 20
Motivation
Sampling Optimization
Reconstruction
y X u
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 4 / 20
Bayesian Experimental Design
Bayesian Experimental Design
Posterior: Uncertainty inreconstructionExperimental design:Find poorly determineddirectionsSequential search withinterjacent partialmeasurements
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 5 / 20
Bayesian Experimental Design
Bayesian Experimental Design
Posterior: Uncertainty inreconstructionExperimental design:Find poorly determineddirectionsSequential search withinterjacent partialmeasurements
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 5 / 20
Bayesian Experimental Design
Maximizing Information Gain
Score design extension X ∗ by information gain:
I(X ∗) = I(y∗,u |y ) = H[P(u |y )]︸ ︷︷ ︸Before
−H[P(u |y∗,y )]︸ ︷︷ ︸After
Most work: Combinatorial aspectsAssume: I(X ∗) tractable to computeAssume: I(X ∗) cheap to compute (many X ∗)Simple greedy forward works well in practice . . .
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 6 / 20
Bayesian Experimental Design
Maximizing Information Gain
Score design extension X ∗ by information gain:
I(X ∗) = I(y∗,u |y ) = H[P(u |y )]︸ ︷︷ ︸Before
−H[P(u |y∗,y )]︸ ︷︷ ︸After
Most work: Combinatorial aspectsAssume: I(X ∗) tractable to computeAssume: I(X ∗) cheap to compute (many X ∗)Simple greedy forward works well in practice . . .
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 6 / 20
Bayesian Experimental Design
Maximizing Information Gain
Score design extension X ∗ by information gain:
I(X ∗) = I(y∗,u |y ) = H[P(u |y )]︸ ︷︷ ︸Before
−H[P(u |y∗,y )]︸ ︷︷ ︸After
Most work: Combinatorial aspectsAssume: I(X ∗) tractable to computeAssume: I(X ∗) cheap to compute (many X ∗)Simple greedy forward works well in practice . . .
So is it . . . ?
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 6 / 20
Bayesian Experimental Design
Challenges
I(X ∗) = H[P(u |y )]− H[P(u |y∗,y )
Assume: I(X ∗) tractable to compute.Only if P(u |y ) Gaussian . . .
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 7 / 20
Bayesian Experimental Design
Image Statistics
Whatever images are . . .
they are not Gaussian!
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 8 / 20
Bayesian Experimental Design
Challenges
I(X ∗) = H[P(u |y )]− H[P(u |y∗,y )
Assume: I(X ∗) tractable to compute?No: Needs approximate inferenceAssume: I(X ∗) cheap to compute (many X ∗).
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 9 / 20
Bayesian Experimental Design
Size Does Matter
1 Global covariancesScores I(X ∗) need full CovP [u |y ]
2 Massive scaleR131072 (just one slice).
3 Many timesPosterior after each design extension
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 10 / 20
Bayesian Experimental Design
Challenges
I(X ∗) = H[P(u |y )]− H[P(u |y∗,y )
Assume: I(X ∗) tractable to compute?No: Needs approximate inferenceAssume: I(X ∗) cheap to compute (many X ∗)?No: Needs new algorithms and high performance computing
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 11 / 20
Variational Bayesian Inference
Variational Bayesian Inference
Approximate inference for non-Gaussian modelsComputations driven by Gaussian inference
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 12 / 20
Variational Bayesian Inference
Variational Bayesian Inference
P(u |y ) =P(y |u)× P(u)
P(y )
Variational Inference ApproximationWrite intractable integration as optimizationRelax to tractable optimization problem
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 13 / 20
Variational Bayesian Inference
Variational Bayesian Inference
P(u |y ) =P(y |u)× P(u)
P(y )
Variational Relaxation: Bound the master function
− log P(y ) = − log∫
P(u ,y )du ≤ 12
minγ
minu∗
φ(u∗,γ)
Approximate posterior P(u |y ) by GaussianIntegration⇒ Convex optimization
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 13 / 20
Variational Bayesian Inference
Variational Bayesian Inference
P(u |y ) =P(y |u)× P(u)
P(y )
Variational Relaxation: Bound the master function
− log P(y ) = − log∫
P(u ,y )du ≤ 12
minγ
minu∗
φ(u∗,γ)
Approximate posterior P(u |y ) by GaussianIntegration⇒ Convex optimization
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 13 / 20
Variational Bayesian Inference
No Inference Without . . .
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 14 / 20
Variational Bayesian Inference
Double Loop Algorithm
Double loop algorithmInner loop optimization:Standard MAP EstimationOuter loop update:Gaussian Variances
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 15 / 20
Variational Bayesian Inference
Double Loop Algorithm
Double loop algorithmInner loop optimization:Standard MAP EstimationOuter loop update:Gaussian Variances
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 15 / 20
Variational Bayesian Inference
Tricks of the Trade
Monte Carlo Gaussian variances: Perturb&MAP Papandreou, Yuille, NIPS 2010
Inner loop: Fast first-order MAP solversWarmstarting variational optimization:Small changes after each (X ∗,y∗)
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 16 / 20
Variational Bayesian Inference
Tricks of the Trade
Monte Carlo Gaussian variances: Perturb&MAP Papandreou, Yuille, NIPS 2010
Inner loop: Fast first-order MAP solversWarmstarting variational optimization:Small changes after each (X ∗,y∗)
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 16 / 20
Experimental Results
Optimizing Cartesian MRI
Bayes Optim. VD Random Low Pass
Seeger et.al., MRM 63(1), 2010 Lustig, Donoho, Pauli, MRM 58(6), 2007 Common MRI practice
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 17 / 20
Experimental Results
Experimental Results: Test Set Errors
80 100 120 140 1601
2
3
4
5
6
7
Number phase encodes
L 2 rec
onst
ruct
ion
erro
r
axial short TE
80 100 120 140 160Number phase encodes
axial long TE
1
2
3
4
5
6
7
L 2 rec
onst
ruct
ion
erro
r
sagittal short TE
Low PassVD RandomBayes Optim
sagittal long TE
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 18 / 20
Outlook
Large Scale Bayesian Inference
Advanced Bayesian experimental designSignal acquisition optimization
Computer VisionHierarchically structured image priors Ko, Seeger, ICML 2012
Learning Image Models (fields of experts, . . . )Bayesian dictionary learningIntelligent user interfaces (Bayesian active learning)
Advanced variational inferenceSpeeding up expectation propagation Seeger, Nickisch, AISTATS 2011
Generic frameworkYou can do MAP estimation efficiently?You can do variational Bayesian inference!
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 19 / 20
Outlook
Large Scale Bayesian Inference
Advanced Bayesian experimental designSignal acquisition optimization
Computer VisionHierarchically structured image priors Ko, Seeger, ICML 2012
Learning Image Models (fields of experts, . . . )Bayesian dictionary learningIntelligent user interfaces (Bayesian active learning)
Advanced variational inferenceSpeeding up expectation propagation Seeger, Nickisch, AISTATS 2011
Generic frameworkYou can do MAP estimation efficiently?You can do variational Bayesian inference!
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 19 / 20
Outlook
People& Code
glm-ie: Toolbox by Hannes Nickisch
mloss.org/software/view/269/
Generalized sparse linear modelsMAP reconstruction and variational Bayesianinference (double loop algorithm forsuper-Gaussian bounding)Matlab 7.x, GNU Octave 3.2.x
Hannes Nickisch (now Philips Research, Hamburg)Rolf Pohmann, Bernhard Schölkopf (MPI Tübingen)Young Jun KoEmtiyaz Khan
Seeger (EPFL) Large Scale Bayesian Inference 9/12/2013 20 / 20