Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and...
-
Upload
avice-mosley -
Category
Documents
-
view
215 -
download
1
Transcript of Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and...
![Page 1: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/1.jpg)
Deducing Local Influence Neighbourhoods in Images Using Graph Cuts
Ashish Raj, Karl Young and Kailash Thakur
Assistant Professor of RadiologyUniversity of California at San Francisco, ANDCenter for Imaging of Neurodegenerative
Diseases (CIND) San Francisco VA Medical Center
email: [email protected]
Webpage: http://www.cs.cornell.edu/~rdz/SENSE.htmhttp://www.vacind.org/faculty
![Page 2: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/2.jpg)
CIND, UCSF Radiology 2
San Francisco, CA
![Page 3: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/3.jpg)
CIND, UCSF Radiology 3
Overview
We propose a new image structure called local influence neighbourhoods (LINs)
LINs are basically locally adaptive neighbourhoods around every voxel in image
Like “superpixels” Idea of LIN not new, but first principled cost
minimization approach Thus LINs allow us to probe the intermediate
structure of local features at various scales LINs were developed initially to address image
processing tasks like denoising and interpolation But as local image features they have wide
applications
![Page 4: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/4.jpg)
CIND, UCSF Radiology 4
Local neighbourhoods as intermediate image structures
Pixel-level Neighbourhood-level
12
3
Region-level
Low level High level
Too cumbersomeComputationally expensive
Not suited for pattern recognition
Prone to error propagationGreat for graph theoretic and pattern recognition
Good intermediaries between low and high levels?
![Page 5: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/5.jpg)
CIND, UCSF Radiology 5
Outline Intro to Local Influence Neighbourhoods How to compute LINs?
– Use GRAPH CUT energy minimzation Some examples of LINs in image filtering and denoising Other Applications:
– Segmentation– Using LINs for Fractal Dimension estimation– Use as features for tracking, registration
![Page 6: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/6.jpg)
CIND, UCSF Radiology 6
Local Influence Neighbourhoods A local neighbourhood around a voxel (x0, y0) is the set of voxels “close” to it
– closeness in geometric space– closeness in intensity
First attempt: use a “space-intensity box”
Definition of , arbitrary Produces disjoint, non-contiguous, “holey”, noisy neighbourhoods! Need to introduce prior expectations about contiguity We develop a principled probabilistic approach, using likelihood and prior distributions
![Page 7: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/7.jpg)
CIND, UCSF Radiology 7
Example: Binary image denoising
Suppose we receive a noisy fax:– Some black pixels in the original image
were flipped to white pixels, and some white pixels were flipped to black
We want to recover the original
input image
output image
![Page 8: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/8.jpg)
CIND, UCSF Radiology 8
Problem Constraints
Our Constraints:1. If a pixel is black (white) in the original image,
it is more likely to get the black (white) label2. Black labeled pixels tend to group together,
and white labeled pixels tend to group together
original image
good labeling bad labeling(constraint 1)
bad labeling(constraint 2)
likelihood
prior
![Page 9: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/9.jpg)
CIND, UCSF Radiology 9
Example of box vs. smoothness
![Page 10: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/10.jpg)
CIND, UCSF Radiology 10
Example of box vs. smoothness
![Page 11: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/11.jpg)
CIND, UCSF Radiology 11
A Better neighbourhood criterion1. Incorporate closeness, contiguity and smoothness assumptions2. Set up as a minimization problem3. Solve using everyone’s favourite minimization algorithm
– Simulated Annealing– (just kidding) - Graph Cuts!
A) Closeness: lets assume neighbourhoods follow Gaussian shapes around a voxel
![Page 12: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/12.jpg)
CIND, UCSF Radiology 12
A) Closeness criterion in action
![Page 13: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/13.jpg)
CIND, UCSF Radiology 13
B) Contiguity and smoothness
This is encoded via penalty terms between all neighbouring voxel pairs
p qG(x) = p,q V(xp, xq)
V(xp, xq) = distance metric
A) ClosenessB) Contiguity/smoothness
Define a binary field Fp around voxel p
s.t. 0 means not in LIN, 1 means in LIN
Bayesian interpretation: this is the log-prior for LINs
![Page 14: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/14.jpg)
CIND, UCSF Radiology 15
Markov Random Field Priors
Imposes spatial coherence (neighbouring pixels are similar)
G(x) = p,q V(xp, xq)
V(xp, xq) = distance metric
pq
Potential function is discontinuous, non-convex Potts metric is GOOD but very hard to minimize
![Page 15: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/15.jpg)
CIND, UCSF Radiology 16
Bottomline
Maximizing LIN prior corresponds to the minimization of
E(x) = Ecloseness(x) + Esmoothness(x)
MRF priors encode general spatial coherence properties of images
E(x) can be minimized using ANY available minimization algorithm
Graph Cuts can speedily solve cost functions involving MRF’s, sometimes with guaranteed global optimum.
![Page 16: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/16.jpg)
Graph Cut based Energy Minimization
![Page 17: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/17.jpg)
CIND, UCSF Radiology 18
How to minimize E?
Graph cuts have proven to be a very powerful tool for minimizing energy functions like this one
First developed for stereo matching– Most of the top-performing algorithms for stereo rely
on graph cuts Builds a graph whose nodes are image pixels, and
whose edges have weights obtained from the energy terms in E(x)
Minimization of E(x) is reduced to finding the minimum cut of this graph
![Page 18: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/18.jpg)
CIND, UCSF Radiology 19
Minimum cut problem
Mincut/maxflow problem:– Find the cheapest way to
cut the edges so that the “source” is separated from the “sink”
– Cut edges going from source side to sink side
– Edge weights now represent cutting “costs”
a cut C
“source”
A graph with two terminals
S T
“sink”
![Page 19: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/19.jpg)
CIND, UCSF Radiology 20
Graph construction
Links correspond to terms in energy function Single-pixel terms are called t-links Pixel-pair terms are called n-links A Mincut is equivalent to a binary segmentation
I.e. mincut minimizes a binary energy function
![Page 20: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/20.jpg)
CIND, UCSF Radiology 21
Table1: Edge costs of induced graph
n-links
s
tt-
link
t-link
![Page 21: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/21.jpg)
CIND, UCSF Radiology 22
Graph Algorithm
Repeat graph mincut for each voxel p
![Page 22: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/22.jpg)
CIND, UCSF Radiology 23
Examples of Detected LINs
![Page 23: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/23.jpg)
CIND, UCSF Radiology 24
Results: Most Popular LINs
![Page 24: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/24.jpg)
CIND, UCSF Radiology 25
Filtering with LINs
Use LINs to restrict effect of filter– Convolutional filters:
– Rank order filter:
=
![Page 25: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/25.jpg)
CIND, UCSF Radiology 26
Maximum filter using LINs
![Page 26: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/26.jpg)
CIND, UCSF Radiology 27
Median filter using LINs
![Page 27: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/27.jpg)
CIND, UCSF Radiology 28
EM-style Denoising algorithm
Likelihood for i.i.d. Gaussian noise:
Image prior:
Maximize the posterior:
Noise model: O = I + n
![Page 28: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/28.jpg)
CIND, UCSF Radiology 29
Bayesian (Maximum a Posteriori) Estimate
Bayes Theorem:
Pr(x|y) = Pr(y|x) . Pr(x)
Pr(y)
likelihood
priorposterior
Here x is LIN, y is observed image Bayesian methods maximize the posterior probability:
Pr(x|y) Pr(y|x) . Pr(x)
![Page 29: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/29.jpg)
CIND, UCSF Radiology 30
EM-style image denoisingJoint maximization is challengingWe propose EM-style approach: Start with Iterate:
We show that
![Page 30: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/30.jpg)
CIND, UCSF Radiology 31
Results: LIN-based Image Denoising
![Page 31: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/31.jpg)
CIND, UCSF Radiology 32
Results: Bike image
![Page 32: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/32.jpg)
CIND, UCSF Radiology 33
Table1: Denoising Results
![Page 33: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/33.jpg)
Other Applications of LINs
LINs can be used to probe scale-space of image data
– By varying scale parameters x and n Measuring fractal dimensions of brain images Hierarchical segmentation – “superpixel” concept Use LINs as feature vectors for
– image registration – Object recognition– Tracking
![Page 34: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/34.jpg)
CIND, UCSF Radiology 35
Hierarchical segmentation
Begin with LINs at fine scale Hierarchically fuse finer LINs to obtain coarser LINS
segmentation
![Page 35: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/35.jpg)
How to measure Fractal Dimension using LINs?
How LINs vary with changing x and n depends on local image complexity Fractal dimension is a stable measure of complexity of multidimensional structuresThus LINs can be used to probe the multi-scale structure of image data
![Page 36: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/36.jpg)
CIND, UCSF Radiology 37
FD using LINs For each voxel p, for each value of x, n:
count the number N of voxels included in Bp
.CP1 CP2 ln x
ln N
extend to (x , n) plane
phase transition
Slope of each segment = local fractal dimension
![Page 37: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/37.jpg)
CIND, UCSF Radiology 38
Possible advantages of LIN over current techniques
LINs provide FD for each voxel Captures the FD of local regions as well as global Ideal for directional structures and oriented
features at various scales Far less susceptible to noise
– (due to explicit intensity scale n which can be tuned to the noise level)
Enables the probing of phase transitions
![Page 38: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/38.jpg)
CIND, UCSF Radiology 39
Possible Discriminators of Neurodegeneration
Fractal measures may provide better discriminators of neurodegeneration (Alzheimer’s Disease, Frontotemporal Dementia, Mild Cognitive Disorder, Normal Aging, etc)
Possibilities:– Mean (overall) FD -- D(0)– Critical points, phase transitions in (x, n) plane– More general Renyi dimensions D(q) for q ¸ 1– Summary image feature f() D(q)– Phase transitions in f()
Fractal structures can be characterized by dimensions D(q), summary f() and various associated critical points
These quantities may be efficiently probed by the Graph Cut –based local influence neighbourhoods
These fractal quantities may provide greater discriminability between normal, AD, FTD, etc.
![Page 39: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/39.jpg)
CIND, UCSF Radiology 40
Summary
We proposed a general method of estimating local influence neighbourhoods
Based on an “optimal” energy minimization approach
LINs are intermediaries between purely pixel-based and region-based methods
Applications include segmentation, denoising, filtering, recognition, fractal dimension estimation, …
… in other words, Best Thing Since Sliced Bread
![Page 40: Deducing Local Influence Neighbourhoods in Images Using Graph Cuts Ashish Raj, Karl Young and Kailash Thakur Assistant Professor of Radiology University.](https://reader036.fdocuments.net/reader036/viewer/2022070403/56649f315503460f94c4ce39/html5/thumbnails/40.jpg)
Ashish RajCIND, UCSF
email: [email protected]
Webpage: http://www.cs.cornell.edu/~rdz/SENSE.htmhttp://www.vacind.org/faculty
Deducing Local Influence Neighbourhoods in Images Using Graph
Cuts