A Gentle Introduction to Bilateral Filtering and its Applications
Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge...
Transcript of Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge...
![Page 1: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/1.jpg)
Edge Preserving Filtering
Median Filter
Bilateral Filter
Shai Avidan
Tel-Aviv University
![Page 2: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/2.jpg)
Slide Credits� (partial list)
• Rick Szeliski
• Steve Seitz
• Alyosha Efros
• Yacov Hel-Or
• Marc Levoy
• Bill Freeman
• Fredo Durand
• Sylvain Paris
![Page 3: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/3.jpg)
A Gentle Introduction
to Bilateral Filtering
and its Applications
“Fixing the Gaussian Blur”:
the Bilateral Filter
Sylvain Paris – MIT CSAIL
![Page 4: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/4.jpg)
Box Average
average
input
square neighborhood
output
![Page 5: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/5.jpg)
normalized
box function
sum over
all pixels q
intensity at
pixel qresult at
pixel p
Equation of Box Average
��
��S
IBIBAq
qp qp )(][ �
0
![Page 6: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/6.jpg)
Square Box Generates Defects
• Axis-aligned streaks
• Blocky results
input
output
![Page 7: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/7.jpg)
unrelated
pixels
unrelated
pixels
related
pixels
Box Profile
pixel
position
pixel
weight
![Page 8: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/8.jpg)
Strategy to Solve these
Problems
• Use an isotropic (i.e. circular) window.
• Use a window with a smooth falloff.
box window Gaussian window
![Page 9: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/9.jpg)
Gaussian Blur
average
input
per-pixel multiplication
output*
![Page 10: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/10.jpg)
input
![Page 11: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/11.jpg)
box average
![Page 12: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/12.jpg)
Gaussian blur
![Page 13: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/13.jpg)
normalized
Gaussian function
Equation of Gaussian Blur
� ���
��S
IGIGBq
qp qp ||||][ �
Same idea: weighted average of pixels.
0
1
![Page 14: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/14.jpg)
unrelated
pixels
unrelated
pixels
uncertain
pixels
uncertain
pixels
related
pixels
Gaussian Profile
pixel
position
pixel
weight��
���
��
2
2
2exp
2
1)(
����
xxG
![Page 15: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/15.jpg)
size of the window
Spatial Parameter
� ���
��S
IGIGBq
qp qp ||||][ �
small � large �
input
limited smoothing strong smoothing
![Page 16: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/16.jpg)
Blur Comes from
Averaging across Edges
*
*
*
input output
Same Gaussian kernel everywhere.
![Page 17: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/17.jpg)
Bilateral Filter
No Averaging across Edges
*
*
*
input output
The kernel shape depends on the image content.
[Aurich 95, Smith 97, Tomasi 98]
![Page 18: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/18.jpg)
space weight
not new
range weight
I
new
normalization
factor
new
Bilateral Filter Definition:
an Additional Edge Term
� � � ���
���S
IIIGGW
IBFq
qqp
p
p qp ||||||1
][rs ��
Same idea: weighted average of pixels.
![Page 19: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/19.jpg)
Illustration a 1D Image
• 1D image = line of pixels
• Better visualized as a plot
pixel
intensity
pixel position
![Page 20: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/20.jpg)
space
Gaussian Blur and Bilateral Filter
space rangenormalization
Gaussian blur
� � � ���
���S
IIIGGW
IBFq
qqp
p
p qp ||||||1
][rs ��
Bilateral filter[Aurich 95, Smith 97, Tomasi 98]
space
space
range
p
p
q
q
� ���
��S
IGIGBq
qp qp ||||][ �
![Page 21: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/21.jpg)
Space and Range Parameters
• space �s : spatial extent of the kernel, size of
the considered neighborhood.
• range �r : “minimum” amplitude of an edge
� � � ���
���S
IIIGGW
IBFq
qqp
p
p qp ||||||1
][rs ��
![Page 22: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/22.jpg)
How to Set the Parameters
Depends on the application. For instance:
• space parameter: proportional to image size
– e.g., 2% of image diagonal
• range parameter: proportional to edge
amplitude
– e.g., mean or median of image gradients
• independent of resolution and exposure
![Page 23: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/23.jpg)
A Few
More Advanced
Remarks
![Page 24: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/24.jpg)
Bilateral Filter Crosses Thin Lines• Bilateral filter averages across
features thinner than ~2�s
• Desirable for smoothing: more pixels = more robust
• Different from diffusion that stops at thin lines
close-up kernel
![Page 25: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/25.jpg)
Iterating the Bilateral Filter
• Generate more piecewise-flat images
• Often not needed in computational
photo.
][ )()1( nn IBFI ��
![Page 26: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/26.jpg)
input
![Page 27: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/27.jpg)
1 iteration
![Page 28: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/28.jpg)
2 iterations
![Page 29: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/29.jpg)
4 iterations
![Page 30: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/30.jpg)
Bilateral Filtering Color Images
� � � ���
���S
IIIGGW
IBFq
qqp
p
p qp ||||||1
][rs ��
� � � ���
���S
GGW
IBFq
qqp
p
p CCCqp ||||||||1
][rs ��
For gray-level images
For color images
intensity difference
color difference
The bilateral filter isThe bilateral filter is
extremely easy to adapt to your need.extremely easy to adapt to your need.
scalar
3D vector
(RGB, Lab)
input
output
![Page 31: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/31.jpg)
Applications
• Image denoising
• HDR Compression
– Key idea:
– Break image into base and detail layers
– Compress base
– Recompose image
![Page 32: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/32.jpg)
Fast Implementation: 3D Kernel
• Idea: represent image data such that the weights
depend only on the distance between points
[Paris and Durand 06]
pixel
intensity
pixel position
1D image
Plot
I = f ( x )
far in range
close in space
![Page 33: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/33.jpg)
1st Step: Re-arranging
Symbols� � � �
� � � ��
�
�
�
���
���
S
S
IIGGW
IIIGGW
IBF
q
qpp
q
qqp
p
p
qp
qp
||||||
||||||1
][
rs
rs
��
��
� � � �
� � � � 1||||||
||||||][
rs
rs
�
�
�
�
���
���
S
S
IIGGW
IIIGGIBFW
q
qpp
q
qqppp
qp
qp
��
��
Multiply first equation by Wp
![Page 34: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/34.jpg)
1st Step: Summary
• Similar equations
• No normalization factor anymore
• Don’t forget to divide at the end
� � � �
� � � � 1||||||
||||||][
rs
rs
�
�
�
�
���
���
S
S
IIGGW
IIIGGIBFW
q
qpp
q
qqppp
qp
qp
��
��
![Page 35: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/35.jpg)
2nd Step: Higher-dimensional Space
pp
space
range
• “Product of two Gaussians” = higher dim.
Gaussian
![Page 36: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/36.jpg)
2nd Step: Higher-dimensional Space
pp
space
range
• 0 almost everywhere, I at “plot location”
![Page 37: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/37.jpg)
2nd Step: Higher-dimensional Space
pp
• 0 almost everywhere, I at “plot location”
• Weighted average at each point = Gaussian blur
![Page 38: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/38.jpg)
2nd Step: Higher-dimensional Space
pp
• 0 almost everywhere, I at “plot location”
• Weighted average at each point = Gaussian blur
• Result is at “plot location”
![Page 39: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/39.jpg)
�������������� �������
������� ��
������
����
������
�������
���������
������
������
�������
���������
������
New num. scheme:
• simple operations
• complex space
![Page 40: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/40.jpg)
�������������� �������
���������� ����
������
����
�����������������
������������
����� �
������ ��
����� �
������ ��
Strategy:
downsampled
convolution
Conceptual view,
not exactly
the actual algorithm
![Page 41: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/41.jpg)
The Algorithm
![Page 42: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/42.jpg)
NEW IDEA : ‘Joint’ or ‘Cross’
Bilateral’ Petschnigg(2004) and
Eisemann(2004)Bilateral � two kinds of weights
NEW : get them from two kinds of images.
• Smooth image A pixels locally, but
• Limit to ‘similar regions’ of image B
Why do this? To get ‘best of both
images’
![Page 43: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/43.jpg)
Ordinary Bilateral Filter
Bilateral � two kinds of weights, one image A :
� � � ���
���S
AAAGGW
ABFq
qqp
p
p qp ||||||1
][rs ��
cc
ss
Image A:
������������
��������
f(x)f(x)xx
![Page 44: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/44.jpg)
‘Joint’ or ‘Cross’ Bilateral Filter
NEW: two kinds of weights, two images
� � � ���
���S
ABBGGW
ABFq
qqp
p
p qp ||||||1
][rs ��
cc
ss
A: Noisy, dim(ambient image)
cc
ss
B: Clean,strong
(Flash image)
![Page 45: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/45.jpg)
Image A: Warm, shadows, but too Noisy(too dim for a good quick photo)(too dim for a good quick photo)
No-flash
![Page 46: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/46.jpg)
Image B: Cold, Shadow-free, Clean(flash: simple light, ALMOST no shadows)(flash: simple light, ALMOST no shadows)
![Page 47: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/47.jpg)
MERGE BEST OF BOTH: apply
‘Cross Bilateral’ or ‘Joint Bilateral’
![Page 48: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/48.jpg)
(it really is much better!)
![Page 49: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/49.jpg)
Video Enhancement Using
Per Pixel Exposures (Bennett, 06)
From this video:
ASTA: Adaptive
SSpatio-
TTemporal
Accumulation Filter
![Page 50: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/50.jpg)
ASTA
Replace pixel difference with a general dissimilarity measure
D(x,x)=0, D(x,y)=D(y,x)
![Page 51: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/51.jpg)
Shot noise
![Page 52: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/52.jpg)
A new dissimilarity measure
Instead of comparing pixel intensities, look at their local
spatial neighborhood
![Page 53: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/53.jpg)
• Raw Video Frame:
(from FIFO center)
• Histogram stretching;
(estimate gain for
each pixel)
• ‘Mostly Temporal’ Bilateral Filter:
– Average recent similar values,
– Reject outliers (avoids ‘ghosting’), spatial avg as needed
– Tone Mapping
The Process for One Frame
![Page 54: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/54.jpg)
The Process for One Frame
• Raw Video Frame:
(from FIFO center)
• Histogram stretching;
(estimate gain for
each pixel)
• ‘Mostly Temporal’ Bilateral Filter:
– Average recent similar values,
– Reject outliers (avoids ‘ghosting’), spatial avg as needed
– Tone Mapping
![Page 55: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/55.jpg)
The Process for One Frame
• Raw Video Frame:
(from FIFO center)
• Histogram stretching;
(estimate gain for
each pixel)
• ‘Mostly Temporal’ Bilateral Filter:
– Average recent similar values,
– Reject outliers (avoids ‘ghosting’), spatial avg as needed
– Tone Mapping
(color: # avg’ pixels)
![Page 56: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/56.jpg)
The Process for One Frame
• Raw Video Frame:
(from FIFO center)
• Histogram stretching;
(estimate gain for
each pixel)
• ‘Mostly Temporal’ Bilateral Filter:
– Average recent similar values,
– Reject outliers (avoids ‘ghosting’), spatial avg as needed
– Tone Mapping
![Page 57: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/57.jpg)
Bilateral Filter Variant: Mostly Temporal
• FIFO for Histogram-stretched video
– Carry gain estimate for each pixel;
– Use future as well as previous values;
• Expanded Bilateral Filter Methods:
– Static scene? Temporal-only avg. works well
– Motion? Bilateral rejects outliers: no ghosts!
• Generalize: ‘Dissimilarity’ (not just || Ip – Iq ||2)
• Voting: spatial filter de-noises motion
![Page 58: Edge Preserving Filtering Median Filter Bilateral Filteripapps/Slides/lecture07.pdf · Edge Preserving Filtering Median Filter Bilateral Filter Shai Avidan Tel-Aviv University](https://reader033.fdocuments.net/reader033/viewer/2022041716/5e4b6cdfa79cc30b5f4d9a15/html5/thumbnails/58.jpg)
Multispectral Bilateral Video Fusion
(Bennett,07)
• Result:
– Produces watchable result from unwatchable input
–– VERYVERY robust; accepts almost any dark video;
– Exploits temporal coherence to emulate
Low-light HDR video, without special equipment