High-quality Video Denoising for Motion-based Exposure Control
description
Transcript of High-quality Video Denoising for Motion-based Exposure Control
![Page 1: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/1.jpg)
High-quality Video Denoising for Motion-based Exposure Control
Travis Portz, Li Zhang, and Hongrui Jiang
University of Wisconsin–Madison
Presented by Brandon M. Smith
IWMV 2011
![Page 2: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/2.jpg)
University of Wisconsin-Madison 2
Motivation
• Motion blur from dynamic scenesor handheld cameras
• Example: mobile phone camera– Small and light → easy to introduce handshake blur– Small aperture →
IWMV 2011
Long ExposureLow Gain
Motion Blur
Short ExposureHigh Gain
Noise
![Page 3: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/3.jpg)
University of Wisconsin-Madison 3
Deblurring vs. Denoising• Blind image deconvolution is ill-posed
• L. Zhang, A. Deshpande, X. Chen, Denoising versus Deblurring: HDR Techniques Using Moving Cameras, CVPR 2010
IWMV 2011
? ⊗ ? =
Blurry Deblurred Noisy Denoised
![Page 4: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/4.jpg)
University of Wisconsin-Madison 4
Motion-based Auto Exposure
• Not all frames need shorter exposure time• Estimate motion in previous frames to set
exposure for next frame• Some patents on this idea, not many academic
papers• Single-shot versions in Canon Powershot,
Nikon Coolpix
IWMV 2011
![Page 5: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/5.jpg)
University of Wisconsin-Madison 5
Motion-based Auto Exposure
IWMV 2011
Constant exposure time(Canon EOS 7D)
Motion-based exposure(Point Grey Grasshopper)
![Page 6: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/6.jpg)
University of Wisconsin-Madison 6
Motion-based Auto Exposure
IWMV 2011
Constant exposure time(Canon EOS 7D)
Motion-based exposure(Point Grey Grasshopper)
![Page 7: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/7.jpg)
University of Wisconsin-Madison 7
Motivation
• Denoise videos captured using motion-based auto exposure
• Exploit varying noise levels to obtain higher quality results
IWMV 2011
![Page 8: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/8.jpg)
University of Wisconsin-Madison 8
Inspiration: Spatial Image Denoising
• BM3D– Dabov, et al., Image denoising by sparse 3D
transform-domain collaborative filtering. TIP August 2007
IWMV 2011
Stack of similar blocks
3D Filter
Stack of block
estimatesWeighted Average
Denoised block
![Page 9: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/9.jpg)
University of Wisconsin-Madison 9
Inspiration: Spatiotemporal Video Denoising
• Liu and Freeman, A High-Quality Video Denoising Algorithm based on Reliable Motion Estimation. ECCV 2010
IWMV 2011
• Non-local means using approximate K-nearest neighbors and optical flow
![Page 10: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/10.jpg)
University of Wisconsin-Madison 10
Our Approach
Spatial Methods+ Abundant locally similar structure− Unique local patches
Motion-compensated Filtering+ Unique local patches+ Reliable motion estimation− Abundant locally similar
structure– Unreliable motion estimation
IWMV 2011
• Combine spatial and motion-compensated (temporal) denoising based on reliability of flow
![Page 11: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/11.jpg)
University of Wisconsin-Madison 11
Temporal Denoising
• Filter along optical flow– Temporal window of ±H frames– Hierarchical block matching
to handle large displacements• Not all flows are reliable– Pixels from unreliable flows
will degrade results– Solution: detect unreliable flows
and exclude them from filter
IWMV 2011
u
v
![Page 12: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/12.jpg)
University of Wisconsin-Madison 12
Flow Reliability Estimation
Consistentvij = -vji
Inconsistent
vij ≠ -vji
Reliable if:
Forward/backward consistency
IWMV 2011
![Page 13: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/13.jpg)
University of Wisconsin-Madison 13
Temporal Denoising• Set of frames with consistent flow:
• Filter along flow:
• Weight function:
• Threshold:
IWMV 2011
![Page 14: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/14.jpg)
University of Wisconsin-Madison 14
Combining Spatial/Temporal Denoising
• Quality of temporal result is based on number of samples used in filter
• Interpolate between spatial and temporal results:
IWMV 2011
![Page 15: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/15.jpg)
University of Wisconsin-Madison 15
Combining Spatial/Temporal Denoising
Temporal Weights
ConsistencyOptical Flow
(forward/backward)
i+1
i-2i-1
⋮
⋮i+H
i - H
IWMV 2011
![Page 16: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/16.jpg)
University of Wisconsin-Madison 16
Combining Spatial/Temporal Denoising
• Example weight map
IWMV 2011
![Page 17: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/17.jpg)
University of Wisconsin-Madison 17
Simulation
• Sliding window over panoramic image
– Temporally-varying motion– AWGN with noise level proportional to window
displacement
IWMV 2011
![Page 18: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/18.jpg)
University of Wisconsin-Madison 18
Results on Synthetic Videos
IWMV 2011
Constant ExposureTime
Noisy Input CBM3D
Liu and Freeman Our Method Ground Truth
![Page 19: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/19.jpg)
University of Wisconsin-Madison 19
Results on Synthetic Videos
IWMV 2011
Constant ExposureTime
Noisy Input CBM3D
Liu and Freeman Our Method Ground Truth
![Page 20: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/20.jpg)
University of Wisconsin-Madison 20
Results on Synthetic Videos• Constant short exposure vs. motion-based exposure
IWMV 2011
![Page 21: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/21.jpg)
University of Wisconsin-Madison 21
Real Video
• Motion-base AE– Global, hierarchical image registration between
previous two frames• Scenes with approximately planar motion
IWMV 2011
![Page 22: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/22.jpg)
University of Wisconsin-Madison 22
Results on Real Video
IWMV 2011
Input
BM3D
Liu and Freeman
Our Method
![Page 23: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/23.jpg)
University of Wisconsin-Madison 23
Results on Real Video
IWMV 2011
Constant exposure time(Canon EOS 7D)
Motion-based exposure(Point Grey Grasshopper) Our denoising result
![Page 24: High-quality Video Denoising for Motion-based Exposure Control](https://reader036.fdocuments.net/reader036/viewer/2022081604/56815e8b550346895dcd1b0f/html5/thumbnails/24.jpg)
University of Wisconsin-Madison 24
Future Work
• Different weighting schemes– Temporal coherence
• Dynamic scenes, spatially varying motion• Higher frame rates → more accurate flow– More AE latency → blurred frames– Use noisy/high-quality frames to enhance blurred
frames
IWMV 2011