Software image stabilization comes of age
description
Transcript of Software image stabilization comes of age
![Page 1: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/1.jpg)
Software image stabilization comes of age
Michal Šorel
![Page 2: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/2.jpg)
Removing camera motion blur
• Alternative to OIS (optical image stabilization) systems
• Camera motion, not subject motion
![Page 3: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/3.jpg)
Talk outline
• How to describe the blur? (velocity field, space-variant PSF )
• Common setups– Single blurred image– Multiple blurred images– Multiple underexposed/noisy images– One blurred and one underexposed image
• Perspectives
![Page 4: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/4.jpg)
Stabilizer of 3D camera rotation
• Rigid body – 6 degrees of freedom
• Natural coordinate system
• Camera rotation is mostly dominant
• Blur is independent of scene depth (that is why optical image stabilizers can work)
![Page 5: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/5.jpg)
Camera rotates downwards ↓
![Page 6: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/6.jpg)
Velocity field
![Page 7: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/7.jpg)
Rotation about optical axis
![Page 8: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/8.jpg)
General 3D rotation
![Page 9: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/9.jpg)
• PSF h ... depends on position (x,y)
• Generalized convolution
• Convolution case – h is called convolution kernel or convolution mask
Space-variant PSF
![Page 10: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/10.jpg)
PSF for camera shake
h(s,t; x2,y2)
h(s,t; x1,y1)
h(s,t; x3,y3)
(x1,y1)(x2,y2)
(x3,y3)
![Page 11: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/11.jpg)
Talk outline
• How to describe the blur? (velocity field, space-variant PSF )
• Common setups– Single blurred image– Multiple blurred images– Multiple underexposed/noisy images– One blurred and one underexposed image
• Perspectives
![Page 12: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/12.jpg)
Single image deblurring
• Rob Fergus building on the work of James Miskin
• Bayesian approach• Approximation – conditional distributions of
PSF and image are considered independent• Priors on image gradients and blur kernels as
a mixture of Gaussians and exponential functions
![Page 13: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/13.jpg)
Bayesian approach
![Page 14: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/14.jpg)
Image prior
Intensity histogram Gradient histogram
![Page 15: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/15.jpg)
Image prior
Gradient histogram
![Page 16: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/16.jpg)
Image priors
Tikhonov regularization
TV regularization
![Page 17: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/17.jpg)
PSF prior
![Page 18: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/18.jpg)
Functional to minimize
![Page 19: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/19.jpg)
Rob Fergus (Example I)
![Page 20: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/20.jpg)
Rob Fergus (Example II)
![Page 21: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/21.jpg)
Talk outline
• How to describe the blur? (velocity field, space-variant PSF )
• Common setups– Single blurred image– Multiple blurred images– Multiple underexposed/noisy images– One blurred and one underexposed image
• Perspectives
![Page 22: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/22.jpg)
Multiple blurred images
• Multichannel blind deconvolution
• Convolution model of blurring
• Solved by minimization of
![Page 23: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/23.jpg)
Multiple blurred images
![Page 24: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/24.jpg)
Talk outline
• How to describe the blur? (velocity field, space-variant PSF )
• Common setups– Single blurred image– Multiple blurred images– Multiple underexposed/noisy images– One blurred and one underexposed image
• Perspectives
![Page 25: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/25.jpg)
Multiple noisy images
• Noise variance of the sum of N images is the same as of the original image
• The difficult part is registration
• Main problem slow read-out
N imagestime t’=t/N
noise variance σ2/N
1 imagetime t =1s
noise variance σ2
![Page 26: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/26.jpg)
Talk outline
• How to describe the blur? (velocity field, space-variant PSF )
• Common setups– Single blurred image– Multiple blurred images– Multiple underexposed/noisy images– One blurred and one underexposed image
• Perspectives
![Page 27: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/27.jpg)
Blurred + underexposed image
• noisy ~ underexposed (exposure time changes contrast)
• patented in 2006
• since 2006 - several papers assuming convolution model
• our space-variant version sent to BMVC 2008
![Page 28: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/28.jpg)
Deblurring algorithm
Blurredimage
Noisyimage
![Page 29: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/29.jpg)
Image registration
• Small change of camera position – small stereo base
• Static parts of the scene can be modelled by projective tranform found by RANSAC
• Lens distortion can be neglected
• Less important parts of scene can move
![Page 30: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/30.jpg)
Blurred + underexposed results
![Page 31: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/31.jpg)
Blur kernel adjustment
• Regions lacking texture
• Regions of pixel saturation
![Page 32: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/32.jpg)
Restoration
• Minimization of functional
• PSF h interpolated from estimated convolution kernels
![Page 33: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/33.jpg)
Shopping center (details)
![Page 34: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/34.jpg)
Bookcase example
![Page 35: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/35.jpg)
Bookcase (details)
![Page 36: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/36.jpg)
Summary/Perspectives
• Denoising – artifacts or readout problems• Single image approach – takes time,
imprecise PSF, unable to distinguish intentional depth of focus, limited to convolution model
• Multiple blurred images – computationally expensive, fewer artifacts
• Blurred + underexposed image – fastest,algorithm for space-variant deblurring exists
![Page 37: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/37.jpg)
Discussion, questions...
Michal ŠorelInstitute of Information Theory and Automation of the ASCR
![Page 38: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/38.jpg)
Project ?
• Segmentation of moving objects from noisy-blurry image pair
• Restoration from one noisy and two blurred images
• Space-variant single image deblurring
• Combination with demosaicing to get full resolution image
• Implementation in fix-point arithmetic? ...
![Page 39: Software image stabilization comes of age](https://reader035.fdocuments.net/reader035/viewer/2022062500/56815166550346895dbf9395/html5/thumbnails/39.jpg)
Cause of blur
• Long exposure time -> apparent image motion more than about half pixel
• Why do we need long time?– enough photons to avoid quantization and shot
noise– Small aperture to achieve high depth of focus– Small aperture because of tele lens
construction limitations