Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots...
Transcript of Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots...
![Page 1: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/1.jpg)
Photo Enhancement with Inpainting
Gustaf Kylberg1, Paul Nemes2, Martin Maska3, andSandor Fazekas4
1Uppsala University, Sweden2Technical University of Cluj-Napoca, Romania
3Masaryk University, Czech Republic4Computer and Automation Institute (SZTAKI), Hungary
SSIP 2008
Project PresentationJuly 15, 2008
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 2: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/2.jpg)
The group
Gustaf Kylberg, Paul Nemes, Martin Maska, and Sandor Fazekas
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 3: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/3.jpg)
Photo manipulations
Enhance qualityRemove scratches and spots“Highly adaptive median filter”Photo restoration
Enhance contentRemove unwanted parts
Objects, peopleWires, graffitiTV logo, text
Change the appearance of peopleGlassesMustache, beardTattoo
Make people look nicerRemove wrinkles, freckles
. . .Nikolai Yezhov removed
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 4: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/4.jpg)
Work flow of the proposed method
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 5: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/5.jpg)
Defect detection
Color based segmentationHough transform
Detect linesLong scratchesWires (disturbing the view)
Detect circlesCircular blobsDirty spots
Parallel close edgesThreshold on large gradientsThin defected areas assumed
Semi-automatic defect detectionRefine a manually selected regionSuitable for thin scratches and small blobs
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 6: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/6.jpg)
Implemented inpainting methods
Input and mask1
Output of inpainting
Isotropic diffusion Anisotropic diffusion Distance transform
1Intentionally so bad :)G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 7: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/7.jpg)
Inpainting with diffusion
Diffusion process
∂tρ =∑α,β
∂α
(Dαβ∂βρ
)(1)
Applied to inpaint a region Ω (defined by µ(Ω) = 0)ρ and the derivatives of ρ are known outside the maskAssume some initial value inside the maskSolve Eq. (1) on Ω with fixed boundary conditionsLet the system evolve until a steady state is reached∑
α,β
∂α
(Dαβ∂βρ
)= 0 (2)
Result = Smooth patchTexture synthesized in an additional step
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 8: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/8.jpg)
Inpainting with diffusion
Isotropic diffusionDαβ = Dδαβ (3)
Tends to smooth away edgesAnisotropic diffusion
Eigenvectors parallel and perpendicular to image gradientsDiffusion only perpendicular to gradient⇒ Keeps edges
Can be applied to multichannel images tooThe diffusivity Dαβ should be the same for all channelsDefine a norm in the “color space”Diffusion along isophote lines
Convergence speeded up with a multiscale approachBuild a Gaussian pyramid and apply diffusion on all scalesPropagate information for lower to higher scalesStart diffusion form the result obtained on lower scale
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 9: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/9.jpg)
Discretization
Isotropic diffusion
ρt+1 − ρt = D(ρt − 4ρt+1) (4)
where
ρ =
0 1 01 0 10 1 0
? ρ (5)
Information taken from neighboring pixelsIterative solution
ρt+1 =ρt + Dρt
1 + 4D(6)
Anisotropic diffusion discretized in a similar wayMultiscale approach can be applied to speed up convergence
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 10: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/10.jpg)
Inpainting Based on Distance Transform
Distance to closest non-zero pixel
Input OutputApply distance transform on the mask µ→ d(µ)
Update image within the mask
ρ(r)← ρ
(r + d(µ)
∇d(µ)
‖∇d(µ)‖
)(7)
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 11: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/11.jpg)
Applications
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 12: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/12.jpg)
Image restoration
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 13: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/13.jpg)
Image restoration
Original Enhanced
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 14: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/14.jpg)
Image restoration
Original Enhanced
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 15: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/15.jpg)
Image restoration
Original Enhanced
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 16: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/16.jpg)
Image restoration
Original Enhanced
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 17: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/17.jpg)
Removing unwantedimage regions
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 18: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/18.jpg)
Removing objects
Original Enhanced
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 19: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/19.jpg)
Removing text
Original Enhanced
Fully automatic!Using color based segmentation
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 20: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/20.jpg)
Removing wires
Original Enhanced
Fully automatic!Using Hough transform
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 21: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/21.jpg)
Removing wires
Original Enhanced
Fully automatic!Using Hough transform
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 22: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/22.jpg)
Removing wires
Original Enhanced
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 23: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/23.jpg)
Making people look niceron photographs
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 24: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/24.jpg)
Removing freckles
Original Enhanced
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 25: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/25.jpg)
Removing freckles
Original Enhanced
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 26: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/26.jpg)
Removing wrinkles
Original Enhanced
High frequency components of the texture kept
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 27: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/27.jpg)
Removing wrinkles
Original Enhanced
High frequency components of the texture kept
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 28: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/28.jpg)
Removing wrinkles
Original Enhanced
High frequency components of the texture kept
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 29: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/29.jpg)
Removing wrinkles
Original Enhanced
High frequency components of the texture kept
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 30: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/30.jpg)
Changing the appearanceof people
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 31: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/31.jpg)
Removing glasses and mustache
Original Enhanced
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 32: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/32.jpg)
Removing glasses and mustache
Original Enhanced
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 33: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/33.jpg)
Limitations!
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 34: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/34.jpg)
Limitations
Original Enhanced
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting
![Page 35: Photo Enhancement with InpaintingPhoto manipulations Enhance quality Remove scratches and spots “Highly adaptive median filter” Photo restoration Enhance content Remove unwanted](https://reader036.fdocuments.net/reader036/viewer/2022062507/5fdb53991c2be671bd42b778/html5/thumbnails/35.jpg)
Thank you for yourattention!
G. Kylberg, P. Nemes, M. Maska, and S. Fazekas Photo Enhancement with Inpainting