Compression and Gradient Domain€¦ · Livingstone, Vision and Art: The Biology of Seeing....
73
Compression and Gradient Domain CS194: Image Manipulation, Comp. Vision, and Comp. Photo Alexei Efros, UC Berkeley, Spring 2020
Transcript of Compression and Gradient Domain€¦ · Livingstone, Vision and Art: The Biology of Seeing....
Compression and Gradient Domain
CS194: Image Manipulation, Comp. Vision, and Comp. PhotoAlexei Efros, UC Berkeley, Spring 2020
Da Vinci and Peripheral Vision
https://en.wikipedia.org/wiki/Speculations_about_Mona_Lisa#Smile
Leonardo playing with peripheral vision
Livingstone, Vision and Art: The Biology of Seeing
Frequency Domain and Perception
Campbell-Robson contrast sensitivity curve
Lossy Image Compression (JPEG)
Block-based Discrete Cosine Transform (DCT)
Using DCT in JPEG The first coefficient B(0,0) is the DC component,
the average intensityThe top-left coeffs represent low frequencies,
the bottom right – high frequencies
Image compression using DCTQuantize
• More coarsely for high frequencies (which also tend to have smaller values)
• Many quantized high frequency values will be zero
Encode• Can decode with inverse dct
Quantization table
Filter responses
Quantized values
JPEG Compression SummarySubsample color by factor of 2
• People have bad resolution for color
Split into blocks (8x8, typically), subtract 128For each block
a. Compute DCT coefficientsb. Coarsely quantize
– Many high frequency components will become zeroc. Encode (e.g., with Huffman coding)
http://en.wikipedia.org/wiki/YCbCrhttp://en.wikipedia.org/wiki/JPEG
Block size in JPEG
Block size• small block
– faster – correlation exists between neighboring pixels
• large block– better compression in smooth regions
• It’s 8x8 in standard JPEG
JPEG compression comparison
89k 12k
Pyramid Blending
Gradient Domain vs. Frequency DomainIn Pyramid Blending, we decomposed our
images into several frequency bands, and transferred them separately• But boundaries appear across multiple bands
But what about representation based on derivatives (gradients) of the image?:• Derivatives represent local changes (across all frequencies) • No need for low-res image
– captures everything (up to a constant)• Blending/Editing in Gradient Domain:
– Differentiate– Copy / Blend / edit / whatever– Reintegrate
Gradients vs. Pixels
Gilchrist Illusion (c.f. Exploratorium)
White?
White?
White?
Drawing in Gradient Domain
James McCann & Nancy PollardReal-Time Gradient-Domain Painting, SIGGRAPH 2009
(paper came out of this class!)
http://www.youtube.com/watch?v=RvhkAfrA0-w&feature=youtu.be
Gradient Domain blending (1D)
Twosignals
Regularblending
Blendingderivatives
bright
dark
Gradient hole-filling (1D)
sourcetarget
It is impossible to faithfully preserve the gradients
sourcetarget
Gradient Domain Blending (2D)
Trickier in 2D:• Take partial derivatives dx and dy (the gradient field)• Fiddle around with them (copy, blend, smooth, feather, etc)• Reintegrate
– But now integral(dx) might not equal integral(dy)• Find the most agreeable solution
– Equivalent to solving Poisson equation– Can be done using least-squares (\ in Matlab)
Example
Source: Evan Wallace
Gradient Visualization
Source: Evan Wallace
+Specify object region
Poisson Blending AlgorithmA good blend should preserve gradients of source region without changing the background
Treat pixels as variables to be solved– Minimize squared difference between gradients of
foreground region and gradients of target region– Keep background pixels constant
Perez et al. 2003
Perez et al., 2003
sourcetarget mask
no blending gradient domain blending
Slide by Mr. Hays
What’s the difference?
no blendinggradient domain blending
- =
Slide by Mr. Hays
Perez et al, 2003
Limitations:• Can’t do contrast reversal (gray on black -> gray on white)• Colored backgrounds “bleed through”• Images need to be very well aligned
editing
Gradient Domain as Image Representation
See GradientShop paper as good example:
http://www.gradientshop.com/
Can be used to exert high-level control over images
Can be used to exert high-level control over images gradients – low level image-features
Can be used to exert high-level control over images gradients – low level image-features
+100pixelgradient
Can be used to exert high-level control over images gradients – low level image-features gradients – give rise to high level image-features
+100pixelgradient
Can be used to exert high-level control over images gradients – low level image-features gradients – give rise to high level image-features
+100
+100
+100
+100
+100
+100pixelgradient
+100
+100
+100
+100
+100
+100pixelgradient
Can be used to exert high-level control over images gradients – low level image-features gradients – give rise to high level image-features
+100
+100
+100
+100
+100
+100pixelgradient
+100
+100
+100
+100
+100
+100pixelgradient
image edge image edge
Can be used to exert high-level control over images gradients – low level image-features gradients – give rise to high level image-features manipulate local gradients to
manipulate global image interpretation
+100
+100
+100
+100
+100
+100pixelgradient
+100
+100
+100
+100
+100
+100pixelgradient
Can be used to exert high-level control over images gradients – low level image-features gradients – give rise to high level image-features manipulate local gradients to
manipulate global image interpretation
+255
+255
+255
+255
+255
pixelgradient
Can be used to exert high-level control over images gradients – low level image-features gradients – give rise to high level image-features manipulate local gradients to
manipulate global image interpretation
+255
+255
+255
+255
+255
pixelgradient
Can be used to exert high-level control over images gradients – low level image-features gradients – give rise to high level image-features manipulate local gradients to
manipulate global image interpretation
+0
+0
+0
+0
+0
pixelgradient
Can be used to exert high-level control over images gradients – low level image-features gradients – give rise to high level image-features manipulate local gradients to
manipulate global image interpretation
+0
+0
+0
+0
+0
pixelgradient
Can be used to exert high-level control over images
Optimization framework
Pravin Bhat et al
Optimization framework Input unfiltered image – u
Optimization framework Input unfiltered image – u Output filtered image – f
Optimization framework Input unfiltered image – u Output filtered image – f Specify desired pixel-differences – (gx, gy)
min (fx – gx)2 + (fy – gy)2
f
Energy function
Optimization framework Input unfiltered image – u Output filtered image – f Specify desired pixel-differences – (gx, gy) Specify desired pixel-values – d
min (fx – gx)2 + (fy – gy)2 + (f – d)2
f
Energy function
Optimization framework Input unfiltered image – u Output filtered image – f Specify desired pixel-differences – (gx, gy) Specify desired pixel-values – d Specify constraints weights – (wx, wy, wd)
min wx(fx – gx)2 + wy(fy – gy)2 + wd(f – d)2
f
Energy function
Pseudo image relighting change scene illumination
in post-production example
input
Pseudo image relighting change scene illumination
in post-production example
manual relight
Pseudo image relighting change scene illumination
in post-production example
input
Pseudo image relighting change scene illumination
in post-production example
GradientShop relight
Pseudo image relighting change scene illumination
in post-production example
GradientShop relight
Pseudo image relighting change scene illumination
in post-production example
GradientShop relight
Pseudo image relighting change scene illumination
in post-production example
GradientShop relight
Pseudo image relighting
u
f
o
Pseudo image relighting
min wx(fx – gx)2 + f wy(fy – gy)2 +
wd(f – d)2
Energy function
u
f
o
Pseudo image relighting
min wx(fx – gx)2 + f wy(fy – gy)2 +
wd(f – d)2
Energy function
Definition: d = u
u
f
o
Pseudo image relighting
min wx(fx – gx)2 + f wy(fy – gy)2 +
wd(f – d)2
Energy function
u
f
o Definition: d = u gx(p) = ux(p) * (1 + a(p)) a(p) = max(0, - u(p).o(p))
Pseudo image relighting
min wx(fx – gx)2 + f wy(fy – gy)2 +
wd(f – d)2
Energy function
u
f
o Definition: d = u gx(p) = ux(p) * (1 + a(p)) a(p) = max(0, - u(p).o(p))
Sparse data interpolation Interpolate scattered data
over images/video
Sparse data interpolation Interpolate scattered data
over images/video Example app: Colorization*
input output*Levin et al. – SIGRAPH 2004
u
f
user data
Sparse data interpolation
u
f
user data
Sparse data interpolation
min wx(fx – gx)2 + f wy(fy – gy)2 +
wd(f – d)2
Energy function
u
f
user data
Sparse data interpolation
min wx(fx – gx)2 + f wy(fy – gy)2 +
wd(f – d)2
Energy function
Definition: d = user_data
u
f
user data
Sparse data interpolation
min wx(fx – gx)2 + f wy(fy – gy)2 +
wd(f – d)2
Energy function
Definition: d = user_data if user_data(p) defined
wd(p) = 1else
wd(p) = 0
u
f
user data
Sparse data interpolation
min wx(fx – gx)2 + f wy(fy – gy)2 +
wd(f – d)2
Energy function
Definition: d = user_data if user_data(p) defined
wd(p) = 1else
wd(p) = 0 gx(p) = 0; gy(p) = 0
u
f
user data
Sparse data interpolation
min wx(fx – gx)2 + f wy(fy – gy)2 +
wd(f – d)2
Energy function
Definition: d = user_data if user_data(p) defined
wd(p) = 1else
wd(p) = 0 gx(p) = 0; gy(p) = 0 wx(p) = 1/(1 + c*|ux(p)|)
wy(p) = 1/(1 + c*|uy(p)|)
