Special Effects Techniques - NUS Computingcs4340/documents/tricks.pdf · 20 CS5245: Vision &...
Transcript of Special Effects Techniques - NUS Computingcs4340/documents/tricks.pdf · 20 CS5245: Vision &...
1
Special Effects TechniquesView Morphing,
Inpainting,Texture Synthesis,
Day for Night
2
CS5245: Vision & Graphics for Special Effects
Agenda
View MorphingOther Special Effects Techniques
InpaintingTexture SynthesisSimulating Night Scenes
3
CS5245: Vision & Graphics for Special Effects
(ref: Seitz and Dyer, “View Morphing”, SIGGRAPH 96)http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm
View morphing is not image morphing. Image morphing encounters distortion when:
• cameras are not parallel
• scene has depth (i.e. 3D variations)
On the other hand, view morphing does not cause distortion
View Morphing
4
CS5245: Vision & Graphics for Special Effects
Source: Seitz and Dyer, “View Morphing”, SIGGRAPH 96)
Image Morphing Example
5
CS5245: Vision & Graphics for Special Effects
Source: Seitz and Dyer, “View Morphing”, SIGGRAPH 96)
Example 2Top: image morphing resultBottom: view morphing result
6
CS5245: Vision & Graphics for Special Effects
Let
I0 be input image from camera 1I1 be input image from camera 2
Is be the new image to be synthesized, with viewpointlying along the straight line joining the optical centres of cameras 1 and 2
Π0 be the projection matrix of camera 1 Π1 be the projection matrix of camera 2Πs be the projection matrix of camera 3 (virtual camera)
Theory: Parallel Views
7
CS5245: Vision & Graphics for Special Effects
8
CS5245: Vision & Graphics for Special Effects
Π, a projection matrix, can always be written as
[ ]HCH −=Π
where C is the optical centre of the camera, and H is 3x3 matrix containing the position & orientation info
Express a 3D scene point [x y z] as homogeneous coordinates
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
1zyx
P
Express the corresponding image point [u v] as homogeneous coordinates
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
1vu
m
9
CS5245: Vision & Graphics for Special Effects
In other words, Pm Π=
For parallel views, we can w.l.o.g. assume that the 1st camera is at the origin with focal length f0, and 2nd camera is at (CX, CY, 0) with focal length f1. Hence:
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=Π
000
1000000
0
0
0 ff
and
Interpolating image coordinates gives the new image point
PZ
PZ
sPZ
smsms
sΠ=
Π+Π−=+−
1
11)1()1( 1021
where10)1( Π+Π−=Π sss
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡−−
=Π0100
0000
1
1
1
1
1 Y
X
CfCf
ff
10
CS5245: Vision & Graphics for Special Effects
Therefore, when camera image planes are parallel, view morphing means a linear interpolation of camera projection matrices. No distortion in this case.
In fact, represents a camera at position Cs with focal length fsgiven by:
sΠ
10)1()0,,(fsfsf
sCsCC
s
YXs
+−==
11
CS5245: Vision & Graphics for Special Effects
What happens if camera image planes are not parallel?
[ ]1111 CHH −=Π [ ]2222 CHH −=Π
In this case, additional steps are required.
12
CS5245: Vision & Graphics for Special Effects
Non-parallel Views
13
CS5245: Vision & Graphics for Special Effects
Step 1: Pre-warp
Note that H0 and H1 are obtained by solving
where F is the fundamental matrix between I0 and I1
and
Step 2: Linearly interpolate positions and intensities of corresponding pixels in and to form
Step 3: Post-warp
11
11 IHI −=01
00 IHI −=
0I 1I sI
sss IHI ˆ=
FHFH T ˆ)( 10
11 =−−
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡−=010100
000F
14
CS5245: Vision & Graphics for Special Effects
Some view morphing examples:
15
CS5245: Vision & Graphics for Special Effects
Dynamic View Morphing
Given two views of a dynamic scene, taken by different cameras in different locations at different times, create a sequence of views showing a smooth transition between the two camera views that also shows the dynamic scene changing smoothly.
http://www.cs.wisc.edu/~rmanning/homepage/research/view.interpolation.html
16
CS5245: Vision & Graphics for Special Effects
Often you need to:
Special effects:Remove objectsFill holesReplicate patternTransfer textureShoot night scenes
TechniquesInpaintingTexture synthesisDay for Night
17
CS5245: Vision & Graphics for Special Effects
InpaintingObject Removal by Exemplar-Based InpaintingCriminisi, Perez & Toyamahttp://www.research.microsoft.com/%7Eantcrim/papers/Criminisi_cvpr2003.pdf
18
CS5245: Vision & Graphics for Special Effects
Idea
Fill target region starting from its boundary and working inwards.Fill order is important: try to preserve strong edges (isophotes)
19
CS5245: Vision & Graphics for Special Effects
3 steps
Which patch to fill first?Fill patch with what?Update confidence values
20
CS5245: Vision & Graphics for Special Effects
Which patch to fill first?
Given a patch Ψp centred at the point p for some p ∈ δΩits priority P(p) is defined as the product of two terms:
P(p) = C(p)D(p).
We call C(p) the confidence term and D(p) the data term,where |Ψp| is the area of Ψp, α is a normalization factor(e.g., α = 255 for a typical grey-level image), and np is aunit vector orthogonal to the front δΩ in the point p.
21
CS5245: Vision & Graphics for Special Effects
Which patch to fill first?
C(p) measures amount of reliable info surrounding p.
Intention is to fill first those patches which have more of their pixels already filled
D(p) measures strength of isophotes hitting boundary δΩ
Encourages propagation of linear structures (e.g. edges)
22
CS5245: Vision & Graphics for Special Effects
Fill with what?
After computing priorities, patch with highest priority is chosen.Then select patch in source region Φ that is most similar to chosen patch:
d( , ) is sum-squared-difference of pixels
Now fill empty region in with corresponding pixels in source patch
23
CS5245: Vision & Graphics for Special Effects
Update confidence
After filling patch , update confidence in the area by:
Repeat 3 steps until target region Ω shrinks to nothing
24
CS5245: Vision & Graphics for Special Effects
Results
(a) Original image(b) Target region manually selected(c) Result of filling by concentric layers(d) Result of their algorithm
25
CS5245: Vision & Graphics for Special Effects
Results
26
CS5245: Vision & Graphics for Special Effects
More results
27
CS5245: Vision & Graphics for Special Effects
28
CS5245: Vision & Graphics for Special Effects
Other inpainting papers
www.inf.ufrgs.br/~oliveira/pubs_files/inpainting.pdf
www.iua.upf.es/~mbertalmio/restoration.html
29
CS5245: Vision & Graphics for Special Effects
Texture Synthesis
How to get from this
To this?
30
CS5245: Vision & Graphics for Special Effects
Texture Synthesis
Image Quilting for Texture Synthesis and Transfer A. Efros and W. Freemanhttp://graphics.cs.cmu.edu/people/efros/research/quilting.html
Basic Idea:Cut up input image into smaller blocksCleverly choose blocks and join themTake care of the boundaryhttp://graphics.cs.cmu.edu/people/efros/research/quilting/efros-siggraph01.ppt
31
CS5245: Vision & Graphics for Special Effects
The Goal of Texture Synthesis
Given a finite sample of some texture, the goal is to synthesize other samples from that same texture
The sample needs to be "large enough“
True (infinite) texture
SYNTHESIS
generated image
input image
32
CS5245: Vision & Graphics for Special Effects
The Challenge
Need to model the whole spectrum: from repeated to stochastic texture
repeated
stochastic
Both?
33
CS5245: Vision & Graphics for Special Effects
Input texture
B1 B2
Random placement of blocks
block
B1 B2
Neighboring blocksconstrained by overlap
B1 B2
Minimal errorboundary cut
34
CS5245: Vision & Graphics for Special Effects min. error boundary
Minimal error boundaryoverlapping blocks vertical boundary
__ ==22
overlap error
Error surface e
35
CS5245: Vision & Graphics for Special Effects
Min. Error Cut Algorithm
For row i = 2 … NFor column j = 1 … M
(Efficiently solved using Dynamic programming)
In last row, search for minTrace this back up to 1st row for best path
i
j
36
CS5245: Vision & Graphics for Special Effects
Our Philosophy
The “Corrupt Professor’s Algorithm”:Plagiarize as much of the source image as you canThen try to cover up the evidence
Rationale: Texture blocks are by definition correct samples of texture so problem only connecting them together
37
CS5245: Vision & Graphics for Special Effects
AlgorithmPick size of block and size of overlapSynthesize blocks in raster order
Search input texture for block that satisfies overlap constraints (above and left)
Easy to optimize using NN search [Liang et.al., ’01]Paste new block into resulting texture
use dynamic programming to compute minimal error boundary cut
38
CS5245: Vision & Graphics for Special Effects
39
CS5245: Vision & Graphics for Special Effects
40
CS5245: Vision & Graphics for Special Effects
41
CS5245: Vision & Graphics for Special Effects
Portilla & Simoncelli
Wei & Levoy Image Quilting
input image
Xu, Guo & Shum
42
CS5245: Vision & Graphics for Special Effects
Failures(ChernobylHarvest)
43
CS5245: Vision & Graphics for Special Effects
Texture TransferTake the texture from one object and “paint” it onto another object
This requires separating texture and shapeThat’s HARD, but we can cheat Assume we can capture shape by boundary and rough shading
Then, just add another constraint when sampling: Then, just add another constraint when sampling: similarity to underlying image at that spotsimilarity to underlying image at that spot
44
CS5245: Vision & Graphics for Special Effects
Modified error
Weighted sum of overlap error and correspondence map error (at current location (x,y))α controls the contribution of each termCorrespondence map could be intensity (pixel value) or anything else.
2),(),(
221 ][)1(][ target
yxsource
yxovov CCBBe −−+−= αα
45
CS5245: Vision & Graphics for Special Effects
++ ==
++ ==
parmesan
rice
46
+ =
47
+ =
48
CS5245: Vision & Graphics for Special Effects
Sourcetexture
Target image
Sourcecorrespondence
image
Targetcorrespondence image
Blurred intensities
49
CS5245: Vision & Graphics for Special Effects
++ ==
50
CS5245: Vision & Graphics for Special Effects
Other papershttp://graphics.cs.cmu.edu/people/efros/research/EfrosLeung.htmlhttp://www.debonet.com/Research/TextureSynthesis/http://research.microsoft.com/research/pubs/view.aspx?msr_tr_id=MSR-TR-2000-32http://www.comp.nus.edu.sg/~cs4243/demo/textsyn/textsyn.html
Benadryl Landscapers Commericalhttp://www.theorphanage.com/ocp/portfolio/recent/1005
51
OBJECTIVE: Transform a ‘day image’ into a ‘night’ version of it including the loss of acuity at low luminances.
+ desired luminance level =
Day for nightVisual Acuity in Day for Night, by G. Haro, M. Bertalmio, V. Caselles2nd IEEE Workshop on Variational, Geometric and Level Set Methods in Computer Vision http://www.tecn.upf.es/~gharo/Download/upc.ppt
52
CS5245: Vision & Graphics for Special Effects
Night vs. Day
Traditionally done using blue filters.Not realistic
Night scene is not simply a darkened version of day scene
Colors are different (rods vs. cones)Contrast lowerLoss of details (visual acuity)
53
CS5245: Vision & Graphics for Special Effects
TRANSFORMATION IN 5 STEPS
1. Estimation of reflectance values and modification of illuminant.
2. Modification of chromaticity.
3. Modification of luminance.
4. Modification of contrast.
5. Loss of acuity: Diffusion.
Day for night algorithm
54
CS5245: Vision & Graphics for Special Effects
Emulating human vision at night.
Emulating a photographic film (γn=3, γ=1).
Simulated scene at 0.3 log cd/m2
Results
55
CS5245: Vision & Graphics for Special Effects
Without changing the variance, a=1
Increasing the variance, a=0.1
Simulated scene at 0.1 log cd/m2
Results
56
CS5245: Vision & Graphics for Special Effects
Results
Day scene
Simulated night scene