Lecture’8:’’ CameraModels’vision.stanford.edu/teaching/cs131_fall1617/lectures/... ·...
59
Lecture 8 - Fei-Fei Li Lecture 8: Camera Models Dr. Juan Carlos Niebles Stanford AI Lab Professor FeiFei Li Stanford Vision Lab 20Oct16 1
Transcript of Lecture’8:’’ CameraModels’vision.stanford.edu/teaching/cs131_fall1617/lectures/... ·...
Lecture 8 - Fei-Fei Li
Lecture 8: Camera Models
Dr. Juan Carlos Niebles Stanford AI Lab
Professor Fei-‐Fei Li Stanford Vision Lab
20-‐Oct-‐16 1
Lecture 8 - Fei-Fei Li
What we will learn today?
• Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras
• ProjecQon matrix • Intrinsic parameters • Extrinsic parameters
20-‐Oct-‐16 2
Reading: [FP] Chapters 1 – 3 [HZ] Chapter 6
Lecture 8 - Fei-Fei Li
What we will learn today?
• Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras
• ProjecQon matrix • Intrinsic parameters • Extrinsic parameters
20-‐Oct-‐16 3
Reading: [FP] Chapters 1 – 3 [HZ] Chapter 6
Lecture 8 - Fei-Fei Li
Camera and World Geometry
How tall is this woman?
Which ball is closer?
How high is the camera?
What is the camera rotation?
What is the focal length of the camera?
Slide cred
it: J. Hayes
20-‐Oct-‐16 4
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 5
How do we see the world?
• Let’s design a camera – Idea 1: put a piece of film in front of an object – Do we get a reasonable image?
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 6
Pinhole camera
• Add a barrier to block off most of the rays – This reduces blurring – The opening known as the aperture
Lecture 8 - Fei-Fei Li
Camera obscura: the pre-‐camera
• Known during classical period in China and Greece (e.g. Mo-‐Tsi, China, 470BC to 390BC)
Illustration of Camera Obscura Freestanding camera obscura at UNC Chapel Hill
Photo by Seth Ilys
Slide credit: J. Hayes
20-‐Oct-‐16 7
Lecture 8 - Fei-Fei Li
Camera Obscura used for Tracing
Lens Based Camera Obscura, 1568 Slide credit: J. Hayes
20-‐Oct-‐16 8
Lecture 8 - Fei-Fei Li
First Photograph
Oldest surviving photograph – Took 8 hours on pewter plate
Joseph Niepce, 1826
Photograph of the first photograph
Stored at UT Austin
Niepce later teamed up with Daguerre, who eventually created Daguerrotypes
Slide credit: J. Hayes
20-‐Oct-‐16 9
Lecture 8 - Fei-Fei Li
Point of observation
Figures © Stephen E. Palmer, 2002
Dimensionality ReducQon Machine (3D to 2D)
3D world 2D image
20-‐Oct-‐16 10
Lecture 8 - Fei-Fei Li
ProjecQon can be tricky…
Slide source: Seitz
20-‐Oct-‐16 11
Lecture 8 - Fei-Fei Li
Slide source: Seitz
ProjecQon can be tricky…
20-‐Oct-‐16 12
Lecture 8 - Fei-Fei Li
ProjecQve Geometry What is lost? • Length
Which is closer?
Who is taller?
Slide credit: J. Hayes
20-‐Oct-‐16 13
Lecture 8 - Fei-Fei Li
Length is not preserved
Figure by David Forsyth
B’
C’
A’
20-‐Oct-‐16 14
Lecture 8 - Fei-Fei Li
ProjecQve Geometry What is lost? • Length • Angles
Perpendicular?
Parallel?
Slide credit: J. Hayes
20-‐Oct-‐16 15
Lecture 8 - Fei-Fei Li
What is preserved? • Straight lines are sQll straight
ProjecQve Geometry
Slide credit: J. Hayes
20-‐Oct-‐16 16
Lecture 8 - Fei-Fei Li
Vanishing points and lines Parallel lines in the world intersect in the image at a “vanishing point”
Slide credit: J. Hayes
o Vanishing Point o Vanishing Point
Vanishing Line
20-‐Oct-‐16 17
Lecture 8 - Fei-Fei Li
Vanishing point
Vanishing line
Vanishing point
Vertical vanishing point
(at infinity)
Slide from Efros, Photo from Criminisi
Vanishing points and lines
20-‐Oct-‐16 18
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 19
Pinhole camera
⎥⎦
⎤⎢⎣
⎡ʹ′
ʹ′=ʹ′→yx
P⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
zyx
P
⎪⎪⎩
⎪⎪⎨
⎧
=
=
zyfy
zxfx
''
''
Derived using similar triangles
Note: z is always negaQve.
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 20
Pinhole camera
• Common to draw image plane in front of the focal point • Moving the image plane merely scales the image.
f f
⎪⎪⎩
⎪⎪⎨
⎧
=
=
zyf'y
zxf'x
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 21
Pinhole camera
Kate lazuka ©
Is the size of the aperture important?
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 22
Cameras & Lenses
Shrinking aperture size
-‐ Rays are mixed up
Adding lenses! -‐ Why the aperture cannot be too small?
-‐ Less light passes through -‐ DiffracQon effect
Lecture 8 - Fei-Fei Li
What we will learn today?
• Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras
• ProjecQon matrix • Intrinsic parameters • Extrinsic parameters
20-‐Oct-‐16 23
Reading: [FP] Chapters 1 – 3 [HZ] Chapter 6
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 24
Cameras & Lenses
• A lens focuses light onto the film
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 25
Cameras & Lenses
• A lens focuses light onto the film – Rays passing through the center are not deviated – All parallel rays converge to one point on a plane located at the focal
length f
focal point
f
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 26
Cameras & Lenses
• A lens focuses light onto the film – There is a specific distance at which objects are “in focus”
[other points project to a “circle of confusion” in the image]
“circle of confusion”
Lecture 8 - Fei-Fei Li
Cameras & Lenses • Laws of geometric opQcs
– Light travels in straight lines in homogeneous medium – ReflecQon upon a surface: incoming ray, surface normal, and reflecQon are co-‐planar
– RefracQon: when a ray passes from one medium to another
20-‐Oct-‐16 27
Snell’s law
n1 sin α1 = n2 sin α2
α1 = incident angle α2 = refracQon angle ni = index of refracQon
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 28
Thin Lenses
Snell’s law:
n1 sin α1 = n2 sin α2
Small angles: n1 α1 ≈ n2 α2
n2 = n (lens) n1 = 1 (air)
zo
zy'z'y
zx'z'x
⎪⎪⎩
⎪⎪⎨
⎧
=
=
ozf'z +=
)1n(2Rf −
=
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 29
Cameras & Lenses
Source wikipedia
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 30
Issues with lenses: Chroma>c Aberra>on • Lens has different refracQve indices for different wavelengths: causes color fringing
)1n(2Rf −
=
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 31
Issues with lenses: Spherical aberra>on
• Rays farther from the opQcal axis focus closer
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 32
Issues with lenses: Radial Distor>on
No distortion
Pin cushion
Barrel (fisheye lens)
– DeviaQons are most noQceable for rays that pass through the edge of the lens
Image magnificaQon decreases with distance from the opQcal axis
Lecture 8 - Fei-Fei Li
What we will learn today?
• Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras
• ProjecQon matrix • Intrinsic parameters • Extrinsic parameters
20-‐Oct-‐16 33
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 34
Rela>ng real-‐world point to a point on a camera f
f = focal length c = center of the camera
c
P = (x, y, z)→ P ' = ( f xz, f y
z)
2E
3 ℜ→ℜ
P
P’
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 35
No — division by z is nonlinear!
Is this a linear transformaQon?
How to make it linear?
P = (x, y, z)→ P ' = ( f xz, f y
z)
Rela>ng real-‐world point to a point on a camera
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 36
Homogeneous coordinates – a reminder
homogeneous image coordinates
homogeneous scene coordinates
• ConverQng from homogeneous coordinates
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 37
RelaQng a real-‐world point to a point on the camera
P = (x, y, z)→ P ' = ( f xz, f y
z)
In Cartesian coordinates:
In homogeneous coordinates:
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
10100000000
'zyx
ff
zyfxf
PPMP ='
M
“Projec>on matrix”
3H
4 ℜ→ℜ
Lecture 8 - Fei-Fei Li
Interlude: why does this maser?
20-‐Oct-‐16 38
Lecture 8 - Fei-Fei Li
Object RecogniQon (CVPR 2006)
Slide credit: J. Hayes
20-‐Oct-‐16 39
Lecture 8 - Fei-Fei Li
InserQng photographed objects into images (SIGGRAPH 2007)
Original Created
Slide credit: J. Hayes
20-‐Oct-‐16 40
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 41
RelaQng a real-‐world point to a point on the camera
In homogeneous coordinates:
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
10100000000
'zyx
ff
zyfxf
P
M ideal world
Intrinsic Assumptions • Unit aspect ratio • Optical center at (0,0) • No skew
Extrinsic Assumptions • No rotation • Camera at (0,0,0)
Lecture 8 - Fei-Fei Li 20-‐Oct-‐16 42
RelaQng a real-‐world point to a point on the camera
In homogeneous coordinates:
P ' =f xf yz
!
"
####
$
%
&&&&
=
f 0 0 00 f 0 00 0 1 0
!
"
###
$
%
&&&
xyz1
!
"
####
$
%
&&&&
= K I 0[ ]P
Intrinsic Assumptions • Unit aspect ratio • Optical center at (0,0) • No skew
Extrinsic Assumptions • No rotation • Camera at (0,0,0)
K
Lecture 8 - Fei-Fei Li
Remove assumpQon: known opQcal center
P ' = K I 0!"
#$P
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
101000000
10
0
zyx
vfuf
vu
w
Intrinsic Assumptions • Optical center at (0,0) • Optical center at (u0, v0) • Square pixels • No skew
Extrinsic Assumptions • No rotation • Camera at (0,0,0)
Slide inspiraQon: S. Savarese
20-‐Oct-‐16 43
Lecture 8 - Fei-Fei Li
Remove assumpQon: square pixels
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
101000000
10
0
zyx
vu
vu
w β
α
Extrinsic Assumptions • No rotation • Camera at (0,0,0)
Intrinsic Assumptions • Optical center at (u0, v0) • Square pixels • Rectangular pixels • No skew
P ' = K I 0!"
#$P
Slide inspiraQon: S. Savarese
20-‐Oct-‐16 44
Lecture 8 - Fei-Fei Li
Remove assumpQon: non-‐skewed pixels
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
10100000
10
0
zyx
vus
vu
w β
α
Extrinsic Assumptions • No rotation • Camera at (0,0,0)
Intrinsic Assumptions • Optical center at (u0, v0) • Rectangular pixels • No skew • Small skew
P ' = K I 0!"
#$P
x
y
xc
yc
C=[cx, cy] vSlide inspiraQon: S. Savarese
20-‐Oct-‐16 45
Lecture 8 - Fei-Fei Li
Remove assumpQon: non-‐skewed pixels
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
10100000
10
0
zyx
vus
vu
w β
α
Extrinsic Assumptions • No rotation • Camera at (0,0,0)
Intrinsic Assumptions • Optical center at (u0, v0) • Rectangular pixels • Small skew
Intrinsic parameters
P ' = K I 0!"
#$P
Slide inspiraQon: S. Savarese
20-‐Oct-‐16 46
Lecture 8 - Fei-Fei Li
Real world camera: Translate + Rotate
Ow
iw
kw
jw
t
R
Slide inspiraQon: S. Savarese, J. Hayes
20-‐Oct-‐16 47
Lecture 8 - Fei-Fei Li
Remove assumpQon: allow translaQon
Extrinsic Assumptions • No rotation • Camera at (0,0,0) -> (tx,ty,tz)
Intrinsic Assumptions • Optical center at (u0, v0) • Rectangular pixels • Small skew
P ' = K I t[ ] P⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
1100010001
1000
0
10
0
zyx
ttt
vu
vu
w
z
y
x
β
α
Slide inspiraQon: S. Savarese
20-‐Oct-‐16 48
Lecture 8 - Fei-Fei Li
Remove assumpQon: allow rotaQon
Extrinsic Assumptions • No rotation • Camera at (tx,ty,tz)
Intrinsic Assumptions • Optical center at (u0, v0) • Rectangular pixels • Small skew
P
P’ γ
y
z ⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡ −
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−=
1000cossin0sincos
)(
cos0sin010
sin0cos)(
cossin0sincos0001
)(
γγ
γγ
γ
ββ
ββ
β
αα
ααα
z
y
x
R
R
R
Rotation around the coordinate axes, counter-clockwise
Slide inspiraQon: S. Savarese
20-‐Oct-‐16 49
Lecture 8 - Fei-Fei Li
Remove assumpQon: allow rotaQon
Extrinsic Assumptions • No rotation • Camera at (tx,ty,tz)
Intrinsic Assumptions • Optical center at (u0, v0) • Rectangular pixels • Small skew
P ' = K R t[ ] P
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
1100
01 333231
232221
131211
0
0
zyx
trrrtrrrtrrr
vus
vu
w
z
y
x
β
α
Slide inspiraQon: S. Savarese
20-‐Oct-‐16 50
Lecture 8 - Fei-Fei Li
A generic projecQon matrix
Extrinsic Assumptions • Allow rotation • Camera at (tx,ty,tz)
Intrinsic Assumptions • Optical center at (u0, v0) • Rectangular pixels • Small skew
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
1100
01 333231
232221
131211
0
0
zyx
trrrtrrrtrrr
vus
vu
w
z
y
x
β
α
P ' = K R t[ ] P
Slide inspiraQon: S. Savarese
20-‐Oct-‐16 51
Lecture 8 - Fei-Fei Li
A generic projecQon matrix
Extrinsic Assumptions • Allow rotation • Camera at (tx,ty,tz)
Intrinsic Assumptions • Optical center at (u0, v0) • Rectangular pixels • Small skew
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
1100
01 333231
232221
131211
0
0
zyx
trrrtrrrtrrr
vus
vu
w
z
y
x
β
α
P ' = K R t[ ] P
Degrees of freedom?? Slide inspiraQon: S. Savarese
20-‐Oct-‐16 52
Lecture 8 - Fei-Fei Li
A generic projecQon matrix
Extrinsic Assumptions • Allow rotation • Camera at (tx,ty,tz)
Intrinsic Assumptions • Optical center at (u0, v0) • Rectangular pixels • Small skew
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
1100
01 333231
232221
131211
0
0
zyx
trrrtrrrtrrr
vus
vu
w
z
y
x
β
α
P ' = K R t[ ] P
Degrees of freedom??
5 6
Slide inspiraQon: S. Savarese
20-‐Oct-‐16 53
Lecture 8 - Fei-Fei Li
CS231a: Camera CalibraQon esQmate all intrinsic and extrinsic parameters
Extrinsic Assumptions • Allow rotation • Camera at (tx,ty,tz)
Intrinsic Assumptions • Optical center at (u0, v0) • Rectangular pixels • Small skew
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
1100
01 333231
232221
131211
0
0
zyx
trrrtrrrtrrr
vus
vu
w
z
y
x
β
α
P ' = K R t[ ] P 5 6
Slide inspiraQon: S. Savarese
20-‐Oct-‐16 54
Lecture 8 - Fei-Fei Li
Orthographic ProjecQon • Special case of perspecQve projecQon
– Distance from the COP to the image plane is infinite
– Also called “parallel projecQon” – What’s the projecQon matrix?
Image World
Slide credit: Steve Seitz
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
1100000100001
1zyx
vu
w
20-‐Oct-‐16 55
Lecture 8 - Fei-Fei Li
Scaled Orthographic ProjecQon • Special case of perspecQve projecQon
– Object dimensions are small compared to distance to camera
– Also called “weak perspecQve” – What’s the projecQon matrix?
Image World
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
1000000000
1zyx
sf
fvu
w
Slide credit: Steve Seitz
20-‐Oct-‐16 56
Lecture 8 - Fei-Fei Li
Field of View (Zoom)
20-‐Oct-‐16 57
Lecture 8 - Fei-Fei Li
Things to remember
• Vanishing points and vanishing lines
• Pinhole camera model and camera projecQon matrix M • Intrinsic parameters • Extrinsic parameters
• Homogeneous coordinates
Vanishing point
Vanishing line
Vanishing point
Vertical vanishing point
(at infinity)
P ' = K R t!"
#$P
Slide inspiraQon: J. Hayes
20-‐Oct-‐16 58
Lecture 8 - Fei-Fei Li
What we have learned today?
• Pinhole cameras • Cameras & lenses • The geometry of pinhole cameras
• ProjecQon matrix • Intrinsic parameters • Extrinsic parameters
20-‐Oct-‐16 59
Reading: [FP] Chapters 1 – 3 [HZ] Chapter 6
