Computer Vision and Robotics Group Institut d’Informàtica i Aplicacions Jordi Pagès Joaquim...
-
Upload
penelope-walters -
Category
Documents
-
view
216 -
download
0
Transcript of Computer Vision and Robotics Group Institut d’Informàtica i Aplicacions Jordi Pagès Joaquim...
Computer Vision and Robotics Group
Institut d’Informàtica i Aplicacions
Jordi PagèsJoaquim Salvi
Overview of coded light projection techniques for automatic 3D
profiling
2
Formació
Presentation outline
1/36
• Introduction• Coded pattern classification
– Time-multiplexing– Spatial codification– Direct codification
• Experimental results• Sub-pixel matching• Conclusions & guidelines
Conclusions
Sub-pixel matching
Experiments
Classification
Introduction
3
Conclusions
Sub-pixel matching
Experiments
Classification
Introduction
Shape acquisition
Stereovision
Encoding/
decoding
Introduction: shape acquisition
2/36
Shape acquisition techniques
Contact
Non-destructive Destructive
CMM Jointed armsSlicing
Non-contact
Reflective Transmissive
Non-optical
Optical
Industrial CT
Microwave radar
Sonar
PassiveActive
Stereo Shading Silhouettes TextureMotion
Shape from X
Imaging radar
Triangulation
Interferometry
(Coded) Structured
light
Moire Holography
Stereo
Source: Brian Curless
4
Conclusions
Sub-pixel matching
Experiments
Classification
Introduction
Shape acquisition
Stereovision
Encoding/
decoding
Introduction: passive stereovision
3/36
• Correspondence problem
• geometric constraints
search along epipolar lines
• 3D reconstruction of matched pairs by triangulation
5
Conclusions
Sub-pixel matching
Experiments
Classification
Introduction
Shape acquisition
Stereovision
Encoding/
decoding
Introduction: active stereo (coded structured light)
4/36
• One of the cameras is replaced by a light emitter
• Correspondence problem is solved by searching the pattern in the camera image (pattern decoding)
6
Conclusions
Sub-pixel matching
Experiments
Classification
Introduction
Shape acquisition
Stereovision
Encoding/
decoding
Introduction: pattern encoding/decoding (I)
5/36
• A pattern is encoded when after projecting it onto a surface, a set of regions of the observed projection can be easily matched with the original pattern. Example: pattern with two-encoded-columns
• The process of matching an image region with its corresponding pattern region is known as pattern decoding similar to searching correspondences
• Decoding a projected pattern allows a large set of correspondences to be easily found thanks to the a priori knowledge of the light pattern
Pixels in red and yellow are directly matched with the pattern columns
Codification using colors
Object scene
7
Conclusions
Sub-pixel matching
Experiments
Classification
Introduction
Shape acquisition
Stereovision
Encoding/
decoding
6/36
Introduction: pattern encoding/decoding (II)
• Two ways of encoding the correspondences: single and double axis codification it determines how the triangulation is calculated
• Decoding the pattern means locating points in the camera image whose corresponding point in the projector pattern is a priori known
Double-axis encoding
Triangulation by line-to-plane intersection
Triangulation by line-to-line intersection
Single-axis encoding
8
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Coded structured light patterns: a classification proposal
7/36
Binary codes Posdamer et al., Inokuchi et al.,
Minou et al., Trobina, Valkenburg and McIvor, Skocaj and Leonardis,
Rocchin et al., …
n-ary codes Caspi et al., Horn and Kiryati, Osawa et al.,…
Gray code + Phase shifting
Bergmann, Sansoni et al., Wiora, Gühring, …
TIME-MULTIPLEXING
Hybrid methods K. Sato, Hall-Holt and Rusinkiewicz, Wang et al., …
Non-formal codification
Maruyama and Abe, Durdle et al., Ito and Ishii, Boyer and Kak, Chen
et al., …
De Bruijn sequences Hügli and Maître, Monks et al.,
Vuylsteke and Oosterlinck, Salvi et al. Lavoi et al., Zhang et al., …
SPATIAL CODIFICATION
M-arrays Morita et al., Petriu et al., Kiyasu et al., Spoelder et al., Griffin and Yee, Davies and Nixon, Morano et
al., …
Grey levels Carrihill and Hummel, Chazan and Kiryati, Hung, ... DIRECT
CODIFICATION Colour Tajima and Iwakawa, Smutny and Pajdla, Geng, Wust and Capson, T.
Sato, …
9
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Time-multiplexing
8/36
• The time-multiplexing paradigm consists of projecting a series of light patterns so that every encoded point is identified with the sequence of intensities that receives
• The most common structure of the patterns is a sequence of stripes increasing its length by the time single-axis encoding
• Advantages:
– high resolution a lot of 3D points
– High accuracy (order of m)
– Robustness against colorful objects since binary patterns can be used
• Drawbacks:
– Static objects only
– Large number of patterns
Pattern 1
Pattern 2
Pattern 3
Projected over time
Example: 3 binary-encoded patterns which allows the
measuring surface to be divided in 8 sub-
regions
10
Formació
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Time-multiplexing: binary codes (I)
9/36
Posdamer et al.
Inokuchi et al.
Minou et al.
Trobina
Valkenburg and McIvor
Skocaj and Leonardis
Binary codes
Rocchini et al.
Static
Scene applicability Moving
Binary
Grey levels
Pixel depth
Colour
Periodical Coding strategy
Absolute
• Coding redundancy: every edge between adjacent stripes can be decoded by the sequence at its left or at its right
11
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Time-multiplexing: Binary codes (II)
10/36
Pattern 1
Pattern 2
Pattern 3
Projected over time
• Every encoded point is identified by the sequence of intensities that receives
• n patterns must be projected in order to encode 2n stripes
Example: 7 binary patterns proposed
by Posdamer & Altschuler
…
Codeword of this píxel: 1010010 identifies the corresponding pattern stripe
12
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Time-multiplexing: n-ary codes (I)
11/36
3 patterns based on a n-ary code of 4 grey levels (Horn & Kiryati) 64 encoded stripes
• n-ary codes reduce the number of patterns by increasing the number of projected intensities (grey levels/colours) increases the basis of the code
• The number of patterns, the number of grey levels or colours and the number of encoded stripes are strongly related fixing two of these parameters the reamaining one is obtained
Using a binary code, 6 patterns are
necessary necessary to encode
64 stripes
13
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Time-multiplexing: n-ary codes (II)
12/36
Caspi et al. √ √ √
Horn and Kiryati √ √ √ N-ary codes
Osawa et al. √ √ √
Static
Scene applicability Moving
Binary
Grey levels
Pixel depth
Colour
Periodical Coding strategy
Absolute
• n-ary codes reduce the number of patterns by increasing the number of projected intensities (grey levels/colours)
14
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Time-multiplexing: Gray code + Phase shifting (I)
13/36
Gühring’s line-shift technique
• A sequence of binary patterns (Gray encoded) are projected in order to divide the object in regions
• An additional periodical pattern is projected
• The periodical pattern is projected several times by shifting it in one direction in order to increase the resolution of the system similar to a laser scanner
Example: three binary patterns
divide the object in 8 regions
Without the binary patterns we would not be
able to distinguish
among all the projected slits
Every slit always falls in the same
region
15
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
14/36
Bergmann
Sansoni et al.
Wiora
Gray Code + Phase Shift
Gühring
Static
Scene applicability Moving
Binary
Grey levels
Pixel depth
Colour
Periodical Coding strategy
Absolute
• A periodical pattern is shifted and projected several times in order to increase the resolution of the measurements
• The Gray encoded patterns permit to differentiate among all the periods of the shifted pattern
Time-multiplexing: Gray code + Phase shifting (II)
16
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Formació
Time-multiplexing: hybrid methods (I)
15/36
• In order to decode an illuminated point it is necessary to observe not only the sequence of intensities received by such a point but also the intensities of few (normally 2) adjacent points
• The number of projected patterns reduces thanks to the spatial information that is taken into account
• The redundancy on the binary codification
is eliminated
Hall-Holt and Rusinkiewicz technique:
4 patterns with 111 binary stripes
Edges encoding: 4x2 bits (every
adjacent stripe is a bit)
1
1
1
1
1
0
0
0 Pattern 1
Pattern 2
Pattern 3
Pattern 4
Edge codeword: 10110101
17
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
16/36
Time-multiplexing: hybrid methods (II)
K. Sato
Hall-Holt and Rusinkiewicz Hybrid methods
Wang et al.
Static
Scene applicability Moving
Binary
Grey levels
Pixel depth
Colour
Periodical Coding strategy
Absolute
18
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Spatial Codification
17/36
• Spatial codification paradigm encodes a set of points with the information contained in a neighborhood (called window) around them
• The codification is condensed in a unique pattern instead of multiplexing it along time
• The size of the neighborhood (window size) is proportional to the number of encoded points and inversely proportional to the number of used colors
• The aim of these techniques is to obtain a one-shot measurement system moving objects can be measured
• Advantages:
― Moving objects supported
― Possibility to condense the codification to a unique pattern
• Drawbacks:
― Discontinuities on the object surface can produce erroneous window decoding (occlusions problem)
― The higher the number of used colours, the more difficult to correctly identify them when measuring non-neutral surfaces
― Maximum resolution cannot be reached
19
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Spatial codification: non-formal codification (I)
18/36
• The first group of techniques that appeared used codification schemes with no mathematical basis
• Drawbacks:
- the codification is not optimal and often produces ambiguities since different regions of the pattern are identical
- the structure of the pattern is too complex for a good image processing
Maruyama and Abe complex structure
based on slits containing randomly
placed cuts
Durdle et al. periodic pattern
20
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Spatial codification: non-formal codification (II)
19/36
Maruyama and Abe
Durdle et al.
Ito and Ishii
Boyer and Kak
Non-formal codification
Chen et al.
Static
Scene applicability Moving
Binary
Grey levels
Pixel depth
Colour
Periodical Coding strategy
Absolute
21
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
20/36
Spatial codification: De Bruijn sequences (I)
• A De Bruijn sequence (or pseudorrandom sequence) of order m over an alphabet of n symbols is a circular string of length nm that contains every substring of length m exactly once (in this case the windows are unidimensional).
1000010111101001
• The De Bruijn sequences are used to define coloured slit patterns (single axis codification) or grid patterns (double axis codification)
• In order to decode a certain slit it is only necessary to identify one of the windows in which it belongs to
Zhang et al.: 125 slits encoded with a De Bruijn sequence of 8 colors and
window size of 3 slits
Salvi et al.: grid of 2929 where a De Bruijn sequence of 3 colors and window
size of 3 slits is used to encode the vertical and horizontal slits
m=4 (window size)
n=2 (alphabet symbols)
22
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Spatial codification: De Bruijn sequences (II)
21/36
Hügli and Maître
Monks et al.
Vuylsteke and Oosterlinck
Salvi et al.
Lavoie et al.
De Bruijn sequences
Zhang et al.
Static
Scene applicability Moving
Binary
Grey levels
Pixel depth
Colour
Periodical Coding strategy
Absolute
23
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Spatial codification: M-arrays (I)
22/36
• An m-array is the bidimensional extension of a De Bruijn sequence. Every window of wh units appears only once. The window size is related with the size of the m-array and the number of symbols used
0 0 1 0 1 00 1 0 1 1 01 1 0 0 1 10 0 1 0 1 0
Morano et al. M-arry represented with an array of coloured dots
Example: binary m-array of size 46 and window size of 22
M-array proposed by Vuylsteke et al. Represented with shape primitives
Shape primitives used to represent
every symbol of the alphabet
24
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
23/36
Spatial codification: M-arrays (II)Morita et al.
Petriu et al.
Kiyasu et al.
Spoelder et al.
Griffin and Yee
Davies and Nixon
M-arrays
Morano et al.
Static
Scene applicability Moving
Binary
Grey levels
Pixel depth
Colour
Periodical Coding strategy
Absolute
25
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Direct Codification
24/36
• Every encoded pixel is identified by its own intensity/colour
• Since the codification is usually condensed in a unique pattern, the spectrum of intensities/colours used is very large
• Additional reference patterns must be projected in order to differentiate among all the projected intensities/colours:
• Ambient lighting (black pattern)• Full illuminated (white pattern)• …
• Advantages:– Reduced number of patterns– High resolution can be teorically achieved
• Drawbacks:– Very noisy in front of reflective properties of the objects, non-
linearities in the camera spectral response and projector spectrum non-standard light emitters are required in order to project single wave-lengths
– Low accuracy (order of 1 mm)
26
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
25/36
Direct codification: grey levels (I)
• Every encoded point of the pattern is identified by its intensity level
Carrihill and Hummel Intensity Ratio Sensor:
fade from balck to white
Requirements to obtain high resolution
Every slit is identified by its own intensity
• Every slit must be projected using a single wave-length
• Cameras with large depth-per-pixel (about 11 bits) must be used in order to differentiate all the projected intensities
27
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Direct codification: grey levels (II)
26/36
Carrihill and Hummel
Chazan and Kiryati Grey levels
Hung
Static
Scene applicability Moving
Binary
Grey levels
Pixel depth
Colour
Periodical Coding strategy
Absolute
28
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
Direct codification: Colour (I)
27/36
• Every encoded point of the pattern is identified by its colour
Tajima and Iwakawa rainbow
pattern
(the rainbow is generated with a
source of white light passing through a
crystal prism)
T. Sato patterns capable of cancelling the object colour by
projecting three shifted patterns
(it can be implemented with an LCD projector if few colours are projected drawback: the pattern
becomes periodic in order to maintain a good
resolution)
29
Conclusions
Sub-pixel matching
Experiments
Introduction
Classification
Time-multiplexing
Spatial codification
Direct codification
28/36
Direct codification: Colour (II)
Tajima and Iwakawa
Smutny and Pajdla
Geng
Wust and Capson
Colour
T. Sato
Static
Scene applicability Moving
Binary
Grey levels
Pixel depth
Colour
Periodical Coding strategy
Absolute
30
Conclusions
Sub-pixel matching
Classification
Introduction
Experiments
Quantitative results
Qualitative results
Experiments: quantitative results
29/36
TechniqueStDev(m
)3D
PointsResolutio
n (%)*Pattern
s
Time-multiplexin
g
Binary code
Posdamer (128 stripes)
37.6 13013 11.18 9
n-ary codeHorn (64 stripes)
9.6 12988 11.15 5
Phase Shift +
Gray code
Gühring (113 slits)
4.9 27214 23.38 14
Spatical codification
De Bruijn sequence
De Bruijn (64 slits)
13.1 13899 11.94 1
Salvi (2929 grid)
72.3 372 0.32 1
M-arrayMorano
(4545 array)23.6 926 0.80 1
Direct codification
ColourSato (64 stripes)
11.9 10204 8.77 3
Results obtained by reconstructing 30 times two flat panels separated by 40mm. The distance between both panels obtained for each technique was calculated for every reconstruction. The Standard Deviation is indicated.
(*) % of pixels inside a window of 515x226 of the camera image that have been triangulated
31
Conclusions
Sub-pixel matching
Introduction
Experiments
Quantitative results
Qualitative results
Experiments: qualitative results
30/36
Time-multiplexi
ng
Posdamer (128 stripes)
Horn (64 stripes) Gühring (113 slits)
Spatial codificati
on
De Bruijn (64 slits) Salvi (29x29 slits)Morano (45x45 dot
array)
Direct codificati
on
Sato (64 slits)
GühringDe Bruijn
Classification
32
Conclusions
Experiments
Introduction
Sub-pixel matching
Stripe patterns
Other patterns
Classification
Sub-pixel matching: a key point
31/36
• A key point in the accuracy of the 3D measurements is to locate the correspondences between camera image and projector pattern with sub-pixel coordinates
• Qualitative example:
Horse reconstruction with pixel-accuracy triangulation (20000
points)
Horse reconstruction with subpixel-accuracy
triangulation (10000 points)
Camera image
The reflected rays of light fall between
ajdacent pixels in the camera sensor
This must be taken into account when
triangulating the 3D points
Projected pattern
33
Conclusions
Experiments
Sub-pixel matching
Stripe patterns
Other patterns
Classification
Sub-pixel matching: stripe-patterns
32/36
• Only points belonging to the edges between adjacent stripes are decoded and reconstructed. Two possible strategies:
- Intersecting the stripe intensity profile with and adaptative binarization threshold (calculated from two additional images: full illuminated and ambient lighting)
- More accurate: Projecting positive and negative patterns and intersecting the stripe profiles
+
Introduction
34
Conclusions
Experiments
Introduction
Sub-pixel matching
Stripe patterns
Other patterns
Classification
Sub-pixel matching: other patterns
33/36
Arry of dots: the subpixel position of the dots is often calculated with their mass centre not very accurate since the circles are observed like deformed ellipses due to the change of perspective and the measuring surface
Slit patterns: every observed slit can be modelled with a gaussian profile and peak detectors can be applied (like Blais & Rioux detector) very accurate
35
Sub-pixel matching
Experiments
Introduction
Conclusions
Guidelines
Classification
Conclusions
34/36
Types of techniques Time-multiplexing
• Highest resolution• High accuracy• Easy implementation
• Inapplicability to moving objects• Large number of patterns
Spatial codification
• Can measure moving objects• A unique pattern is required
• Lower resolution than time-multiplexing• More complex decoding stage• Occlusions problem
Direct codification• High resolution• Few patterns
• Very sensitive to image noise cameras with large depth-per-pixel required• Sensitive to limited bandwith of LCD projectors special projector devices are usually required• Inapplicability to moving objects
36
Sub-pixel matching
Experiments
Introduction
Conclusions
Guidelines
Classification
Guidelines
35/36
Requirements Best technique
• High accuracy• Highest resolution• Static objects• No matter the number of patterns
Phase shift + Gray code Gühring’s line-shift technique
• High accuracy• High resolution• Static objects• Minimum number of patterns
N-ary pattern Horn & Kiryati Caspi et al.
• High accuracy• Good resolution• Moving objects
De Bruijn pattern Zhang et al.
Computer Vision and Robotics Group
Institut d’Informàtica i Aplicacions
Jordi PagèsJoaquim Salvi
END