Non Conventional Imaging Systems : Application to 3D ... fileNon Conventional Imaging Systems :...

Non Conventional Imaging Systems :

Application to 3D scanning of Transparent Objects

Fabrice MERIAUDEAU

Laboratory Le2i UMR 5158

Jakarta, Indonesia

06- 01-2012

Contributors:

M. Ferraton, R. Rantoson, G. Eren, L. Sanchez-Sécades (PhD Students)

C. Stolz, O. Aubreton, F. Truchetet, D. Fofi, R. Seulin (Academic collaborators)

3D Digitization

Contact

Touching Probe

…..

Non Contact

Reflection Transmission Emission

Outlines

Introduction

State of the Art

Scanning From Heating

Background

Method

Application to Glass

Implementation & Experimental Results

Shape from polarization

Polarization Imaging

Applications

« Shape from polarization »

Experimental System and result

Shape from UV

Background

Implemenation and results

Conclusion

Passive 3D Scanning Stereo Vision Shape From Focus

Active 3D Scanning

Laser Triangulation Pattern Projection Time of Flight

Manual Scanning

• Long

• Tedious

• Highly skilled operator

• Results

=> tied to operator’s expertise

Design Solution for 3D Digitization

◦ Results => Independent of User Skills

◦ Automatic & Fast Acquisition

◦ Optimized View Planning

for Maximal Surface Coverage

◦ Post-Processing of Delivered 3D Model

Object 3D Model

Manual

Teaching

Model Based

Non Model Based

Automation Robot CMM Arm => Internal 7-axis articulated arm with an external skeleton driven by electromotors Drives laser line scanner => programmed motion path

Automation => Manual Teaching

http://www.metris.com

../../../Recherche/presentations/3D_Scanning_ViBot_2010/353.rca_lr.wmv

Automation Fringe Projection Scanner Head Mounted on Industrial Robot Arm

Automation => Manual Teaching

http://www.steinbichler.de

../../../Recherche/presentations/3D_Scanning_ViBot_2010/Comet_512k.wmv

Model Based – Offline View Planning

Sensor & Robot Modeling

1 Face = 1 ViewPoint

Visibility Study : Binary Table

Optimization : Set Covering Problem

Automation => Model Based

Non Model Based – Online View

Planning Surface

•Mass Vector Chains (MVC)

•Sum of Surface Normal’s

•Weighted by Surface‘s Area

Automation => Non Model Based

Components

Fringes Projection

3D scanner head

6 DOF

Robot Turntable

Automation => Implementation

3D Digitization Cell

Automation => Implementation

../../../Recherche/presentations/3D_Scanning_ViBot_2010/Robot_Demo_Finished_1_.avi

Types of Surfaces

Diffuse (Lambertian) Glossy Specular

Translucent Transparent

Type de surface

19 Diffuse Surface Specular Surface Transparent Surface

Type de surface

20 Diffuse Surface Specular Surface Transparent Surface

Problem to be solved

« Laser scanning »

21 Surface transparente

Outlines

Introduction

State of the Art


Background

Method





Applications



Shape from UV

Background


Conclusion

➡ New methods:

➡“Scanning From Heating”

➡Shape from polarization

➡Shape from UV

‣ Capable of scanning different transparent materials and type of surfaces

Increasing demand for three-dimensional (3D) applications

◦ object modeling, preservation of historic artifacts, quality control,...

Transparent objects have not received much attention

High demand for in-line 3D quality control of transparent products

For complex geometric forms: touch probe scanners

◦ too slow for inline inspection

◦ statistical sampling

INTRODUCTION

State of the Art Methods

Structured LightS. Hata, Y. Saitoh, S. Kumamura, and

K. Kaida, “Shape extraction of

transparent object using genetic

algorithm," in Pattern Recognition,

Proceedings of the 13th International

Conference on, vol. 4, 1996.

Shape From MotionM. Ben-Ezra and S. Nayar, "What

does motion reveal about

transparency?" in Proc. IEEE Int'l

Conf. Computer Vision, 2003, pp.

1025-1032.

Optical FlowS. Agarwal, S. Mallick, D. Kriegman,

and S. Belongie, "On refractive optical

flow," in Proc. ECCV'04, 2004, pp.

483-494.

Structured LightS. Hata, Y. Saitoh, S. Kumamura, and

K. Kaida, “Shape extraction of

transparent object using genetic

algorithm," in Pattern Recognition,

Proceedings of the 13th International

Conference on, vol. 4, 1996.

Shape From MotionM. Ben-Ezra and S. Nayar, "What

does motion reveal about

transparency?" in Proc. IEEE Int'l

Conf. Computer Vision, 2003, pp.

1025-1032.

Optical FlowS. Agarwal, S. Mallick, D. Kriegman,

and S. Belongie, "On refractive optical

flow," in Proc. ECCV'04, 2004, pp.

483-494.

State of the Art Methods

FluorescenceM. B. Hullin, M. Fuchs, I. Ihrke, H.-P.

Seidel, and H. P. A. Lensch,

"Fluorescent immersion range

scanning," ACM Trans. Graph., vol.

27, no. 3, pp. 1-10, 2008.

PolarizationD. Miyazaki and K. Ikeuchi, "Inverse

polarization raytracing: estimating

surface shapes of transparent

objects," in IEEE Computer Society

Conference on Computer Vision and

Pattern Recognition, vol. 2, 2005, p.

910.

Shape From

DistortionM. Tarini, H. Lensch, M. Goesele, and

H. Seidel, "3d acquisition of mirroring

objects using striped patterns,"

Graphical Models, vol. 67, no. 4, pp.

233-259, 2005.

FluorescenceM. B. Hullin, M. Fuchs, I. Ihrke, H.-P.

Seidel, and H. P. A. Lensch,

"Fluorescent immersion range

scanning," ACM Trans. Graph., vol.

27, no. 3, pp. 1-10, 2008.

PolarizationD. Miyazaki and K. Ikeuchi, "Inverse

polarization raytracing: estimating

surface shapes of transparent

objects," in IEEE Computer Society

Conference on Computer Vision and

Pattern Recognition, vol. 2, 2005, p.

910.

Shape From

DistortionM. Tarini, H. Lensch, M. Goesele, and

H. Seidel, "3d acquisition of mirroring

objects using striped patterns,"

Graphical Models, vol. 67, no. 4, pp.

233-259, 2005.

Outlines

Introduction

State of the Art


Background

Method





Applications



Shape from UV

Background


Conclusion

Transparency

Human flesh is transparent to X-Ray

while bone is not

Electromagnetic Spectrum

Transparency - Example Case of

Glass

Transparent glass bottle in front of an infrared heat sourceand Image

taken with a long wave infrared camera

Evolution of the refraction and absorption index

of glass depending on the wavelength

Glass is Opaque in UV and in IR

We have developed methods making use of this property

Thermal Radiation

Energy distribution of a blackbody Thermal Image of an house

Thermal Radiation

*>47.7°C

*<-11.6°C

-10.0

-5.0

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

45.0

Application to Glass - Emissivity

Emissivity of a glass sphere

๏ G. Gaussorgues and S. Chomet, Infrared thermography. Kluwer Academic Publishers,

1994.

A new approach:

Scanning From Heating G. Eren, F. Meriaudeau and al., Optics Express 2009.


For the method to work properly:

The object surface should be opaque to the laser source i.e. the laser should be absorbed by the

surface

The object has to emit Once the surface is heated, the thermal emission should be omnidirectional

so that the thermal camera can capture accurately the heated point on different curvatures of the

surface

Homogenous, Isotropic Uniform in structure and composition

Physical properties of the material are independent of direction

Application to Glass - Heating

Model

P0, power of the CO2 laser beamr, radius of the CO2 laser beam∂(z), impulse function

๏ Jiao, J., Wang, X.: A numerical simulation of machining glass by

dual

CO2-laser beams. Optics and Laser Technology 40(2) (2008) 297-301

Application to Glass - Heating

Model • To bring the surface form 20 to 80 degrees with v=10mm/s, r=1.5mm

➡Laser power : 3W

Experimental results Experimental results compared to the heating model

Application to Glass - Selection of

the Camera

Transmission of light as a percentage in the infrared

domain

Transmission of light as a percentage in the

infrared domain of commonly used glasses

The Scanner

Back View

Camera Calibration

Custom calibration plate

Calibration image set

Results - Glass Plate

Results obtained on a 10x5 cm glass plate

• The reconstruction is compared to a perfect plane

• Average deviation is 150 µm

Results - Glass Cup

Transparent glass cup

3D reconstruction

by SFH Scanner 3D reconstruction (after being powdered)BY Minolta 3D Laser Scanner

Comparison of the reconstructions

(average deviation is 210 µm)

Results - Automotive Glass

Automotive Glass

Reconstruction by a Touch Probe Scanner Comparison of the reconstructions


Results - Plastic

Transparent plastic bottle 3D reconstruction by SFH Scanner

Comparison to Minolta VI-910


Line Projection : an extension

Experimental Setup

Meriaudeau and al., IEEE TIM, 2010

Object Reconstruction Object is Powdered

for Comparison

Comparison to Minolta VI-910


46

Specular object: an extension, A. Bajard, PhD Thesis, 2009/2012

Outlines

Introduction

State of the Art


Background

Method





Applications



Shape from UV

Background


Conclusion

Electromagnetic Plane waves

Non polarized wave : random phase

Polarized wave :

◦ Linear polarization

◦ Circular polarisation

rktieEE

0

22

11

cos

cos

aE

aE

y

x

rktavec 12: shiftphase

t

0dt

d

mwithm

mwithm

aa

212

21

49

Stokes Vector :

Total Intensity :

Degree of Polarization :

Angle of polarization :

1

2tana

a

0

2

3

2

2

2

1

s

sss

I

I

tot

pol

2

3

2

2

2

1

2

0 ssss

sin2sin

cos2sin

2cos

sin2

cos2

21

21

2

2

2

1

2

2

2

1

3

2

1

0

tot

tot

tot

tot

I

I

I

I

aa

aa

aa

aa

s

s

s

s

s

50

Mueller Matrix : describes the effects of

linear optical systems (polarizers, wave

plates, reflecting surfaces…)

Example : Linear polarizer inclined with an

angle α

51

0000

02sin2sin2cos2sin

02sin2cos2cos2cos

02sin2cos1

22

2

polM

52

Polarization imaging

Polarization imaging in industrial vision:

◦ Avoid specular reflections

◦ Distinguish dielectrics from metallics (Wolff)

◦ …

> Shape from polarization:

Polarization Images Information on the surface normals

Physical principle: after reflection on a surface, an unpolarized light

wave becomes partially linearly polarized.

Studying the state of polarization of the reflected light enables to get

information on the surface normals (Fresnel coefficients).

53

Polarization imaging

Polarization state of a light wave:

◦ Goal: study the state of a partially linearly polarized light

No need to have a stokes polarimeter

Just a rotating polarizer in front of the camera

Imi

n

Imax

180°

minmax

minmax

II

II

• Degree of polarization:

• Magnitude of the light: minmax III

• Angle of polarization: 3 parameters:

Partial Stokes’ polarimeters

Measure of S0, S1, S2

Set-up with a linear polarizer

54

Set-up with elliptical nematics liquid crystal

Set-up with LVCR

55 Wolff et al., 1997

Bigue & Cheney, 2007

Partial Stokes’ polarimeters

Polarization imaging applications

Active polarimetry

◦ Depolarization

56 Morel et al., 2006

Alouini, 2005


57

Study of the polarization state

of the reflected wave

Normals Determination

Normal field integration

so as to obtain the surface


58

Principle:

◦ Unpolarized light

◦ Reflective surface

x

y

z

n

i

r

1

sintan

costan

r

r

q

p

n

( , r ) ?

◦ Angle of polarization

◦ Degree of polarization

r

Fresnel reflection coefficients

Angle of polarization

)²(tan

)²(tan

)²(sin

)²(sin

//

ti

ti

ti

ti

F

F

//FF

x

y

z

i

r

n

The linearly polarized component is orthogonal to the plan of incidence

x

y

2


1st ambiguïty

Degree of polarization r

n z

i

r

t

)²(tan

)²(tan

)²(sin

)²(sin

//

ti

ti

ti

ti

F

F

ti n sinsin

Fresnel reflection coefficients Snell-Descartes Law

)(//

//rf

FF

FF

x

y

z

i

r

n



Relation between the degree of

polarisation ρ and the zenithal angle θ

61

2222

22

tansinsin

sintansin2

n

n

2nd ambiguïty

62

Previous approaches of the « Shape from

polarization » for transparent objects



63 Miyazaki et al., 2002

64 Miyazaki et al., 2004



Solutions


65 [Ferraton, 2009] Ferraton, M., Stolz, C. and Meriaudeau, F., "Optimization of a polarization imaging system for 3D measurements

of transparent objects", Optics Express, Optical Society of America, vol. 17 (23), pp. 21077-21082, 2009.

ρ = f(θ,n)

n = f(λ)

λ1 ρ1

λ2 ρ2

ρ = f(θ,λ)

θ

ρ

Solutions


66 Ferraton & Meriaudeau, 2009

θB θ

ρ

θ1

ρλ1

ρλ2

ρλ2 - ρλ1 > 0 0° < θ1 < θB

Solutions



θB θ

ρ

θ2

ρλ1 ρλ2

ρλ2 - ρλ1 < 0 θB < θ2 < 90°

Solutions



θB θ

ρ

θ2

ρλ2 - ρλ1 < 0 ρλ2 - ρλ1 > 0

θ1

0° < θ1 < θB θB < θ2 < 90°

ρλ1

ρλ2

ρλ1

Outlines

Introduction

State of the Art


Background

Method





Applications



Shape from UV

Background


Conclusion

Experimental System

70

12 bit-depth Camera

Orthographic projection Rotating polarizer

Experimental system

71

0° 10°

180°

Acquisition

I

Approximation

Normal extraction

1

sintan

costan

q

p

n

Time < 1sec

Liquid Crystal Variable Retarder (LCVR)

72

λ = 632.8 nm

Experimental system

Optical calibration

73

Rotating polarizer

74

Optical calibration

75

Angle (°) Tension (V)

0 1.565

-5 1.590

-10 1.620

-15 1.650

-20 1.675

-25 1.705

-30 1.745

-35 1.775

-40 1.805

-45 1.835

-50 1.865

-55 1.900

-60 1.935

-65 1.975

-70 2.010

-75 2.055

-80 2.095

-85 2.145

… …

Optical calibration

◦ Saliency operator

76 Walter, N., Aubreton, O. and Laligant, O., "Salient point characterization for low resolution meshes", IEEE International Conference

on Image Processing , pp. 1512-1515, 2008

Optical calibration

77

λB = 472 nm

Inte

nsité lu

min

eu

se

Optical calibration

Optical Calibration Zenithal Angle Ambiguity

78

79


80


81

2222 1tansin

tansin2

kn

n


82

2222 1tansin

tansin2

kn

n

11ˆ 222 knn

n̂


Pseudo-index

3D Reconstruction

83

0° 10°

180°

Acquisition

I

Approximation

Normal extraction

3D Reconstruction

Multispectral approach to relieve the ambiguity on the Zenithal

Angle

84

Active lighting approach to relieve the ambiguity on the azimutal

angle

85

2/.1

,03100.2 quadquadquad IIIsi

Morel et al., 2005

3D Reconstruction

Active lighting approach to relieve the ambiguity on the azimutal

angle

86

2/.1

,03100.2 quadquadquad IIIsi

Morel et al., 2005

3D Reconstruction

Normal Evaluation

Integration

87

1

sintan

costan

1

,

,

q

p

y

yxfx

yxf

n

22

~~,

~,0,0,

vu

qjvpjuvufvu

Frankot, R. and Chellappa, R., "A method for enforcing integrability in shape from shading algorithms", IEEE Transactions on

Pattern analysis and Machine Intelligence, vol. 10, pp. 439-451, 1988

yxgyxfyxf ,,, 0

kqypxyxg 0,0~0,0~,

88

Std: 0.070 mm , Mean. : -0.015 mm, Min : -0.226 mm, Max : 0.150 mm

90

Shape from polarisation in the IR (results from september 2011)

Expérimenta Set-upl: IR camera Flir 3µm - 5 µm, polariser ZnSe 1µm -15µm, « IR dome »

made of 4x14 resistors (12Ω et 0.25W) with a 9V power generator. One resistor 60° ~

maximal radiation around 8.7µm according to de Wien’s Law), a piece of glass of complexe

shape

91

« Two ambiguities solved»

Azimutal angle Iquad in the IR

Shape from polarisation in the IR

92

Zénithal Angle complex refraction index for the glass


« Two ambiguities solved»

93

Camera Calibration Bouguet ‘s Toolbox (z0 ~ 300mm)


94

α=0°

α=0° α=90°

Without polariseur


95

Reconstruction par polarisation dans l’IR

Méthode de validation : Carte binaire

Black pixel(110,325) White Pixel (322,292)

70% of the points are useful

)(I

)(I

96


Implémentation et résultats dans l’IR

Polarisation angle Azimutal Angle

97


Degree of polarisation Zenithal Angle

Integration

Outlines

Introduction

State of the Art


Background

Method





Applications



Shape from UV

Background


Conclusion

99

Shape From UV

Principle:

Under the UV irradiations, transparent surface reemit fluorescence in the Visible

Optimization of the excitation wavelegnth

Absorbance

Emission Spectra

100

Principle : Strutured lighting system

Reconstruction by active triangulation

Calibration

Matching

Structure lighting: point or line (or pattern)

Sensors: one or two cameras

3D

Shape From UV

101

Shape From UV

102

Results

Internationale Conferences (2)

• 3D Reconstruction of Transparent Objects Exploiting Surface Fluorescence caused by

UV Irradiation, Rindra Rantoson, Christophe Stolz, David Fofi, Fabrice Meriaudeau, IEEE

International Conference on Image Processing (ICIP), Hong Kong, September 2010.

• Non Contact 3D Measurement Scheme for Transparent Objects using UV Structured

Light, Rindra Rantoson, David Fofi, Christophe Stolz, Fabrice Meriaudeau, IEEE International

Conference on Pattern Recognition (ICPR), Istanbul, Turkey, August 2010

Workshop (1)

• Triangulation par stéréovision basée sur l'exploitation des images de fluorescence d'une

surface transparente, Rindra Rantoson, Christophe Stolz, David Fofi, Fabrice Meriaudeau,

Journées imagerie optique non conventionnelle, Paris, France, GDR ISIS, 22 March 2010

Book (1)

• "Scanning from Heating" and "Shape from Fluorescence" two Non Conventional

Imaging Systems for 3D Digitization of transparent objects, Fabrice Mériaudeau, R.

Rantoson, G. Eren, L. Sanchez-Sécades, O.Aubreton, A. Bajard, D. Fofi, I. Mohammed, O.

Morel, C. Stolz, F. Truchetet, IGI Global, Novembre 2011

Active Triangulation by stereovision : Publications

Shape From UV

103

Active Triangulation by monocular vision

two experimental set-ups

Point structured lighting system Line structured lighting system

Shape From UV

104

Shape From UV

Experimental set-up

Active Triangulation by monocular vision

105

Implementation and results : Expérimental Set-up

- UV Laser (266m, 10mW, spot elliptical shape of 2mm)

- CCD RGB (Guppy F-080C, 480x640, 1/3 inch, 8mm focal, 1.4 f-number

- Displacement table

Shape From UV

106

Shape From UV

107

Shape From UV

108

µ = 0.07mm, σ=0.07mm

µ = 0.08mm, σ=0.1mm

Shape From UV

109

µ = 0.08mm, σ= 0.09mm

Shape From UV

110

Experimental set-up

- A UV lasre (266m, 10mW, spot elliptique de 2mm)

- A semi-cylindrical « UV proof » lens

- CCD RGB (Guppy F-080C, 480x640, 1/3 inch, 8mm focal,1.4 f-number

- Displacement table

Line structured lighting system

Shape From UV

111

µ = 0.14mm

σ = 0.12mm

Shape From UV

112

Shape From UV

113

Potential Extensions

Specular Surfaces

Optimizing the tracking

Light pre-heating of the surface

Modelisation of the shape of the UV spot or line

Shape From UV

114

Scanning from Heating

High accuracy

No a priori for the

object

Adapted for online

processes

High Cost

High accuracy

No a priori for the object

Adapted for online processes

Low cost

Pre/post-

processing needed

Multispectral « shape

from polarization »

High accuracy

Tedious Calibrations

Post-processing needed

Scanning from UV

Thank you for your attention

Terima Kasih

115

Non Conventional Imaging Systems :

Application to 3D scanning of Transparent Objects

Fabrice MERIAUDEAU

Laboratory Le2i UMR 5158

Jakarta, Indonesia

06- 01-2012

Contributors:

M. Ferraton, R. Rantoson, G. Eren, L. Sanchez-Sécades (PhD Students)

C. Stolz, O. Aubreton, F. Truchetet, D. Fofi, R. Seulin (Academic collaborators)

Non Conventional Imaging Systems : Application to 3D ... fileNon Conventional Imaging Systems :...

Documents

Transcript of Non Conventional Imaging Systems : Application to 3D ... fileNon Conventional Imaging Systems :...