MIT Media Lab 2

Polarized 3DAchuta Kadambi, MIT Media Lab

Joint work with Vahe Taamazyan, Boxin Shi, and Ramesh Raskar

MIT Media Lab 3

MIT Media Lab 4

Microsoft Kinect v2

MIT Media Lab 5

Microsoft Kinect v2

MIT Media Lab 6

Microsoft Kinect v2

MIT Media Lab 7

MIT Media Lab 8

Multistripe Laser Scan

MIT Media Lab 9

Multistripe Laser Scan

MIT Media Lab 10

Multistripe Laser Scan

NextEngine 3D$3000 USDRaster

MIT Media Lab 11

Multistripe Laser Scan

NextEngine 3D$3000 USDRaster

MIT Media Lab 12

MIT Media Lab 13

Polarized 3D

3D Photo w. $30 Pol. Filter

MIT Media Lab 14

Polarized 3D

3D Photo w.$30 Pol. Filter

MIT Media Lab 15

Polarized 3D

3D Photo w.$30 Pol. Filter

MIT Media Lab 16

Plenoptic Light Transport

Adelson and Bergen “The Plenoptic Function…” MIT Press 1991

MIT Media Lab 17

Plenoptic Light Transport

Viewpoint Diversity (Light Field Cam)

Adelson and Bergen “The Plenoptic Function…” MIT Press 1991

MIT Media Lab 18

Plenoptic Light Transport

Viewpoint Diversity (Light Field Cam)

Wavelength Diversity (Hyperspectral Cam)

Adelson and Bergen “The Plenoptic Function…” MIT Press 1991

MIT Media Lab 19

Plenoptic Light Transport

Viewpoint Diversity (Light Field Cam)

Wavelength Diversity (Hyperspectral Cam)

Polarization Diversity(Photos, Shape, Scatter)

Adelson and Bergen “The Plenoptic Function…” MIT Press 1991

MIT Media Lab 20

Plenoptic Light Transport

Viewpoint Diversity (Light Field Cam)

Wavelength Diversity (Hyperspectral Cam)

Polarization Diversity(Photos, Shape, Scatter)

Time of Flight(3D, Scattering)

Adelson and Bergen “The Plenoptic Function…” MIT Press 1991

MIT Media Lab 21

Plenoptic Light Transport

Viewpoint Diversity (Light Field Cam)

Wavelength Diversity (Hyperspectral Cam)

Polarization Diversity(Photos, Shape, Scatter)

Time of Flight(3D, Scattering)

Bounce Index(Scattering)

Adelson and Bergen “The Plenoptic Function…” MIT Press 1991

MIT Media Lab 22

Plenoptic Light Transport

Viewpoint Diversity (Light Field Cam)

Wavelength Diversity (Hyperspectral Cam)

Polarization Diversity(Photos, Shape, Scatter)

Time of Flight(3D, Scattering)

Bounce Index(Scattering)

Adelson and Bergen “The Plenoptic Function…” MIT Press 1991

MIT Media Lab 23

Plenoptic Light Transport

Viewpoint Diversity (Light Field Cam)

Wavelength Diversity (Hyperspectral Cam)

Polarization Diversity(Photos, Shape, Scatter)

Time of Flight(3D, Scattering)

Bounce Index(Scattering)

Adelson and Bergen “The Plenoptic Function…” MIT Press 1991

MIT Media Lab 24

Plenoptic Light Transport

Viewpoint Diversity (Light Field Cam)

Wavelength Diversity (Hyperspectral Cam)

Polarization Diversity(Photos, Shape, Scatter)

Time of Flight(3D, Scattering)

Bounce Index(Scattering)

Adelson and Bergen “The Plenoptic Function…” MIT Press 1991

MIT Media Lab 25

Plenoptic Light Transport

Viewpoint Diversity (Light Field Cam)

Wavelength Diversity (Hyperspectral Cam)

Polarization Diversity(Photos, Shape, Scatter)

Time of Flight(3D, Scattering)

Bounce Index(Scattering)

Adelson and Bergen “The Plenoptic Function…” MIT Press 1991

MIT Media Lab 26

Plenoptic Light Transport

Viewpoint Diversity (Light Field Cam)

Wavelength Diversity (Hyperspectral Cam)

Polarization Diversity(Photos, Shape, Scatter)

Time of Flight(3D, Scattering)

Bounce Index(Scattering)

Adelson and Bergen “The Plenoptic Function…” MIT Press 1991

MIT Media Lab 27

Plenoptic Light Transport

Viewpoint Diversity (Light Field Cam)

Wavelength Diversity (Hyperspectral Cam)

Polarization Diversity(Photos, Shape, Scatter)

Time of Flight(3D, Scattering)

Bounce Index(Scattering)

Adelson and Bergen “The Plenoptic Function…” MIT Press 1991

MIT Media Lab 28

Plenoptic Light Transport

Viewpoint Diversity (Light Field Cam)

Wavelength Diversity (Hyperspectral Cam)

Polarization Diversity(Photos, Shape, Scatter)

Time of Flight(3D, Scattering)

Bounce Index(Scattering)

Adelson and Bergen “The Plenoptic Function…” MIT Press 1991

MIT Media Lab 29

Polarization of Light

MIT Media Lab 30

Polarization of Light

MIT Media Lab 31

Polarization of Light

Plane of Polarization

MIT Media Lab 32

Polarization of Light

Plane of Polarization Plane of Polarization

MIT Media Lab 33

Brewster’s Angle

MIT Media Lab 34

Cool 2D Photos

Photo Credit: Bob Atkins

MIT Media Lab 35

Cool 2D Photos

Photo Credit: Bob Atkins

MIT Media Lab 36

Polarization used in 2D photography…

… But what about Polarizers for 3D Cams?

MIT Media Lab 37

Shape from Polarization

MIT Media Lab 38

Shape from Polarization

MIT Media Lab 39

Shape from Polarization

MIT Media Lab 40

Shape from Polarization

MIT Media Lab 41

Shape from Polarization

MIT Media Lab 42

Shape from Polarization

Old Principle [Fresnel 1819]

cos cosc oso cs

i

i t

tnrn

θ θθ θ⊥ +−

=

coscos cco

oss i t

t i

rnnθ θ

θ θ+−

=

SfP crux: Solve for theta

MIT Media Lab 43

Shape from Polarization

Old Principle [Fresnel 1819]

cos cosc oso cs

i

i t

tnrn

θ θθ θ⊥ +−

=

coscos cco

oss i t

t i

rnnθ θ

θ θ+−

=

SfP crux: Solve for theta

Need to know refractive index

MIT Media Lab 44

MIT Media Lab 45

MIT Media Lab 46

MIT Media Lab 47

MIT Media Lab 48

MIT Media Lab 49

MIT Media Lab 50

MIT Media Lab 51

MIT Media Lab 52

MIT Media Lab 53

MIT Media Lab 54

Can use Schechner 15 ICCP

MIT Media Lab 55

Image Formation Model

MIT Media Lab 56

Image Formation Model

( )( )max min max minpol pol( ) cos 2

2 2I II II φ φ φ+

+ −= −

MIT Media Lab 57

Image Formation Model

( )( )max min max minpol pol( ) cos 2

2 2I II II φ φ φ+

+ −= −

Suppose and 'φ φ φ π∃ = +

MIT Media Lab 58

Image Formation Model

( )( )max min max minpol pol( ) cos 2

2 2I II II φ φ φ+

+ −= −

Suppose and 'φ φ φ π∃ = +

Azimuthal Ambiguity problem with 2 solutionsP

MIT Media Lab 59

Why is Shape from Polarization Unpopular?

1. \pi Ambiguity in Surface Normal

MIT Media Lab 60

Why is Shape from Polarization Unpopular?

1. \pi Ambiguity in Surface Normal

MIT Media Lab 61

Why is Shape from Polarization Unpopular?

1. \pi Ambiguity in Surface Normal

2. Refractive Distortion

MIT Media Lab 62

Why is Shape from Polarization Unpopular?

1. \pi Ambiguity in Surface Normal

2. Refractive Distortion

MIT Media Lab 63

Why is Shape from Polarization Unpopular?

1. \pi Ambiguity in Surface Normal

2. Refractive Distortion

3. Low SNR for some geometries

MIT Media Lab 64

Why is Shape from Polarization Unpopular?

1. \pi Ambiguity in Surface Normal

2. Refractive Distortion

3. Low SNR for some geometries

4. Usual challenges of integrating surface normals..

MIT Media Lab 65

Shape from Polarization

Miyazaki ICCV 2003 Atkinson TIP 2006

Polarization Inverse Rendering Shape from Diffuse Polarization

MIT Media Lab 66

Shape from Polarization

Miyazaki ICCV 2003 Atkinson TIP 2006

Polarization Inverse Rendering Shape from Diffuse Polarization

SfP never as popular as shading or photometric stereo

MIT Media Lab 67

Potential of Polarization?

• Passive capture

• Break past Lambertian assumption

• No baseline

• Robust to Lighting

MIT Media Lab 68

Frequency Analysis

MIT Media Lab 69

Frequency Analysis

MIT Media Lab 70

Frequency Analysis

MIT Media Lab 71

Frequency Analysis

MIT Media Lab 72

Frequency Analysis

MIT Media Lab 73

Frequency Analysis

MIT Media Lab 74

Frequency Analysis

MIT Media Lab 75

Polarized 3D Fuses Depth and Polarization

Spanning Tree Integration

MIT Media Lab 76

Polarized 3D Fuses Depth and Polarization

Spanning Tree Integration

MIT Media Lab 77

Polarized 3D Fuses Depth and Polarization

Spanning Tree Integration

MIT Media Lab 78

Polarized 3D Fuses Depth and Polarization

Spanning Tree Integration

MIT Media Lab 79

Polarized 3D Fuses Depth and Polarization

Spanning Tree Integration

MIT Media Lab 80

Visual Debugging

Kinect 3 Polar Photos

MIT Media Lab 81

Visual Debugging

Kinect 3 Polar Photos SfP

MIT Media Lab 82

Visual Debugging

Kinect 3 Polar Photos SfP Grad. Corr.

MIT Media Lab 83

Visual Debugging

Kinect 3 Polar Photos SfP Grad. Corr. Spanning Tree

MIT Media Lab 84

Versus Expensive Laser Scanner?

Polarized 3D NextEngineMultistripe

MIT Media Lab 85

Assumptions

Unpolarized World Assumption

Dielectric or Low-frequency Material Transition

No specular interreflections

Diffuse-dominant or Specular-dominant surfaces with slack

MIT Media Lab 86

Break the Lambertian Assumption

MIT Media Lab 87

Break the Lambertian AssumptionKinect

MIT Media Lab 88

Break the Lambertian AssumptionShading [Wu 14]Kinect

MIT Media Lab 89

Break the Lambertian AssumptionPolarized 3DShading [Wu 14]Kinect

MIT Media Lab 90

Break the Lambertian AssumptionPolarized 3DShading [Wu 14]Kinect

MIT Media Lab 91

Break Lighting Assumptions

Kinect

MIT Media Lab 92

Break Lighting Assumptions

Polarized 3DShading [Wu 14]Kinect

MIT Media Lab 93

Break Lighting Assumptions

Polarized 3DShading [Wu 14]Kinect

MIT Media Lab 94

Break Lighting Assumptions

Polarized 3DShading [Wu 14]Kinect

MIT Media Lab 95

Polarized 3D

Collaborators:Vage Taamazyan

Boxin ShiRamesh Raskar

DIY Guide planned for 2016

Thanks to:Gary Atkinson

Terry BoultIn Kyu Park

media.mit.edu/~achoo/polar3Dachoo@mit.edu

MIT Media Lab 96

MIT Media Lab 97

MIT Media Lab 98

MIT Media Lab 99

MIT Media Lab 100

Polarized 3DAchuta Kadambi, MIT Media Lab

Joint work with Vahe Taamazyan, Boxin Shi, and Ramesh Raskar

MIT Media Lab 101

Polarized 3DAchuta Kadambi, MIT Media Lab

Joint work with Vahe Taamazyan, Boxin Shi, and Ramesh Raskar

MIT Media Lab 102

Polarized 3D Power of Physics VisionAchuta Kadambi, MIT Media Lab

Joint work with Vahe Taamazyan, Boxin Shi, and Ramesh Raskar