Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Camera CultureAssociate Professor, MIT Media Lab http://raskar.info

How can we create an entirely new class of imaging platforms

that have an understanding of the world that far exceeds human ability

and produce meaningful abstractions that are well within human comprehensibility ?

Ramesh Raskar http://raskar.info

Can you look around a corner ?

Can you decode a 5 micron feature Can you decode a 5 micron feature from 3 meters away from 3 meters away

with an ordinary camera ?with an ordinary camera ?

Beyond Multi-touch: Mobile 3D Interfaces?

6D DisplayLight sensitive 4D display

One Pixel of a 6D Display = 4D Display Raskar, Saakes, Fuchs, Siedel, Lensch, 2008

A. Introduction 5 minutes ‐‐

B. Cameras of the future (Raskar, 30 minutes) * Form factors, Modalities and Interaction * Enabling Visual Social Computing

C. Reconstruction the World (Seitz, 30 minutes) * Photo tourism and beyond * Image based modeling and rendering on a massive scale ‐* Scene summarization

D. Understanding a Billion Photos (Efros, 30 minutes) * What will the photos depict? * Photos as visual content for computer graphics * Solving computer vision

E. Discussion 10 minutes ‐‐

Course: Next Billion CamerasWedn at 3:30pm

Looking around a corner via Transient Imaging

Coded AperturePost-capture Focus

Control

Bokode: Long Distance Barcode

Camera Culture Group, MIT Media Lab Ramesh Raskar http://raskar.info

BiDi Screen

Light Field

Augmented Light Field

WDF

Theory of Light Propagation

International Conference on Computational Photography

Papers due November 2,

2009

http://cameraculture.media.mit.edu/iccp10

Traditional PhotographyTraditional Photography

Lens

Detector

Pixels

Image

Mimics Human Eye for a Single Snapshot:

Single View, Single Instant, Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy: Shree

Nayar

Computational Illumination

Computational Camera

Scene: 8D Ray Modulator

Display

GeneralizedSensor

Generalized OpticsProcessing

4D Ray BenderUpto 4D

Ray Sampler

Ray Reconstruction

Generalized Optics

Recreate 4D Lightfield

Light Sources

Modulators

4D Incident Lighting

4D Light Field

Computational PhotographyComputational Photography

Computational Photography [Raskar and Tumblin]

1. Epsilon Photography– Low-level vision: Pixels– Multi-photos by perturbing camera parameters– HDR, panorama, …– ‘Ultimate camera’

2. Coded Photography– Mid-Level Cues:

• Regions, Edges, Motion, Direct/global– Single/few snapshot

• Reversible encoding of data– Additional sensors/optics/illum– ‘Scene analysis’

3. Essence Photography– High-level understanding

• Not mimic human eye• Beyond single view/illum

– ‘New artform’

Digital

Epsilon

Coded

Essence

Goal and Experience

Low Level Mid Level HighLevel

Hyper realism

Raw

Angle, spectrum

aware

Non-visual Data, GPS

Metadata

Priors

Comprehensive

Computational Photography aims to make progress on

both axis

Camera ArrayHDR, FoV Focal stack

Decomposition problems

Depth

Spectrum

LightFields

Human Stereo Vision

8D reflectance field

Dylan’s bicycle

Transient Imaging

Virtual Object Insertion

Relighting

Augmented Human

Experience

Material editing from single photo

Scene completion from photos

Motion Magnification

• Ramesh Raskar and Jack Tumblin

• Book Publishers: A K Peters• Siggraph 2009 booth: 20% off • Booth #2527

• ComputationalPhotography.org

• Meet the Authors• Thursday at 2pm-2:30pm

Can you look around a corner ?

Can you look around a corner ?

Kirmani, Hutchinson, Davis, Raskar 2009Accepted for ICCV’2009, Oct 2009 in Kyoto

Impulse Response of a Scene

Femtosecond Laser as Light SourcePico-second detector array as

Camera

Coded Aperture CameraCoded Aperture Camera

The aperture of a 100 mm lens is modified

Rest of the camera is unmodifiedInsert a coded mask with chosen binary pattern

Captured Blurred Photo

Refocused on Person

MIT media lab camera culture

Bokode:

ankit mohan, grace woo, shinsaku hiura,quinn smithwick, ramesh raskar

camera culture group, MIT media lab

imperceptible visual tags for camera based interaction from a distance

MIT media lab camera culture

Barcodesmarkers that assist machines in

understanding the real world

• Smart Barcode size : 3mm x 3mm• Ordinary Camera: Distance 3 meter

Computational Probes: Computational Probes: Long Distance Bar-codesLong Distance Bar-codes

Mohan, Woo,Smithwick, Hiura, RaskarAccepted as Siggraph 2009 paper

MIT Media Lab Camera Culture

Bokode

MIT Media Lab Camera Culture

Defocus blur of Bokode

MIT Media Lab Camera Culture

Image greatly magnified.

Simplified Ray Diagram

MIT Media Lab Camera Culture

Our Prototypes

MIT media lab camera culture

street-view tagging

MIT media lab camera culture

tabletop/surface interaction

MIT media lab camera culture

multi-user interaction

MIT media lab camera culture

Varying Exposure VideoAmit Agrawal MERL, Yi Xu Purdue, Ramesh Raskar, MIT

Deblurred Result

Blurred Photo

Varying Exposure Video

DFT

Exposure Time

Exposure Time

Exposure Time

Can we deal with particle-wave duality of light with modern Lightfield theory ?

45

Young’s Double Slit Expt

first null (OPD = λ/2)

Diffraction and Interferences modeled using Ray representation

Light Fields

•Radiance per ray•Ray parameterization:•Position : x •Direction : θ Reference

plane

positiondirection

Goal: Representing propagation, interaction and image formation of light using purely position and angle

parameters

Light Fields for Wave Optics EffectsLight Fields for Wave Optics Effects

Wigner Distribution Function

Light Field

LF < WDF

Lacks phase propertiesIgnores diffraction, phase masks

Radiance = Positive

Light Field

Augmented Light Field

WDF

ALF ~ WDF

Supports coherent/incoherent

Radiance = Positive/NegativeVirtual light sources

Limitations of Traditional Lightfields

Wigner Distribution Function

TraditionaTraditional Light l Light FieldField

ray optics basedsimple and powerful

rigorous but cumbersomewave optics based

limited in diffraction & interference

holograms

beam shaping

rotational PSF

Example: New Representations Augmented Lightfields

Wigner Distribution Function

TraditionaTraditional Light l Light FieldField

WDF

TraditionaTraditional Light l Light FieldField

Augmented LF

Interference & DiffractionInteraction w/ optical elements

ray optics basedsimple and powerfullimited in diffraction & interference

rigorous but cumbersomewave optics based

Non-paraxial propagation

http://raskar.scripts.mit.edu/~raskar/lightfields/

(ii) Augmented Light Field with LF Transformer

50

WDF

Light Light FieldField

Augmented LF

Interaction at the optical elements

LF propagatio

n

(diffractive)

optical element

LF LF LF LF

LF propagatio

n

light field transformer

negative radiance

Augmenting Light Field to Model Wave Optics Effects , [Oh, Barbastathis, Raskar]

Virtual light projector with real valued (possibly negative radiance) along a ray

51

real projector

real projector

first null (OPD = λ/2)

virtual light projector

(ii) ALF with LF Transformer

52

Converting LCD Screen = large Camera for 3D Interactive HCI and Video Conferencing

Matthew Hirsch, Henry HoltzmanDoug Lanman, Ramesh Raskar

BiDi Screen*

Light Sensing Pixels in LCD

Dis

play

with

em

bedd

ed o

ptic

al s

enso

rs

Sharp Microelectronics Optical Multi-touch Prototype

Beyond Multi-touch: Hover Interaction

• Seamless transition of multitouch to gesture

• Thin package, LCD

Beyond Multi-touch: Mobile

Laptops

Mobile

Design Vision

Object Collocated Captureand Display

Bare Sensor

Spa

tial L

ight

Mod

ulat

or

Touch + Hover using Depth Sensing LCD Sensor

Overview: Sensing Depth from Array of Virtual Cameras in

LCD

Design Overview

Dis

play

with

em

bedd

ed o

ptic

al s

enso

rs

LCD , displaying mask

Opt

ical

sen

sor a

rray

~2.5 cm~50 cm

International Conference on Computational Photography

Papers due November 2,

2009

http://cameraculture.media.mit.edu/iccp10

Looking around a corner via Transient Imaging

Coded AperturePost-capture Focus

Control

Bokode: Long Distance Barcode

Camera Culture Group, MIT Media Lab Ramesh Raskar http://raskar.info

BiDi Screen

Light Field

Augmented Light Field

WDF

Theory of Light Propagation