The Camera Computational Photography - ?· 1 1 Computational Photography Yacov Hel-Or and Yossi...

download
  • date post

    25-Aug-2018
  • Category

    Documents
  • view

    212
  • download

    0

Embed Size (px)

transcript

  • 1

    1

    Computational PhotographyComputational Photography

    Yacov Hel-Or

    and

    Yossi Rubner

    2

    The CameraThe Camera

    A camera is a device that takes photos of images

    Camera Obscura (Latin = "dark chamber")

    19th century camera Sonys smile recognition camera

    3

    Camera DevelopmentsCamera Developments

    1825 19901913

    permanent capturing

    (wet plates)

    1850

    exposuretime and

    motion capture(dry plates)

    digital cameras

    quality and size (35 mm)

    1885

    portability(film-Kodak)

    1933

    optics (SLR)

    1950

    instancy(polaroid)

    2000

    computationalphotography

    1826 - Earliest surviving photograph. This image required an eight-hour exposure.

    4

    Computational PhotographyComputational Photography

    Computational photography refers broadly to computational imaging techniques that enhance or extend the capabilities of digital photography. The output of these techniques is an ordinary photograph, but one that could not have been taken by a traditional camera.

    Steve Mann: The Cyberman

    Generated by Foxit PDF Creator Foxit Softwarehttp://www.foxitsoftware.com For evaluation only.

  • 2

    5

    Goal: Record a richer, multiGoal: Record a richer, multi--layered visual experiencelayered visual experience

    1.1. Overcome limitations of todayOvercome limitations of todays camerass cameras

    2.2. Support better postSupport better post--capture processingcapture processing

    3.3. Enables new classes of recording the visual signal Enables new classes of recording the visual signal

    4.4. Synthesize Synthesize impossibleimpossible photosphotos

    6

    AdministrationAdministration

    Pre-requisites / prior knowledge

    Course Home Page:

    http://www1.idc.ac.il/toky/CompPhoto-09/

    Whats new

    Lecture slides and handouts

    Matlab guides

    Homework, grades

    Exercises:

    Programming in Matlab, ~3 Assignments

    Final project

    7

    Administration (Cont.)Administration (Cont.)

    Matlab software: Available in PC labs

    Student version

    For next week: Run Matlab demo and read Matlab primer until section 13.

    Grading policy: Final Grade will be based on: Exercises (60%) , Final project

    (40%)

    Exercises will be weighted

    Exercises can be submitted in pairs

    Office Hours: by email appointment to toky@idc.ac.il

    8

    Project presentation29.01.09

    Light Field22.01.09

    Single View Modeling15.01.09

    Segmentation and Matting08.01.09

    Data Driven Synthesis01.01.09

    Multi exposure enhancement25.12.08

    Multi exposure enhancement18.12.08

    Appearance-based registration11.12.08

    Blending and Composition04.12.08

    Panoramas and feature-based registration27.11.08

    Single exposure enhancement20.11.08

    Acquisition and camera model13.11.08

    Intro and image formation06.11.08

    TopicDate

    ScheduleSchedule

    Generated by Foxit PDF Creator Foxit Softwarehttp://www.foxitsoftware.com For evaluation only.

  • 3

    9

    ReadingsReadings

    Related papers

    New book: Computational Photography by R. Raskarand J. Tumblin

    10

    SyllabusSyllabus Image Formation

    Image formation HVS pathwayColor models

    Acquisition and camera modelCamera model + perspective projectionsSensorsNoise models & DistortionsSampling (spatial+temporal) and quantizationCamera parametersCamera Parameters trade-offs.

    Single exposure enhancementWhite BalancingDe-mosaicingDe-noisingDe-blurringGeometrical distortion correction

    Panoramas and feature based registrationImage featuresSIFTPanoramasFeature based registrationPanoramasHomographyRANSAC Image stitching

    11

    Syllabus Syllabus cont.cont. Blending and Composition

    Pyramid blendingOptimal cutSeam CarvingGraph-cutGradient domain editing

    Appearance based registrationSimilarity measures Lucas Kanade optical flowMulti-modal registrationApplications

    Multi exposure enhancement (2 weeks)HDRSuper-resolutionmulti-exposure fusion

    Data Driven SynthesisTexture synthesisVideo textureQuiltingImage analogiesSuper-ResolutionImage Completion

    12

    Syllabus Syllabus cont.cont. Segmentation and Matting

    Segmentation using Graph cut.mean-shiftSpectral clusteringInteractive and semi-automatic Matting

    Single View ModelingCamera CalibrationMeasurements in affine camera3D reconstruction

    Light FieldPlenoptic function and the Lumiograph

    Re-sampling the plenoptic function

    Generated by Foxit PDF Creator Foxit Softwarehttp://www.foxitsoftware.com For evaluation only.

  • 4

    13

    1. Image Formation1. Image Formation

    Taking a picture

    HVS pathway

    Color models

    Optic NerveFovea

    Vitreous

    Optic Disc

    Lens

    Pupil

    Cornea

    Ocular Muscle

    Retina

    Humor

    Iris

    14

    2. Camera Model and Acquisition2. Camera Model and Acquisition

    Perspective projections

    Camera pipeline and parameters

    Sensors

    Sampling and quantization

    Noise models & Distortions

    Camera Parameters trade-offs.

    15

    3. Single Exposure Enhancement3. Single Exposure Enhancement

    White Balancing

    De-mosaicing

    De-noising

    De-blurring

    Geometrical distortion correction

    Difference in white point

    16

    4. Panoramas and Feature Based Registration4. Panoramas and Feature Based Registration

    Image features

    SIFT

    Feature based registration

    Panoramas

    Homography

    RANSAC

    Image stitching

    Generated by Foxit PDF Creator Foxit Softwarehttp://www.foxitsoftware.com For evaluation only.

  • 5

    17

    5. Blending and Composition5. Blending and Composition

    Pyramid blending

    Gradient domain editing

    Optimal cut

    Graph-cut

    18

    6. Appearance Based Registration (warping?)6. Appearance Based Registration (warping?)

    Similarity measures

    Lucas Kanade optical flow

    Multi-modal registration

    Applications

    19

    7. Multi Exposure Enhancement 7. Multi Exposure Enhancement

    HDR

    Super-resolution

    Different-exposures fusion

    20

    8. Data Driven Synthesis 8. Data Driven Synthesis

    Texture synthesis

    Video texture

    Quilting

    Image analogies

    Super-Resolution

    Image Completion

    Generated by Foxit PDF Creator Foxit Softwarehttp://www.foxitsoftware.com For evaluation only.

  • 6

    21

    9. Segmentation and Matting 9. Segmentation and Matting

    Segmentation using Graph cut.

    mean-shift

    Spectral clustering

    Interactive and semi-automatic Matting

    22

    10. Single View Modeling10. Single View Modeling

    Camera Calibration

    3D reconstruction

    Metrology

    Flagellation by Pietro della Francesca (1416-92, Italian Renaissance period)Animation by Criminisi et al., ICCV 99

    23

    11. Light Field11. Light Field

    Plenoptic function and the Lumiograph

    Re-sampling the plenoptic function

    24

    TodayTodays Topic s Topic -- Image FormationImage Formation

    What is an image ?

    What is a color ?

    Generated by Foxit PDF Creator Foxit Softwarehttp://www.foxitsoftware.com For evaluation only.

  • 7

    25

    Computer Vision

    Rendering

    Image/video Processing

    Model3D Object

    Geometric Modeling

    2D Images

    The Visual SciencesThe Visual Sciences

    26

    What is an Image ?What is an Image ?

    An image is a projection of a 3D scene into a 2D projection plane.

    An image can be defined as a 2 variable function I(x,y) , where for each position (x,y) in the projection plane, I(x,y) defines the light intensity at this point.

    27

    The Pinhole Camera ModelThe Pinhole Camera Model

    Pinhole model: Captures pencil of rays all rays through a single point

    The point is called Center of Projection (COP)

    The image is formed on the Image Plane

    Effective focal length f is distance from COP to Image Plane

    Slide by Steve Seitz

    COP

    Image plane

    Focal length

    28

    Funny things happenFunny things happen

    Generated by Foxit PDF Creator Foxit Softwarehttp://www.foxitsoftware.com For evaluation only.

  • 8

    29

    Parallel lines arenParallel lines arentt

    30

    Lengths canLengths cant be trusted...t be trusted...

    31

    Projection Model (where)Projection Model (where)

    The coordinate system We will use the pin-hole model as an approximation

    Put the optical center (Center Of Projection) at the origin

    Put the image plane (Projection Plane) in front of the COP

    The camera looks down the negative z axis

    Slide by Steve Seitz 32

    The Shading Model (what)The Shading Model (what)

    Shading Model: Given the illumination incident at a point on a surface, what is reflected?

    Generated by Foxit PDF Creator Foxit Softwarehttp://www.foxitsoftware.com For evaluation only.

  • 9

    33

    ShadingShading Model ParametersModel Parameters

    The factors determining the shading effects are:

    The light source properties:

    Positions, Electromagnetic Spectrum, Shape.

    The surface properties:

    Position, orientation, Reflectance properties.

    The eye (camera) properties:

    Position, orientation, Sensor spectrum sensitivities.

    34

    Newtons Experiment, 1665 Cambridge.Discovering the fundamental spectral components of light.

    Light and the Visible SpectrumLight