Internet Vision - Lecture 3

Tamara BergSept 10

New Lecture Time

Mondays 10:00am-12:30pm in 2311

Monday (9/15) we will have a general Computer Vision & Machine Learning review

Please look at papers and decide which one you want to present by Monday – read topic/titles/abstracts to get an idea of which

you are interested in

Thanks to Lalonde et al for providing slides!

Algorithm Outline

Inserting objects into images

Have an image and want to add realistic looking objects to that image

Inserting objects into images

User picks a location where they want to insert an object

Inserting objects into images

Based some properties calculated about the image, possible objects are presented.

Inserting objects into images

User selects which object to insert and the object is placed in the scene at the correct scale for the location

Inserting objects into images – Possible approaches

Insert a clip art object Insert a clip art object with some idea of the environment

Insert a rendered object with full model of the environment

Some objects will be easy to insert because they already “fit” into the scene

Collect a large database of objects.Let the computer decide which examples are easy to insert.Allow the user to select only among those.

When will an object “fit”?

1.) When the lighting conditions of the scene and object are similar2.) When the camera pose of the scene & object match

2D vs 3D

Use 3d information for:

1.) Annotating objects in the clip-art library with camera pose2.) Estimating the camera pose in the query image3.) Computing illumination context in both library & query images

Phase 1 - Database Annotation

For each object we want:– Estimate of its true size and the camera pose it

was captured under– Estimate of the lighting conditions it was captured

under

Phase 1 - Database AnnotationEstimate object size

Objects closer to the camera appear larger than objects further from the camera

Phase 1 - Database AnnotationEstimate object size

*If* you know the camera pose then you can estimate the real height of an object from:location in the image,pixel height

Phase 1 - Database AnnotationEstimate object size

Annotate objects with their true heights and resize examples to a common reference size

Phase 1 - Database AnnotationEstimate object size & camera pose

Don’t know camera pose or object heights! Trick - Infer camera pose & object heights across all object classes in the database given only the height distribution for one class

Phase 1 - Database AnnotationEstimate object size & camera pose

Start with known heights for people

Phase 1 - Database AnnotationEstimate object size & camera pose

Estimate camera pose for images with multiple people

Phase 1 - Database AnnotationEstimate object size & camera pose

Use these images to estimate a prior over the distribution of poses

How do people usually take pictures? Standing on the ground at eye level.

Phase 1 - Database AnnotationEstimate object size & camera pose

Use the learned pose distribution to estimate heights of other object categories that appear with people.

Iteratively use these categories to learn more categories.

Annotate all objects in the database with their true size and originating camera pose.

Phase 1 - Database AnnotationEstimate object size & camera pose

Phase 1 - Database Annotation

For each object we want:– Estimate of its true size and the camera pose it

was captured under– Estimate of the lighting conditions it was captured

under

Phase 1 - Database AnnotationEstimate lighting conditions

Estimate which pixels are ground, sky, vertical

Black box for now (we’ll cover this paper later in the course)

Ground

Vertical

Sky

Phase 1 - Database AnnotationEstimate lighting conditions

Distribution of pixel colors

Phase 2 – Object Insertion

Query Image

Phase 2 – Object Insertion

User specifies horizon line – use to calculate camera pose with respect to ground plane (lower -> tilted down, higher -> tilted up).

Illumination context is calculated in the same way as for the database images.

Phase 2 – Object Insertion

Insert an object into the scene that has matching lighting, and camera pose to the query image

Phase 2 – Object Insertion

But wait it still looks funny!

Phase 2 – Object Insertion

Shadows are important!

Phase 2 – Object Insertion

Phase 2 – Object Insertion

Phase 2 – Object Insertion

Phase 2 – Object Insertion

Shadow Transfer

Categorize images for easy selection in user interface

Big Picture

• It’s all about the data!

• Use lots of data to turn a hard problem into an easier one!– Place “my car” in a scene is much harder than

place “some car” in a scene. Allow the computer to choose from among many examples of a class to find the easy ones.