IIIT HYDERABAD

Image-based walkthroughs from partial and incremental scene

reconstructionsKumar Srijan

Syed Ahsan IshtiaqueC. V. Jawahar

Center for Visual Information Technology,

IIIT-Hyderabadhttp://cvit.iiit.ac.in

Sudipta N. Sinha

Microsoft Research, Redmond

http://research.microsoft.com

IIIT HYDERABAD

Introduction

IIIT HYDERABAD

Problem

• Efficiently organize and browse these huge image collections?

• Keep Incorporating an incoming stream of images into an existing framework?

IIIT HYDERABAD

Related Work

• World-Wide Media Exchange (WWMX)• PhotoCompas• Realityflythrough• Aspen Movie Map• Photowalker• Sea of Images• Google Streetview• Photo Tourism

IIIT HYDERABAD

Photo Tourism

Computing correspondences

Detect Features in each image

Match keypoints between each pair of images

For each pair, estimate an F-

matrix and refine matches

Chain pairwise matches into

tracks

Incremental SfM

Select a good initial pair to seed

reconstruction

Add new images and triangulate new

pointsBundle adjust

Snavely et. al, Photo Tourism: Exploring image collections in 3D

Input Images

Full Scene Reconstruction

IIIT HYDERABAD

Bottlenecks and Issues

• Global scene reconstruction via incremental structure from motion (Sfm)– Sensitivity to the choice of the initial pair– Cascading of errors– O(N4) in the worst case

Snavely et. al, Photo Tourism: Exploring image collections in 3D

IIIT HYDERABAD

Bottlenecks and Issues• Timing Breakdown

Snavely et. al, Photo Tourism: Exploring image collections in 3D

Full Scene Reconstruction for Trafalgar Square dataset with 8000 images took > 50 days

IIIT HYDERABAD

Our approach• “ In a walkthrough, users primarily observe near by

overlapping images.”

• Advantages:– Robustness to errors in incremental SfM module– Worst case linear running time– Scalable– Incremental

Independent Partial Scene Reconstructions instead of

Global Scene Reconstruction

IIIT HYDERABAD

Partial Reconstructions

Image Match

Compute Matches

Refine Matches

Compute partial Reconstructions

Standard SfM

Correct MatchIncorrect Match

IIIT HYDERABAD

Visualization Interface

User interface and navigation

Input images Verified neighbors

Sample image

Partial reconstruction

IIIT HYDERABAD

Global vs. Partial• Global : Allows transition to any image• Partial : Allows transition to a limited number

of overlapping images • A -> B implies B -> A

AB B

A

IIIT HYDERABAD

Incremental insertion

New Image

MatchGeometric Verification

Compute Partial Scene

Reconstruction

Improve Connectivity

IIIT HYDERABAD

Dataset

Fort Dataset

5989 images

Golconda Fort, Hyderabad

IIIT HYDERABAD

Results

IIIT HYDERABAD

Results

IIIT HYDERABAD

Results

• Courtyard Dataset with 687 images

• Initialized with 200 images

• Added 487 image one by one

• Largest CC of 674 images.

IIIT HYDERABAD

Conclusion

• Image navigation system based on partial reconstructions can effectively be used to navigate through large collections of images.

• Robustness to errors• Able incorporate more

images as they become available.

IIIT HYDERABAD

Future Work

• Complete automation– Download images

directly from the internet

– Add into the framework

IIIT HYDERABAD

Acknowledgements

• “Photo tourism: Exploring photo collections in 3D“– Noah Snavely, Cornell

University– Steven M. Seitz,

University of Washington

– Richard Szeliski, Microsoft Research

IIIT HYDERABAD

Acknowledgements

• “Visual Word based Location Recognition in 3D models using Distance Augmented Weighting”– Friedrich Fraundorfer,

Marc Pollefeys ETH Zürich– Changchang Wu ,Jan-

Michael Frahm ,Marc Pollefeys - UNC Chapel Hill

IIIT HYDERABAD

Thank You• Questions