Unsolved Problems in Optical Flow and Stereo Estimation
Transcript of Unsolved Problems in Optical Flow and Stereo Estimation
![Page 1: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/1.jpg)
Unsolved Problems in Optical Flow
and Stereo Estimation
Richard Szeliski
Microsoft Research
and
Daniel Scharstein
Middlebury College
This work was supported in part by NSF grants IIS-0413169 and IIS-0917109
![Page 2: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/2.jpg)
Outline
Prior work: Middlebury benchmarks
Recent work: handling reflections
What are current challenges?
Future evaluation efforts?
![Page 3: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/3.jpg)
Collaborators - Benchmarks
Steve Seitz, U Washington
Brian Curless, U Washington
James Diebel, Stanford
Simon Baker, Microsoft Research
Michael Black, Brown U
JP Lewis, Weta Digital Ltd
Stefan Roth, TU Darmstadt
Heiko Hirschmüller, DLR Germany
Chris Pal, U Rochester
![Page 4: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/4.jpg)
Collaborators – Middlebury students
Anna Blasiak ’07
Padma Ugbabe ’03
Alexander Vandenberg-Rodes
Jiaxin (Lily) Fu ’03
Sarri Al-Nashashibi ’08
Gonzalo Alonso ’06
Jeff Wehrwein ’08 Brad Hiebert-Treuer
’07
Alan Lim ’09 Nera Nesic ’13
Xi Wang ’14
![Page 5: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/5.jpg)
Goal: Extract information from images (both 2D and 3D)
Hard problem:
Noisy data
Lots of it
Need additional assumptions
Computer Vision
![Page 6: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/6.jpg)
Our focus: image matching
Stereo vision
Multi-view stereo
Image motion / optical flow
![Page 7: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/7.jpg)
Applications - Stereo
Video conferencing
Game control
Intelligent cars
![Page 8: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/8.jpg)
Applications – Multiview stereo
3D reconstruction
3D printing
![Page 9: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/9.jpg)
Applications – Optical flow
Video interpolation and compression
Vehicle and people tracking
![Page 10: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/10.jpg)
Stereo vision
Infer 3D structure from 2 (or more)
images of a scene
Seems easy for humans…
![Page 11: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/11.jpg)
Why is matching hard?
Untextured areas
Noisy data / aliasing
Depth discontinuities
Occlusions
Reflections / specularities
Different camera responses
Imperfect calibration
…
![Page 12: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/12.jpg)
Datasets with ground truth
Ground truth = true answer
(e.g. true disparities)
GT needed for quantitative analysis
of algorithms (benchmarks)
Middlebury benchmarks:
http://vision.middlebury.edu/
![Page 13: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/13.jpg)
1. Middlebury Stereo Page
(Scharstein & Szeliski – CVPR 2001, IJCV 2002)
vision.middlebury.edu/stereo
Evaluator with web interface
![Page 14: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/14.jpg)
(Scharstein & Szeliski – CVPR 2001, IJCV 2002)
vision.middlebury.edu/stereo
Evaluator with web interface
v.1 by Lily Fu ’03
Left views
GT
disps
1. Middlebury Stereo Page
![Page 15: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/15.jpg)
(Scharstein & Szeliski – CVPR 2001, IJCV 2002)
vision.middlebury.edu/stereo
Evaluator with web interface
v.1 by Lily Fu ’03 v.2 by Anna Blasiak ’07
Left views
GT
disps
1. Middlebury Stereo Page
![Page 16: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/16.jpg)
Currently 135 entries
![Page 17: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/17.jpg)
2. Multiview Stereo Evaluation
(Seitz, Curless, Diebel, Scharstein, Szeliski – CVPR 2006)
vision.middlebury.edu/mview
Create 3D model from 100s of views
One view
GT
Surface mesh
![Page 18: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/18.jpg)
Currently 58 entries
![Page 19: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/19.jpg)
3. Optical Flow Evaluation
(Baker, Scharstein, Lewis, Roth, Black, Szeliski – ICCV 2007)
vision.middlebury.edu/flow
Input: video sequence
Output: flow vectors Where do pixels move from frame to frame?
![Page 20: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/20.jpg)
Currently 75 entries
![Page 21: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/21.jpg)
How to get ground truth?
1. Stereo – true disparities
2. Multiview stereo – true surface mesh
3. Optical flow – true motion vectors
![Page 22: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/22.jpg)
Setup 2005 / 2006
7 views
3 ambient light setups
3 exposures
2005: 9 datasets 2006: 21 datasets
see vision.middlebury.edu/stereo/data
![Page 23: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/23.jpg)
Version 3 – soon?
Current work: new datasets
Specular surfaces
Point-and-shoot cameras
Possibly outdoor scenes
“Space-time stereo” techniques
Stereo video?
![Page 24: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/24.jpg)
Unpublished datasets
Work in progress on specular scenes
Spray paint motorcycle after color photos are acquired to enable active lighting ranging
![Page 25: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/25.jpg)
Mobile acquisition system, 2012
DSLR Cameras
Point & Shoot Cameras
Projector Laptop for Processing
![Page 26: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/26.jpg)
Motorcycle Scene - Original
![Page 27: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/27.jpg)
Motorcycle Scene - Painting
![Page 28: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/28.jpg)
Motorcycle Scene - Painting
![Page 29: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/29.jpg)
Motorcycle Scene - Painted
![Page 30: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/30.jpg)
Motorcycle Disparity Map
![Page 31: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/31.jpg)
Motorcycle Scene - Original
![Page 32: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/32.jpg)
What can we do about specular scenes?
A1: treat reflections as separate layers
![Page 33: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/33.jpg)
Image-Based Rendering for Scenes with Reflections
Sudipta N. Sinha
Johannes Kopf
Michael Goesele
Daniel Scharstein
Richard Szeliski
![Page 34: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/34.jpg)
Use laser scanner
Merge 100s of scans
Fill holes
Align with image data
2. Multiview stereo: range data
![Page 35: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/35.jpg)
Version 2 – current work
![Page 36: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/36.jpg)
Version 2 – soon?
Have high-quality CT scans
Need better reference views
Need highly accurate camera locations
Include objects from industrial setting
Collaborate with NIST
![Page 37: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/37.jpg)
3. Optical flow: Hidden texture
Can’t use structured light (objects move)
Idea: make pixels “trackable” with
High resolution (downsample by 6)
Hidden fluorescent texture
Very slow motion
![Page 38: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/38.jpg)
Value of benchmarks
Enables quantitative comparison
Summarizes state of the art
Stimulates new research
Challenging data “pushes envelope”
![Page 39: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/39.jpg)
Pitfalls
Overfitting to test data
Focus on ranking
Deemphasizes aspects not evaluated
“Rest” after initial “push”
![Page 40: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/40.jpg)
Solutions
Provide separate training data
Provide diverse datasets
Avoid single ranking
Update benchmarks periodically
![Page 41: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/41.jpg)
Other uses of GT data
Algorithm design
Evaluate algorithm components
Robust data term
Smoothness priors
Machine learning
![Page 42: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/42.jpg)
Evaluation of Cost Functions for Stereo Matching (Hirschmüller & Scharstein, CVPR 2007, PAMI 2009)
![Page 43: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/43.jpg)
Learning Conditional Random Fields for Stereo (Scharstein & Pal, CVPR 2007; Pal et al. IJCV 2010)
Moebius – trained on other 5 Moebius – trained on self
![Page 44: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/44.jpg)
Why is matching hard?
Untextured areas
Noisy data / aliasing
Depth discontinuities
Occlusions
Reflections / specularities
Different camera responses
Imperfect calibration
… what about higher-level semantics?
![Page 45: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/45.jpg)
Semantic scene reconstruction
![Page 46: Unsolved Problems in Optical Flow and Stereo Estimation](https://reader031.fdocuments.net/reader031/viewer/2022022112/62120d5162ad9744a07c95dc/html5/thumbnails/46.jpg)
Conclusion
Benchmarks are important,
stimulate research
Creating ground-truth data is
challenging, fun
Rolling benchmarks
Code archival: source, binaries, and Web services (Web Vision Workshop)