From Dusk till Dawn: Modeling in the...
1
From Dusk till Dawn: Modeling in the Dark Filip Radenović 1 Johannes L. Schönberger 2,3 Dinghuang Ji 2 Jan-Michael Frahm 2 Ondřej Chum 1 Jiří Matas 1 1 CMP, Faculty of EE, CTU in Prague 2 Dept. of Computer Science, The University of North Carolina at Chapel Hill 3 Dept. of Computer Science, ETH Zurich Introduction Problem • Structure-from-Motion from mixed sets of day and night images provides reliable sparse 3D models • Dense 3D reconstruction suffers significantly when done using images with significant illumination difference. Noticeable artifacts appear in this case • Night models are often smaller than day ones Contributions • Automatic separation of day and night images and their separate reconstruction Clean Day Dense Standard Dense Artifacts Night Dense Clean • Geometric fusion of day and night dense models into a single model • Learning the color transfer to recolor untextured model parts Day/Night Clustering Geometric Fusion & Recoloring Pipeline Image Database 7.4M images Sparse Model Retrieval Clustering Day Night Day/Night Clusters Fusion Dense Rec. Dense Model Fused Model Scene Graph Geometric Fusion • Structure stays the same even if the illumination changes significantly • Merge day and night point clouds into a single dense 3D model Recoloring • Repainting: Visible at night but not reconstructed • Inpainting: Visible at day but not in the night images • Blending: Blend colors for scene parts with lacking nighttime coverage Blending Repainted Inpainted + Blended Inpainting Night Image Results Day Image Day Model Night Model Fused Night Model Night Image Evaluation • 7.4M images in the dataset • 239,717 unique images registered • 1,474 models reconstructed • 845 disjoint landmarks • 9:1 day to night image ratio • 1 week on a single machine [1] J. L. Schönberger, F. Radenović, O. Chum, and J.-M. Frahm. From Single Image Query to Detailed 3D Reconstruction. In CVPR 2015. [2] J. L. Schönberger and J.-M. Frahm. Structure-from- motion revisited. In CVPR 2016. Min-cut on Bipartite Visibility Graph • Minimizing the energy over scene graph (labels: day or night) • Unary term on images: • SVM score on color histogram • Classifier trained on single (Colosseum) model • Pairwise term on image - 3D point edges: • Potts model: 0 for the same label, 1 otherwise day day night night Min-cut SfM Recoloring
Transcript of From Dusk till Dawn: Modeling in the...
From Dusk till Dawn: Modeling in the DarkFilip Radenović1 Johannes L. Schönberger2,3 Dinghuang Ji2
Jan-Michael Frahm2 Ondřej Chum1 Jiří Matas1
1CMP, Faculty of EE, CTU in Prague 2Dept. of Computer Science, The University of North Carolina at Chapel Hill 3Dept. of Computer Science, ETH Zurich
Introduction
Problem
• Structure-from-Motion from mixed sets of day and night images provides reliable
sparse 3D models
• Dense 3D reconstruction suffers significantly when done using images with
significant illumination difference. Noticeable artifacts appear in this case
• Night models are often smaller than day ones
Contributions
• Automatic separation of day and night images and their separate reconstruction
Clean
Day Dense
Standard Dense
Artifacts
Night Dense
Clean
• Geometric fusion of day and night dense models into a single model
• Learning the color transfer to recolor untextured model parts
Day/Night Clustering Geometric Fusion & Recoloring
Pipeline
Image Database7.4M images
Sparse Model
Retrieval Clustering Day
Night
Day/Night Clusters
Fusion
DenseRec.
Dense Model Fused ModelScene Graph
Geometric Fusion
• Structure stays the same even if the illumination changes significantly
• Merge day and night point clouds into a single dense 3D model
Recoloring
• Repainting: Visible at night but not reconstructed
• Inpainting: Visible at day but not in the night images
• Blending: Blend colors for scene parts with lacking nighttime coverage
Blending
Repainted Inpainted + Blended
Inpainting
Night Image
Results
Day Image Day Model Night Model Fused Night Model Night ImageEvaluation
• 7.4M images in the dataset
• 239,717 unique images registered
• 1,474 models reconstructed
• 845 disjoint landmarks
• 9:1 day to night image ratio
• 1 week on a single machine
[1] J. L. Schönberger, F. Radenović, O. Chum, and J.-M. Frahm. From Single Image Query to Detailed 3D Reconstruction. In CVPR 2015.
[2] J. L. Schönberger and J.-M. Frahm. Structure-from-motion revisited. In CVPR 2016.
Min-cut on Bipartite Visibility Graph
• Minimizing the energy over scene graph (labels: day or night)
• Unary term on images:
• SVM score on color histogram
• Classifier trained on single (Colosseum) model
• Pairwise term on image - 3D point edges:
• Potts model: 0 for the same label, 1 otherwise
day day night night
Min-cut
SfM Recoloring
