Low-Rank Based Algorithms for Rectification, Repetition Detection and De-Noising in Urban Images

A dissertation proposal by

Juan Liu

Committee

Committee:

Professor Ioannis Stamos, Mentor, Hunter College

Professor Yingli Tian, City College

Professor Zhigang Zhu, City College

Outside Member:

Professor Emmanouil Z. Psarakis, University of Patras, Greece

Current Work

2D image rectification

Façade texture selection

Efficient Kronecker Product model

Automatic repeated patterns detection

Reconstruction & photorealistic rendering of urban environments, symmetry detection, hole filling, etc.

Façade Image Rectification

Vanishing points (VPs) detection

TILT

Vanishing points Draw m hypothetical lines at angular intervals for each vanishing point

Output

Façade Texture Selection

Let C be the non-negative m×m matrix, with each element representing the number of Harris corners inside a block

Slide a window of size r×c along C

Compute the sample mean deviation of the sample median of matrix C, μC

Façade Texture Selection

Kronecker Product Model

A Toy Example

Input facade Repeated patterns

Repeated Patterns Detection via Kronecker Product Model

• Define the cost function:

• The minimization problem:

Solution to The Minimization Problem

Theorem 1: Let be the Singular Value Decomposition of the rearranged counterpart of matrix . Then partition matrices , patterns and weighting factors should satisfy the following:

Solution to The Minimization Problem

Lemma 1: Assuming that matrices are known, then the optimal and are related as follows:

Spatial Periods Estimation via Cross Correlation

• Consider that are known, then we can estimate the periods for facade partition

3×10 partition blocks Fi,j

Column-wise Cross-Correlation sequences: distance between the

adjacent peaks provides the period information.

Partition

Block Vectorization

• Vectorization of partition blocks

• The equivalent cost function:

Vectorization

rank = 30

Estimating K by Clustering

Reshape

rank = 4

Estimation of Mκ, κ =1, 2, …, K★

• Reshape the K★ indicators vectors from Algorithm 2

Computing Patterns and Weighting Factors

• Using Lemma 1 and Algorithm 2

Group 1

Group 2

Group 3

Group 4

Classification

Classification

Classification

Classification

Refinement

Refinement

Refinement

Refinement

1-0 patterns re-construction via Kronecker Product Model

Experiments and Evaluation

• Experiment 1: image rectification and texture selection

• Experiment 2: repeated patterns detection 89 façade images with ground-truth

Success rate 96%

Pixel-wise comparison

Success threshold: 91% match with ground-truth

Results

Proposed Work

Improve the estimation of K★

Extend the method to model nested patterns

Apply the model to 3D point clouds

Improve the estimation of K★

• Limitation: the accuracy and efficiency is drawn back by

– K-means clustering algorithm

Period Computation for Nested Patterns

• The current model may cut a bigger pattern into pieces due to that:– The most frequently appeared pattern dominates the final period

Apply the model to 3D point clouds

Timeline

• My work plan to complete the dissertation is arranged as follows:– Jan. 9 2015 - Jan. 20 2015:

• Finalize the implementation of new methods.

• Run experiments to test the new methods.

• Analyze the performance based upon experiment results.

– Jan. 21 2015 - Feb. 21 2015: Complete the rest part of my thesis and prepare for defense.

– Finally: Defend in March 2015.

Related Publications

• One paper “Automatic Kronecker product model based detection of repeated patterns in 2D urban images” that is related to this proposal has been published in IEEE International Conference on Computer Vision (ICCV) 2013 (accepatance rate 28%, 1600 submissions).

• Another related paper has been submitted to the IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) recently.