Panoramic Imaging using SURF and SIFT of

OpenCV

Eric Jansen – 2210105030

Computer Engineering and Telematics

Department of Electrical Engineering

Institut Teknologi Sepuluh Nopember

November 5th, 2011

Abstract

Technology to provide good panoramic imagesor mosaic images are existed several years ago.And one of the hot topics being developed tilltoday. In this experiment, several images arecombined with finding their matching keypointsto provide image stitching. To solve this prob-lem, the concept of scale-invariant features hasbeen introduced in computer vision. The mainidea here is to have a scale factor associated witheach of the detected feature points. In recentyear, several scale-invariant features have beenproposed and this recipe presents two of them,the SURF and SIFT features. SURF stands forSpeeded Up Robust Features, and as we will see,they are not only scale-invariant features, butthey also offer the advantage of being computedvery efficiently. The SIFT or Scale Invariant Fea-ture Transform is developed from the SURF al-gorithm as an efficient variant of another well-known scale-invariant feature detector. SIFT alsodetects features as local maxima in image andscale space, but uses the Laplacian filter responseinstead of the Hessian determinant. This Lapla-cian at different scales is computed using differ-ence of Gaussian filters. OpenCV has a wrapperclass that detects these features and it is calledin a similar way to the SURF features.

1 Preface

Methods in determining the matching points be-tween two images are determined in calculatingthe color intensity of the images. Previous todetermining the intensity. The images are thenconverted into grayscale color, as to combinethree-scale of RGB colors into one scale (blackto white or from 0 to 255). After converting im-ages into grayscale-color, with Hessian method ofSURF, the images are normalized for better re-

sults. OpenCV provides all these features, eventhe matching key points between images that aregoing to be warped. with RANSAC or RandomSample Consensus, the key points are directlysearching the detected matches points at otherimages.

2 Methodology

The algorithms of both SURF and SIFT are de-scribed as the followings:

2.1 Detecting the scale-invariantSURF and SIFT features

The OpenCV implementation of SURF fea-tures also use the cv::SurfFeatureDetector orcv::SiftFeatureDetector of SIFT features in-terface.

std::vector<cv::KeyPoint> keypoints;

// Construct the SURF or SIFT features

// detector object

cv::Ptr<cv::FeatureDetector> feature =

new cv::SurfFeatureDetector(2500);

// or

new cv::SiftFeatureDetector;

// or the SIFT feature detector object

// Detect the SURF features or

// detect the SIFT features

feature->detect(image,keypoints);

Using cv::drawKeypoints OpenCV function toshow the scale factor associated with each fea-ture:

// Draw the keypoints with scale

cv::drawKeypoints(image,keypoints,

featureImage,

cv::Scalar(255,255,255),

cv::DrawMatchesFlags::

DRAW_RICH_KEYPOINTS);

1

figure 1. img03.jpg

figure 2. img04.jpg

figure 3. SIFT-method keypoints-img01.jpg

figure 4. SIFT-method keypoints-img02.jpg

2.2 Feature Matching

In both images, a SURF or SIFT feature hasbeen detected at that location and the two corre-sponding circles (of different sizes) contain in thesame visual elements. Based on the std::vectorof cv::KeyPoint instances obtained from featuredetection, the descriptors are obtained as follows:

cv::Ptr<cv::DescriptorExtractor>

m_pExtractor =

new cv::

SurfDescriptorExtractor;

// or

cv::Ptr<cv::DescriptorExtractor>

m_pExtractor =

new cv::

SiftDescriptorExtractor;

m_pExtractor->compute(image1,keypoints1,

desc1);

m_pExtractor->compute(image2,keypoints2,

desc2);

The result is a matrix which will contain as manyrows as the number of elements in the keypointsvector. Each of these rows is an N-dimensionaldescriptor vector. In the case of the SURF de-scriptor, by default, it has a size of 64. This vec-tor characterizes the intensity pattern surround-ing a feature point. The more similar the twofeature points, the closer their descriptor vectorsshould be.These descriptors are particularly useful in im-

age matching. Suppose, for example, that twoimages of the same scene are to be matched. Thiscan be accomplished by first detecting features oneach image, and then extracting the descriptorsof these features. Each feature descriptor vectorin the first image is then compared to all featuredescriptors in the second image. The pair thatobtains the best score is then kept as the bestmatch for that feature. This process is repeatedfor all features in the first image. This is themost basic scheme that has been implemented inOpenCV as the cv::BruteForceMatcher. It usused as follows:

cv::BruteForceMatcher< cv::L2<float> >

matcher;

From image 1 to image 2 and image 2to image 1 based on k-nearest neighbourswith k=2. This is accomplished by thecv::BruteForceMatcher::knnMatch method asshown as follows:

std::vector< std::vector<cv::DMatch> >

matches1;

matcher.knnMatch(desc1,desc2,matches1,2);

2

std::vector< std::vector<cv::DMatch> >

matches2;

matcher.knnMatch(desc2,desc1,matches2,2);

As shown in figure 3 and 4, the white lines link-ing the matched points show that even if thereare a large number of good matches, a significantnumber of false matches have survived. There-fore, the second test will be performed to filtermore false matches.The following method called symmetrical

matching scheme imposing that, for a match pairto be accepted, both points must be the bestmatching feature of the other:

for (std::vector< std::

vector<cv::DMatch> >::

const_iterator matIt1=

matches1.begin();

matIt1!=matches1.end();

++matIt1) {

if (matIt1->size() < 2)

continue;

for (std::vector< std::vector

<cv::DMatch> >::

const_iterator matIt2=

matches2.begin();

matIt2!=matches2.end();

++matIt2) {

if (matIt2->size() < 2)

continue;

if (*matIt1)[0].queryIdx ==

(*matIt2)[0].trainIdx &&

(*matIt2)[0].queryIdx ==

(*matIt1)[0].trainIdx) {

symMatches.push_back(

cv::DMatch(

(*matIt1)[0].queryIdx,

(*matIt1)[0].trainIdx,

(*matIt1)[0].distance));

break;

}

}

}

2.3 Outlier Rejection usingRANSAC

This test is based on the RANSAC method thatcan compute the fundamental matrix even whenoutliers are still present in the match set. TheRANSAC algorithm aims at estimating a givenmathematical entity from a data set that maycontain a number of outliers. The idea is torandomly select some data points from the setand perform the estimation only with these. The

number of selected points should be the minimumnumber of points required to estimate the math-ematical entity.

The central idea behind the RANSAC algo-rithm is that the larger the support set is, thehigher the probability that the computed matrixis the right one. Obviously, if one or more ofthe randomly selected matches is a wrong match,then the computed fundamental matrix will alsobe wrong, and its support set is expected to besmall. This process is repeated a number oftimes, and at the end, the matrix with the largestsupport will be retained as the most probable one.When using the cv::findFundamentalMat func-tion with RANSAC, two extra parameters areprovided. The first one is the confidence levelthat determines the number or iterations to bemade. The second one is the maximum distanceto the epipolar line for a point to be consid-ered as an inlier. All matched pairs in whicha point is at a distance from its epipolar linelarger than the one specified will be reported asan outlier. Therefore, the function also returns astd::vector of char value indicating that thecorresponding match has been identified as anoutlier (0) or as an inlier (1).

for (std::vector< std::

vector<cv::DMatch> >::

const_iterator it=

matches.begin();

it!=matches.end(); ++it) {

float x = keypoints1[it->

queryIdx].pt.x;

float y = keypoints2[it->

queryIdx].pt.y;

points1.push_back(cv::

Point2f(x,y));

x = keypoints2[it->

trainIdx].pt.x;

y = keypoints2[it->

trainIdx].pt.y;

points2.push_back(cv::

Point2f(x,y));

}

std::vector<uchar> inlier(

points1.size(),0);

fundamental = cv::

findFundamentalMat(

cv::Mat(points1),

cv::Mat(points2),

inliers,

CV_FM_RANSAC,

m_lfDistance,

m_lfConfidence);

std::vector<uchar>::

const_iterator itIn=

3

inliers.begin();

std::vector<cv::DMatch>::

const_iterator itM=

matches.begin();

for (; itIn!=inliers.end();

++itIn,++itM) {

if (*itIn)

outMatches.

push_back(*itM);

}

The more good matches found in initial matchset, the higher the probability that RANSAC willgive the correct fundamental matrix. This is whyapplying several filters to the match set beforecalling the cv::findFundamentalMat function.

2.4 Recovery Homography fromCorrespondences

After computing the fundamental matrix of animage pair from a set of matches. Another math-ematical entity exists that can be computed frommatch pairs: a homography. Like the fundamen-tal matrix, the homography is a 3×3 matrix withspecial properties and, it applies to two-view im-ages in specific situations. The projective rela-tion between a 3D point and its image on a cam-era is expressed by a 3×4 matrix. The specialcase where two views of a scene are separatedby a pure rotation, then it can be observed thatthe fourth column of the extrinsic matrix will bemad of all null. As a result, the projective rela-tion in this special case becomes a 3×3 matrix.This matrix is called a homography and it impliesthat, under special circumstances, e.g. rotation,the image of a point in one view is related to theimage of the same point in another by a linearrelation: sx′

sy′

s

= H

xy1

In homogenous coordinates, this relation holds upto a scale factor represented here by the scalarvalue s. Once this matrix is estimated, all pointsin one view can be transferred to the second viewusing this relation. Note that as a side effect ofthe homography relation for pure rotation, thefundamental matrix becomes undefined in thiscase.

Suppose two images separated by a pure ro-tation. These two images can be matched usingRobustMatcher class, and applying a RANSACstep which will involve the estimation of a ho-mography based on a match set that contains agood number of outliers. This is done by usingthe cv::findHomography function:

std::vector<uchar> inliers(

points1.size(),0);

cv::Mat homography =

cv::findHomography(cv::

Mat(points1),cv::

Mat(points2),inliers,

CV_RANSAC,1);

The resulting inliers that comply with the foundhomography have been drawn on those images bythe following loop:

std::vector<cv::Point2f>::

const_iterator itPts=

points1.begin();

std::vector<uchar>::

const_iterator itIn=

inliers.begin();

while (itPts != points1.end()) {

if (*itIn)

cv::circle(image1,*itPts,

3,

cv::Scalar(

255,

255,

255),2);

++itPts;

++itIn;

}

itPts = points2.begin();

itIn = inliers.begin();

while (itPts != points2.end()) {

if (*itIn)

cv::circle(image2,

*itPts,

3,

cv::Scalar(

255,

255,

255),2);

++itPts;

++itIn;

}

2.5 Image Compositing

Once the homography is computed, it will trans-fer image points from one image to the other.In fact, for all pixels of an image and the resultwill be to transform this image to the other view.There is an OpenCV function that does exactlythis:

cv::warpPerspective(image1,result,

homography,cv::Size(2 *

image1.cols,image1.rows));

cv::Mat half(result,cv::Rect(

0,0,image2.cols,image2.rows));

image2.copyTo(half);

4

figure 5. SIFT-method matching Key Pointsbetween two images

3 Conclusion

With several experiments in merging these im-ages, the SIFT algorithm compatibilities arehigher than the SURF. Although in fact, thepanoramic image produced of the SURF algo-rithm is the fastest. The SURF’s Hessian methodfor matching key points were set to 1000 could beapplied to combine several images but with lessaccurate matching points.

figure 6. Panoramic Image

5