MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and...

MESA LABMESA LABMESA LABMESA LAB

Multi-view image stitching Multi-view image stitching

Guimei ZhangMESA MESA (Mechatronics, Embedded Systems and Automation)LABLAB

School of Engineering,University of California, Merced

E: [email protected] Phone:209-658-4838Lab: CAS Eng 820 (T: 228-4398)

June 16, 2014. Monday 4:00-6:00 PMApplied Fractional Calculus Workshop Series @ MESA Lab @ UCMerced


Introduction

06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced

Slide-2/1024

Why to do some work on stitching?

1.Generate one panoramic image from a series of smaller, overlapping images.

2.The stitched image can also be of higher

resolutions than a panoramic image acquired by

a panoramic camera. The other is a panoramic

camera is more expensive.

MESA LABMESA LABMESA LABMESA LABIntroduction


Applications:

interactive panoramic viewing of

images ， architectural walk-through ， multi-

node movies and other applications associated

with modelling the 3D environment using images

acquired from the real world (digital surface

models/digital terrain models/ true orthophoto

and full 3D models)06/16/2014 AFC Workshop Series @ MESALAB @ UCMerced

Slide-4/1024

Introduction



Slide-5/1024

Introduction

true orthophoto and full 3D models


• What is multi-view?

capture images at different time, at different view points or using different sensor, such as camera, laser scanner, radar, multispectral camera…..


Slide-6/1024

Introduction


2. MethodThe flowchart of producing a panoramic image



Slide-8/1024

The main work is as following:

1. image acquired

2.How to perform image effective registration

3. How to fulfill image merging

Introduction


• Image acquisition1. Use one camera, at different time or different viewpoints, to

capture images, so there is rotation or translation transformation or both of them. (R,T)

2. Use several cameras located in different viewpoint to capture

images. (R,T)

3. Use different sensors, such as camera, laser scanner, radar, or

multispectral scanner ….(fuse multi sensor information)


Slide-9/1024

Introduction



Slide-10/1024

Introduction

Camera and tripod for acquisition by camera rotations Geometry of overlapping images

Need perform coordination transformation


• Since the orientation of each imaging plane is different in acquisition method. Therefore, images acquired need to be projected onto the same surface, such as the surface of a cylinder or a sphere, before image registration can be performed.

• that means we have to perform coordinate transformation.


Slide-11/1024

Introduction


• Image registration:

To form a larger image with a set of overlapping images, it

is necessary to find the transformations matrix(usually

rigid transformation, only parameters R and T) to align the

images. The process of image registration aims to find

the transformations matrix to align two or more

overlapping images. Because the projection( from the view

point through any position in the aligned images into the

3D world) is unique.


Slide-12/1024

Introduction



Slide-13/1024

5th frame image 6th frame image

Registrated image

Introduction


Multi-view point clouds registration and stitching based on SIFT feature


Slide-14/1024

1. Motivation

2. Method

3. Experiments

4. Conclusion

5. Discussion

SIFT: scale invariant feature transform


1. Motivation

Problems: (Existed method for multi-view

point clouds registration in large scene)

1. Be restricted by the view angle of camera,

single or two viewpoint image can only obtain

local information of local scene;



2. Existed methods need to add special markers

in large reconstruction scenes.

3. Existed method also need ICP(iterative

closest point ) iterating calculation, and can’t

eliminate interference of holes and invalid 3D

point clouds.


Slide-16/1024

1. Motivation


2. Method

• Our work based on Bendels[8] work, put forward a new algorithm of multi-view registration and stitching.

1. Generation 2D texture image of effective point

clouds;

2. we extract SIFT features and match them in

2D effective texture image;


2. Method

3. Then we map the extracted SIFT features and

matching relationship into the 3D point

clouds data and obtain features and matching

relationship of multi-view 3D point clouds;

4. Finally we achieve multi-view point clouds

stitching.


2.1 Generation texture image of effective point

clouds data

Why: 3D point clouds can’t avoid holes and noise, in order

to decrease the effect of registration and stitching precision

about multi-view point clouds, we use mutual mapping

method between 3D point clouds and 2D texture image to

obtain texture image of effective point clouds data.


Slide-19/1024

2. Method


2.1 Generation texture image of effective point

clouds data

•How: Firstly, we projected the 3D point clouds to 2D

plane, secondly, used projection binary graph to make 8

neighborhood seed filling and area filling, so that we can

obtain projection graph of effective point clouds data.


Slide-20/1024

2. Method


2. Mehtod2.1 Generation texture image of effective point clouds data

Texture image of effective point clouds data of a scene


2.2 Extraction and matching with SIFT feature

1.Extract SIFT feature

SIFT is a local feature which is proposed by David[7], we can

extract the SIFT feature which is invariant under translation,

scale and rotation. This paper used SIFT algorithm to extract 2D

features, then used RANSAC method[9]to eliminate error

matching.


Slide-22/1024

2. Mehtod


• 2. 3 3D feature point extractionPixel point of effective texture image,

which is obtained by point clouds

texture mapping has one-to-one

correspondence relationship

to 3D point clouds[10], as shown in

Fig.


Slide-23/1024

2. Mehtod

correspondence relationship


the method is as follows:

(1) We extract SIFT feature points in 2D texture image,

then calculated the coordinate of the point.

(2) Because 2D feature points and 3D point clouds

have one-to-one correspondence relationship, we can

calculate coordinate of corresponding feature points of 3D

point clouds.


Slide-24/1024

2. Method


• 2.4 3D point clouds stitching

The Multi-view 3D point clouds stitching is to make

coordinate transformation of point clouds in different

coordinate systems, the main problem is to estimate the

coordinate transformation R (rotation matrix) and

T(translation matrix).


Slide-25/1024

2. Mehtod


• According to the matching point pairs which are obtained through the above step, we can estimate coordinate transformation relationship, that is to

• estimate parameters R and T, which make the objective function get minimum:


Slide-26/1024

Where pi and qi are matching points pairs of 3D point clouds in two consecutive viewpoint.

2. Method

MESA LABMESA LABMESA LABMESA LAB3. Experiments

(a) SIFT feature effective texture image 1(b) effective texture image 2

(c) SIFT feature of 3D cloud points 1

(d) SIFT feature of 3D cloud points 2


3. Experiments

Our method

Ref [8] Ref [1]

Experimental results


3. Experiments

Performances evaluation criteria

Accuracy: registration rate and

stitching error rate

Efficiency: time consume


3. Experiments



Slide-31/1024



Slide-32/1024


4. Conclusion

1. Use SIFT feature of effective texture image to

achieve registration and stitching of dense multi-view

point clouds, we obtain texture image of effective

point clouds through mutual mapping between 3D

point clouds and 2D texture image, this algorithm

eliminate interference of holes and invalid point

clouds;


4. Conclusion

2. Ensure that effective 3D feature which

corresponding to every 2D feature can be found

in the 3D point clouds, it can eliminate the

unnecessary error matching, so matching

efficiency and matching precision have been

improved;


3. Our algorithm use the correct matching

point pairs to stitch, so it can avoid stepwise

iterative of ICP algorithm, and decrease

computational complexity of matching, it can

also reduce stitching error which is brought

by error matching.



Slide-36/1024 MESA LAB

Problems:• how to extract feature which is robust to

illumination and geometry transformation.

• how to eliminate registration error

• how to perform image merging to decrease the

difference between two adjacent stitched images

5. Discussion


Thanks

MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and...

Documents

Transcript of MESA LAB Multi-view image stitching Guimei Zhang MESA LAB MESA (Mechatronics, Embedded Systems and...