RESEARCH SUMMARY

YALIN BASTANLAR

My research activities mainly cover the topics of computer vision especially when the omnidirectional cameras are employed. My Ph.D. studies condensed on structure-from-motion and 3D reconstruction. For the research projects I have been involved, I conducted research on corner detection and multi-view image-based 3D reconstruction with perspective camera images. I also conducted a few studies on the area of Human-Computer Interaction. One was on investigating the effects of color-multiplex stereoscopic view, another one was on the evaluation of the usability of web-based virtual tour applications. In the following, the research activities which I have been involved are briefly described starting from the recent ones.

• Feature point matching between omnidirectional and perspective camera images

This study is on robustly matching feature points between omnidirectional and perspective camera images. We use scale invariant feature transform (SIFT) as the matching method and propose an algorithm to improve the matching output in terms of number of correct and false matches. The proposed method includes preprocessing the perspective image before matching and it is experimentally shown that omnidirectional-perspective matching performance significantly increases (Fig.1). In addition, we evaluated the use of virtual camera plane (VCP) images, which are perspective images produced by unwarping a certain region of the omnidirectional image. We observe that VCP-perspective matching is more robust to increasing baseline length when compared to direct omnidirectional-perspective matching. We conclude that automatic feature point matching is possible with the described technique [1].

Fig. 1: Matching result of the proposed method for an example mixed image pair. Direct SIFT matching resulted in 25/60 false/total match ratio (at the top). The proposed method resulted in 4/60 false/total match ratio (at the bottom). Red dashed lines indicate false matches, whereas green lines indicate correct ones.

• Structure-from-motion for mixed camera systems

We describe a pipeline for structure-from-motion (SfM) with mixed camera types, namely catadioptric omnidirectional and perspective cameras. We proposed improved methods for several steps to make the pipeline more robust and automatic. We start by using the point correspondences in perspective and omnidirectional image which are matched with an improved SIFT matching algorithm. We robustly estimate the hybrid epipolar geometry using random sample consensus (RANSAC) and evaluate the alternatives of pose estimation methods. Finally, we propose a weighting strategy for iterative linear triangulation which improves the structure estimation accuracy (Fig.2). We presented the results of experiments on simulated and real images for the proposed approaches [2].

Fig. 2: SfM with mixed camera types. Top-left: Depiction of hybrid epipolar geometry. Bottom-left: Corresponding epipolar lines/conics on images. Right: Cameras and reconstructed 3D points.

• Calibration of central catadioptric cameras

We proposed a calibration technique that is valid for all single-viewpoint catadioptric cameras that can be modeled by the sphere camera model [3]. We are able to represent the projection of 3D points on a catadioptric image linearly with a 6x10 projection matrix, which uses lifted

coordinates for image and 3D points. This projection matrix can be computed with enough number of 3D-2D correspondences (minimum 20 points distributed in three different planes). We show how to decompose it to obtain intrinsic and extrinsic parameters. Moreover, we use this parameter estimation followed by a non-linear optimization to calibrate various types of cameras (Fig.3). We tested our method both with simulations and real images. When compared to previous methods, this algorithm brings the advantage of linear and automatic parameter initialization.

side view top view

Fig. 3: (a) Constructed 3D pattern. Reprojections with the estimated parameters after initial step (a) and after non-linear optimization step (b).

• Corner detection

We developed a method to validate and filter a large set of previously obtained corner points [4]. We derived the necessary relationships between image derivatives and estimates of corner angle, orientation and contrast. Commonly used cornerness measures of the auto-correlation matrix estimates of image derivatives are expressed in terms of these estimated corner properties. A candidate corner is validated if the cornerness score directly obtained from the image is sufficiently close to the cornerness score for an ideal corner with the estimated orientation, angle and contrast. We tested this algorithm on both real and synthetic images and observed that this procedure significantly improves the corner detection rates based on human evaluations. We tested the accuracy of our corner property estimates under various noise conditions. Extracted corner properties can also be used for tasks like feature point matching, object recognition and pose estimation.

Fig. 4: Examples of the corners detected with the proposed method. For comparison with other corner detection methods, please examine [4].

• 3D reconstruction for cultural heritage

This research was conducted for a Turkish-Greek (TUBITAK-GSRT) joint research project titled "3D Reconstruction and Web-based Virtual Tour for Cultural Heritage of Aegean Region". The aim of this study is to build a Web-based virtual tour system, focused at the presentation of archaeological sites. The proposed approach is comprised of techniques such as multi-view 3D reconstruction, omnidirectional viewing based on panoramic images, and their integration with GIS technologies [5].

In the proposed image-based 3D reconstruction method, the scene is captured from multiple viewpoints utilizing off-the-shelf equipment and its 3D structure is extracted from the acquired images based on stereoscopic techniques. Texture is also extracted from images and added to the generated 3D model of the scene and the result is converted to a common 3D scene modeling format.

The 3D models and interactive virtual tour tools such as 360° viewing are integrated with a GIS platform, in our study this is Google EarthTM, in which the excavation site plans can be added as detailed raster overlays.

Fig. 5: Example 3D reconstruction and integration with the GIS platform. On the left, overall view of the site together with the reconstructed wall in Google Earth™. In the middle, 3D model of the reconstructed section. The image on the right is a real photograph taken from archaeological site.

• Panoramic image based virtual tours for museums

Web-based virtual tour applications constructed by 360º panoramic images are used extensively all over the world. As the capabilities provided to the user increase, the tour becomes more realistic and functional. I developed a virtual tour application using Java Applet technology. A snapshot from a museum virtual tour page is shown in Fig. 6. At top-right, the viewing window exists under which the pull-down menu and written info area are located. At left, a short info of museum, interactive floor plan and audio info button exist from top to bottom. Interactive floor plan is integrated with viewing window. In other words, section of the museum that is currently presented in the viewing window, field of view (FOV) and direction of view are indicated in the floor plan. It is updated accordingly as the user changes these controls. This tour was developed as part of an e-Government project: ‘Access and Viewing System for Cultural Heritage Inventory’.

Fig. 6: A screenshot from the panoramic image based virtual tour of a museum.

• HCI research for virtual tours

I conducted two human-computer interaction research activities concerning the evaluation of the effectiveness and usability of virtual tours. First one is to discover the user behavior characteristics in a web-based virtual tour application, an example of which was given in the previous section (Fig. 6). There exist several options for the user to navigate in the museum (interactive floor plan, links in the images and pull-down menu). Written and audio information about the sections visited, detailed information for some artworks and several control functions are provided at the webpage. We designed a usability test to answer which option of navigation is preferred, at what rate written info area, audio info option and extra artwork info are used and which way of control (mouse, keyboard, panel buttons) is preferred. Results showed that the floor plan is the most preferred way for changing the location and pull-down menu is the least preferred. Another finding is that the mouse is the most preferred way for control functions [6].

In another study, the effects of stereoscopic view on object recognition and navigation performance of the participants were examined in an indoor Desktop Virtual Reality Environment, which is a two-floor virtual museum having different floor plans and 3D object models inside. This environment was used in two different experimental settings: 1) color-multiplex stereoscopic 3D viewing was provided by colored eye-wear, 2) regular 2D viewing. After the experiment, participants filled a questionnaire that inquires their feeling of presence, their tendency to be immersed and their performance on object recognition and navigation in the environment [7].

References:

[1] Bastanlar, Y. (2009), “Structure-from-Motion for Systems with Perspective and Omnidirectional Cameras”, Ph.D. Thesis, Middle East Technical University, Ankara, Turkey.

[2] Bastanlar, Y., Yardimci, Y., Temizel, A., Sturm, P. “Effective Structure-from-Motion for Mixed Camera Systems”, Proc. of International Conference on Pattern Recognition (ICPR 2010), 23-26 August, Istanbul, Turkey.

[3] Bastanlar, Y., Puig, L., Sturm, P., Guerrero Campo, J., Barreto, J. (2008), “DLT-like Calibration of Central Catadioptric Cameras”, Proc. of Workshop on Omnidirectional Vision (OmniVis 2008), 12-18 October, Marseille, France.

[4] Bastanlar, Y., Yardimci, Y. (2008), “Corner Validation based on Extracted Corner Properties”, Computer Vision and Image Understanding (CVIU), vol.112, p.243-261.

[5] Bastanlar, Y., Grammalidis, N., Zabulis, X., Yilmaz, E., Yardimci, Y., Triantafyllidis, G. (2008), “3D Reconstruction for a Cultural Heritage Virtual Tour System”, XXI Congress of the International Society for Photogrammetry and Remote Sensing (ISPRS 2008), 3-11 July, Beijing, China.

[6] Bastanlar, Y. (2007), “User Behaviour in Web-Based Interactive Virtual Tours”, 29th International Conference on Information Technology Interfaces (ITI 2007), 25-28 June, Dubrovnik, Croatia.

[7] Bastanlar, Y., Cantürk, D., Üke, H. (2007), “Effects of Color-multiplex Stereoscopic View on Memory and Navigation”, 3DTV-Conference: The True Vision: Capture, Transmission and Display of 3D Video, 7-9 May, Cos, Greece.