A NOVEL PANORAMIC STEREO HYPERSPECTRAL IMAGING SYSTEM

Air Can Karaca, Alp Erttirk, M. Kemal Gtillti, Sarp Erttirk

Kocaeli University Laboratory of Image and Signal Processing (KULIS), Kocaeli University, Turkey

{alican.karacal, alp.erturk, kemalg, sertur}@kocaeli.edu.tr

ABSTRACT

This paper proposes a novel panoramic stereo hyperspectral imaging system with theoretical analysis and experiments. The system has two line scan hyperspectral cameras and is set above a rotary stage. This provides the panoramic property of the system, and stereo hyperspectral panoramas are created at each 360 degree rotation. Furthermore, a novel algorithm is proposed, in which multi-band Census transform is implemented to estimate the disparity map information from the acquired stereo hyperspectral images. The system is utilizable for change detection, target detection and classification applications because it provides both disparity map and spectral property for any pixel or area in the image.

Index Terms—Hyperspectral imaging, multi-band Census transform, panoramic, stereo.

1. INTRODUCTION

Hyperspectral imaging systems can acquire data from hundreds of narrow spectral bands in visible and infrared regions, resulting in continuous electromagnetic spectrum information for each pixel. The spectral information of each pixel is dependent on the chemical and physical properties of the underlying material, so it can provide significant features [1]. For this reason, hyperspectral imaging is used not only in remote sensing but also in many applications such as quality analysis in food industry, document analysis in forensics, recycling and so on.

Stereo imaging systems provide 3D information, such as orientations and distances of the objects in the scene, from two or more cameras. Well-designed stereoscopic displays convey a very compelling sense of 3D depth but there are also well-researched algorithms for automatic depth map extraction. Performances of such algorithms evaluated over Tsukuba, Venus, Teddy and Cones datasets in Middlebury website [2]. One of the applications of stereo in multispectral images is 3D building extraction in IKONOS stereo system for remote sensing [3]. Another application is a matrix sensor-based stereo system given in [4], which uses LTCF for capturing multispectral images.

Panoramic imaging, which provides wide angle of view in scanned scenes, has become more popular recently. The images can be created by stitching together a sequence of overlapping normal images, and image stitching is an ongoing research topic. Another method to create a panoramic image is using slit cameras which are also called line-scan cameras, and have only one column-wide sensor. With this method, images can be produced by rotating the line camera while capturing images, as given in [5].

Stereo panorama using omnidirectional vision system is explained in [6]. There are several studies about panoramic line scan stereo cameras. Common idea of these studies is that cameras are mounted on a rotary stage. For example, panoramic 3D reconstruction using a rotating camera and planar mirrors is given in [7]. In [8], stereo line scan panorama construction and theory is explained in detail.

In [9], sensitivity of different stereo matching costs are evaluated for stereo imaging methods, with respect to radiometric variations of the input images, and Census transform is proposed, because of its robustness. Another advantage of Census transform is the low complexity of the algorithm so that it can be easily implemented in real-time embedded systems [10]. In [11]-[12], Census transform is calculated over intensity values of RGB images, as a cost function. Due to Census transform is not applied to hyperspectral images and multi-band image stereo literature, a block matching algorithm which uses multiple spectral image band based Census transform is proposed as a novel method.

In this paper, rotating panoramic stereo hyperspectral system, which has two hyperspectral cameras on variable speed rotating stage, is introduced. Wavelength range of the system is 400-1000 nm with 2.8 nm spectral resolution. The system can collect a full 360 degree panoramic hyperspectral image and extract the disparity map using the proposed approach using multi-band Census transform.

2. PANORAMIC STEREO HYPERSPECTRAL IMAGING SYSTEM

2.1. System Configuration

The panoramic stereo hyperspectral system consists of four major parts given in Fig. 1, namely rotating stage, imaging

978-l-4799-2890-3/14/$31.00 ©2014 IEEE 145 ISCCSP2014

system, notebook and for mobile usage of the system, uninterruptible power supply (UPS). Imaging system consists of two hyperspectral cameras, and is mounted on a rotating state. Angular velocity of the rotating stage can be set between 1 and 30 rpm.

Fig. 1. Major parts of the system.

Each hyperspectral camera has three assembled parts, namely camera lens, visible-near infrared (VNIR) imaging spectrometer and monochromatic CCD camera. Imaging spectrometer provides narrow spectral bands with 2.8 nm spectral resolution and CCD camera has high quantum efficiency distribution in 400-1000 nm wavelengths. Pentax 25 mm F/1.8 lenses are used. Despite that the cameras support 1392 x 1040 resolution in full resolution, 400 x 300 resolution is used in this work to increase frame per second (FPS) rates. In the final configuration, wavelength range is decreased to 500-875 nm and the spectral bands to 300 so that the camera can capture with 40.4 FPS. Note that by using cameras with a higher FPS, 400-1000 nm wavelength range can be fully utilized.

After spatial alignments and spectral calibrations of both hyperspectral cameras are done, the line scan cameras capture images while the stage is rotating. Rotating speed must be stable and not reconfigured during image capturing. For each triggering, left and right image data are taken via USB 2.0 protocol and stored to RAM. After the scanning finishes, software constructs panoramic hyperspectral image cube for each camera. Before extraction of disparity map, offset of cameras must be corrected. Once geometric correction procedure is done, offset parameter can be used until the position of any cameras is changed. Details of geometric correction and disparity map extraction algorithm are given in Section 3.

2.2. Stereo Line Scan Theory

Even though CCD cameras with matrix sensors are used in this work, they behave as line sensor cameras because of the imaging spectrometers assembled in front of them. The geometry of epipolar lines is defined for matrix sensor cameras. However, for line-scan cameras epipolar curves are obtained for the line scan cameras, as defined in [13].

There are some important parameters in rotating line scan theory. As given in Fig. 2, distance between a camera center C and rotation center O is called off-axis distance (R) and the other parameter is sweep angle (co), which is 90 degrees in this work. In our system configuration, focal lengths of cameras (/), off-axis (R) are pairwise identical and

sweep angles (co) are symmetric. Hence, epipolar curves are converted to straight lines. Additionally, W parameter is the captured number of lines in a full 360 degree rotation.

Rotation axis Panoramic image

Fig. 2. Parameters of symmetric configuration [14].

Distribution of spatial samples in stereo panoramic pair as viewed from the top is given in Fig. 3. The analysis of spatial samples is done by projection lines as shown in [8]. Drawing is plotted at constants and W. Projection lines for left and right cameras are superimposed in Fig. 3.a, and depth layer circles are shown in Fig. 3.b. It can be seen that distances between sequential depth layers are increasing exponentially in further distances.

In stereo line scan theory, W directly increases the number of samples and R has no effects over number of samples. But with changing R, distance between depth layers increases linearly.

(a) Samples with lines (b) Sample distribution Fig. 3. Spectral calibration of imaging system.

Theoretical analysis of depth distances is performed with the parameter values of the system which are used in the experiments [8]: W = 2020, R = 5 cm, co = 90 °, / = 25 mm. Depth distances between rotation center and A* depth layer is given in Fig. 4. It can be seen that maximum provided depth distance by the system is equal to 32 meters and the system is more sensitive against depth differences in closer distances compared to further distances.

5 10 15 20 25 30 Depth Layer (k)

Fig. 4 Depth distances in different depth layers.

146

3. DISPARITY MAP ESTIMATION ALGORITHM Proposed algorithm can be classified into five parts, namely offset correction, band reduction, multiple band Census coding, cost calculation and occlusion filling. The processing steps after capturing images are given in details as follows.

1) Although the hyperspectral camera pair is mounted carefully, point of view difference between both cameras will occur. Hence, offset of cameras must be corrected in vertical axis to match epipolar lines. For this purpose, 300 spectral band images for each camera are computed and Phase Correlation algorithm [15] is applied on these average band images. Calculated offset value can be used for multiple operations of the system, unless the positions of the cameras are changed.

2) Number of spectral bands is reduced to N by averaging image bands, for noise reduction. N defines how many bands will be used in multi-band Census coding.

3) Both image pairs (left and right averaged) are coded over Wc x Wc windows for N bands using Census Transform, as given in (1) and (2). Then, each pixel will be presented by a binary code which has Wc

2 x N size. JV (wc-\yi (Wc-\)ii

hc(x,y):=® <8> <8> £(I(x,y,k),I(x+i,y+j,k)) k=l]=-{Wc-\)l2i=-iWc-\)l2

(1) [ 0, if n < m ]

Z(n,m):=< \ (2) [ 1, if n > m J

4) Sum of Hamming distances for different disparities from dmin to dmax are calculated using the window size of Wm x Wm for each pixel. To increase robustness in matching, searching of reference image for a pixel with position («, v) is done not only in v column but also in columns v-1 and v +1. Minimum cost disparity is assigned to disparity map for each pixel. This process is implemented from both left to right and right to left scanning for matching.

5) Median filter is used on each disparity map with a window size of 3 x 3. Small depth areas are assigned as holes and left-right-consistency is checked using disparity maps to define inconsistent areas.

6) Both hole and occlusion areas are filled by defining the borders of hole and occlusion parts, then calculating histograms of a Wflu x Wfiu dimension window around each pixel in borders. Disparity value with the maximum histogram is assigned to the final disparity map for the corresponding hole/occlusion. Then this process continues iteratively until total number of borders is equal to zero.

4. EXPERIMENTAL RESULTS

In the experiment, R is selected as 5 cm. Scanning a 360 degree panorama took 50 seconds and with fixed 40.4 FPS, W is defined as 2020. Camera exposure time is set to 20 ms. A scene consisting of two iron bars, plants and a human

were in the scene different distances is created. The scene is scanned with the system for an angle of 150 degrees and 372 x 881 x 300 dimensional hyperspectral image cube pair is constructed. Color image of the scene is given in Fig. 5.a.

Minimum and maximum disparity values for the scene are set to 3 and 32. Wm is set to 11, Wc is set to 5, number spectral bands which is used for Census transform, N is set to 50 bands and Wfin is set to 11. Parameter values are determined experimentally.

If Wm is increased, disparity maps are smoother although details may be invisible. If Wm is decreased, disparity maps have more detail, but noise may be increased. Occlusion filling parameter Wflu does not change disparity values directly. N and Wc are the most critical parameters in the algorithm and higher values for each parameter will increase accuracies in disparity map, as well as computation time. The disparity maps of the proposed algorithm when N=\ (standard Census) and 7V=50 (multiple-band Census) are given in Fig. 5.d and Fig 5.e. Other results which are given in Fig. 5.b. and Fig 5.c are obtained when Euclidian distance and spectral angle distance is used as cost function. In all disparity maps, mismatched regions (occluded regions and holes) are given in black color. It can be seen that proposed algorithm (N=50) gives more robust and consistent matching results than standard Census, Euclidian and spectral angle cost. Additionally, proposed filling procedure result is given in Fig. 5.f

5. CONCLUSION

In this paper, a portable panoramic stereo hyperspectral imaging system is introduced in detail. Panoramic stereo hyperspectral data is captured by the system, and disparity maps are successfully estimated using the proposed method. Novel multi-band Census algorithm, which uses spatial neighboring properties in different spectral bands, provides these advantages: a) robustness to radiometric variations so that no radiometric correction is required for the hyperspectral cameras, b) low complexity, and suitability to be parallelized in embedded systems for multiple spectral band images, due to its bitwise operations . The system can be used for any application such as change detection, target detection and target tracking. The system provides spectrums of any point, as well as providing depth information with the proposed multi-band Census transform algorithm. In the future version of the system, faster cameras and placing the cameras in a larger radius will be investigated to increase depth range. Additionally, slip ring for power and communication cables will be implemented to enable the use of the system for target tracking applications with continuous scanning of the scene and real-time working algorithms.

6. ACKNOWLEDGEMENT

This work was partly supported by Turkish State Planning Agency Project under project no. DPT2011K120330.

147

(a) Color image of the scene (b) Disparity map using Euclidian distance (JV=50)

(e) Disparity map of proposed algorithm (JV=50) (f) Disparity map of proposed algorithm (filled) Fig 5. Color image of the scene and disparity maps of algorithms.

7. REFERENCES

[1] D. A. Landgrebe, Signal Theory Methods in Multispectral [10] Remote Sensing, Hoboken, NJ: Wiley, 2003.

[2] Middlebury Stereo Evaluation Website, [Online]. Available: http://vision.middlebury.edu/stereo/eval

[3] H. G. Sohn, C. H. Park, H. S. Kim, and J. Heo, "3-D building extraction using IKONOS multispectral images," in Proc. [11] International Geoscience and Remote Sensing Symposium (IGARSS '05). IEEE, 2005, vol.2, pp.1432-1434.

[4] R. Shrestha, A. Mansouri, and J. Y. Hardeberg. "Multispectral imaging using a stereo camera: Concept, [12] design and assessment," EURASIP Journal on Advances in Signal Processing 2011, 2011, no.l, pp. 1-15.

[5] D. Schneider, and M. Hans-Gerd, "A geometric model for linear-array-based terrestrial panoramic cameras", The Photogrammetric Record 21, 2006, no. 115, pp. 198-210. [13]

[6] R. Bunschoten, B. Krose, "Robust scene reconstruction from an omnidirectional vision system," in IEEE Transactions on Robotics and Automation, vol. 19, no.2, pp.351-357, 2003.

[7] W. Jiang, S. Sugimoto, andM. Okutomi, "Panoramic 3D [14] reconstruction using rotating camera with planar mirrors," in Proc. ofOmnivis '05. 2005, vol. 1, pp. 123 -130.

[8] F. Huang, R. Klette, and K. Scheibe, Panoramic Imaging: Sensor-Line Cameras and Laser Range-Finders, Chichester: [15] John Willey and Sons Ltd, 2008.

[9] H. Hirschmuller and D. Scharstein, "Evaluation of Stereo Matching Costs on Images with Radiometric Differences," in

IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.31, no.9, pp.1582-1599, Sept. 2009. J. Seunghun, C. Junguk, D. P. Xuan, L. Kyoung-Mu, P. Sung-Kee, K. Munsang, and J. W. Jeon, "FPGA Design and Implementation of a Real-Time Stereo Vision System," in IEEE Transactions on Circuits and Systems for Video Technology , vol.20, no.l, pp.15,26, Jan. 2010. X. Mei, X. Sun, M. Zhou, S. Jiao, H. Wang, and X. Zhang, "On building an accurate stereo matching system on graphics hardware," in International Conference on Computer Vision Workshops (ICCV Workshops). IEEE, 2011, pp. 467-474. M. Humenberger, T. Engelke, and W. Kubinger, "A census-based stereo vision algorithm using modified semi-global matching and plane fitting to improve matching quality," in Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, 2010, pp. 77-84. F. Huang, K. Shou, and R. Klette, "Geometrical fundamentals of polycentric panoramas," in Eighth International Conference on Computer Vision (ICCV 2001). IEEE, 2001, vol.1, pp.560-565. S. K. Wei, F. Huang, and R. Klette, "Specification of image acquisition parameters for stereo panoramas," ml6th International Conference on Pattern Recognition Proceedings. IEEE, 2002, vol. 3, pp. 603-606). B. Reddy, and B. N. Chatterji, "An FFT-based technique for translation, rotation, and scale-invariant image registration," in IEEE Transactions on Image Processing, vol. 5, no. 8 pp. 1266-1271, 1996.

148