LiDAR and Camera Calibration using Motion Estimated by SensorFusion Odometry

Ryoichi Ishikawa1, Takeshi Oishi1 and Katsushi Ikeuchi2

Abstract— In this paper, we propose a method of targetlessand automatic Camera-LiDAR calibration. Our approach isan extension of hand-eye calibration framework to 2D-3Dcalibration. By using the sensor fusion odometry method, thescaled camera motions are calculated with high accuracy.In addition to this, we clarify the suitable motion for thiscalibration method.

The proposed method only requires the three-dimensionalpoint cloud and the camera image and does not need otherinformation such as reflectance of LiDAR and to give initialextrinsic parameter. In the experiments, we demonstrate ourmethod using several sensor configurations in indoor andoutdoor scenes to verify the effectiveness. The accuracy of ourmethod achieves more than other comparable state-of-the-artmethods.

I. INTRODUCTION

Sensor fusion has been widely studied in the field ofrobotics and computer vision. Compared to a single sensorsystem, higher level tasks can be performed by a fusionsystem combining multiple sensors. This type of systemcan be directly applied to three-dimensional environmentalscanning. For example, by combining cameras with LiDAR,it is possible to perform color mapping on range images(Fig. 1) or estimate accurate sensor motion for mobilesensing systems [1], [2], [3], [4].

In a 2D-3D sensor fusion system composed of a cameraand LiDAR, an extrinsic calibration of the sensors is re-quired. There are methods that can obtain extrinsic parameterby using target cues or manually associating 2D points on theimage with 3D points on the point cloud. However, manuallyestablishing correspondences for accurate calibration is labo-rious because it requires multiple matches. Moreover, eventhough a lot of correspondence can be created in this way,the accuracy of calibration is still insufficient. Automatedmethods such as [5], [6] use targets that can be detectedon both 2D images and 3D point clouds. However, sinceit is necessary to prepare targets detectable by both thecamera and LiDAR, it is impractical and undesirable foron-site calibration. Recently, automatic 2D-3D calibrationmethods that do not require targets have been proposed.However, since the information obtained from each sensor ismulti-modal, the calibration result depends on the modalitybetween the sensors.

In this paper, we propose an automatic and targetlesscalibration method between a fixed camera and LiDAR. As

1 Ryoichi Ishikawa and Takeshi Oishi are with Institute of In-dustrial Science, The University of Tokyo, Japan {ishikawa,oishi}@cvl.iis.u-tokyo.ac.jp

2Katsushi Ikeuchi is with Microsoft, USA,katsuike@microsoft.com

LiDAR

CameraA

B

X X

Fig. 1. Top: Motion-based 2D-3D Calibration. In our method, the LiDARmotion is estimated by ICP algorithm and Camera motion is initiallyestimated by using feature point matching and then estimated with scaleby sensor fusion system. Bottom: Colored scan by HDL-64E. Texture istaken by Ladybug 3 and our calibration result is used for texturing.

shown in Fig. 1, the proposed method is based on hand-eyecalibration. In our method, the motions of the sensors areestimated respectively and calibration is performed using theestimated motions. Each sensor motion is calculated in thesame modal and extrinsic parameter is derived numericallyfrom each sensor motion. In the conventional motion-based2D-3D calibration, the motion of the camera is obtainedfrom only 2D images [7]. However, the motion can onlybe estimated up to scale using only camera images. Theprecision of the extrinsic parameter is greatly affected bythe motion error in the hand-eye calibration. Although thescale itself can be calculated simultaneously with extrinsicparameter from multiple motions, hand-eye calibration withscaleless motion deteriorates the accuracy of calibration.

On the other hand, in the sensor fusion odometry usingthe LiDAR and the camera, the motion of the camera canbe accurately estimated with scale if the extrinsic parameterbetween the sensors are known [1], [2], [3], [4]. In ourmethod, we adopt the idea of camera motion estimation usingsensor fusion odometry. First, an initial extrinsic parameter isobtained from scaleless camera motions and scaled LiDARmotions. Next, the camera motions are recalculated withscale using the initial extrinsic parameter and the point

arX

iv:1

804.

0517

8v1

[cs

.CV

] 1

4 A

pr 2

018

cloud from the LiDAR. Then the extrinsic parameter iscalculated again using the motions. This recalculation ofcamera motions and recalculation of the extrinsic parameterare repeated until the estimation converges.

Our proposed method requires that the camera and theLiDAR have overlap in their measurement ranges and thatthe LiDAR’s measurement range is 2D to align scans for theLiDAR motion estimation. The contributions of this paperare shown below.

• As far as we know, this method is the first approach thatincorporates camera motion estimation through sensorfusion into 2D-3D calibration.

• We study the optimal sensor motion which this calibra-tion method works effectively.

• The input is only the RGB image from a camera and thethree-dimensional point cloud from a LiDAR and doesnot need any other information such as the reflectance ofa LiDAR or the initial value of the extrinsic parameter.The estimation result of the extrinsic parameter is moreaccurate than other methods with a small number ofmotion.

II. RELATED WORKS

Our work is related to the targetless and automatic 2D-3Dcalibration and hand-eye calibration.

A. Target-less multi-modal calibration

Targetless and automatic 2D-3D calibration methods gen-erally use the common information existing in both imageand point cloud. For example, portions that appear as dis-continuous 3D shapes are highly likely to appear as edgeson an RGB image. Methods that align this three-dimensionaldiscontinuous portion with the 2D edge has been proposed[8], [9]. Meanwhile, a multi-modal alignment method usingMutual Information (MI) was proposed by Viola et al. [10],and it has been developed mainly in the field of medicalimaging. 2D-3D calibration methods using MI for evaluatingthe commonality between LiDAR and camera have also beenproposed in recent years. As indicators evaluated throughMI, reflectance - gray scale intensity [11], surface normal -gray scale intensity [12], and multiple evaluation indicatorsincluding discontinuities of LiDAR data and edge strengthin images [13] etc. are used. Taylor and Nieto also proposedgradient based metric in [14].

While these methods align 3D point cloud to 2D imageusing 3D to 2D projections, some texturing methods usingimages taken from multiple places with a camera to constructa stereo, reconstructing 3d structure from the images andaligning it to 3D point cloud has also been proposed. In[15], a method of computing the extrinsic parameter betweenthe LiDAR and the camera for the texturing a dense 3Dscan is proposed. The extrinsic calibration was done byaligning dense three-dimensional range data and sparse three-dimensional data reconstructed from two-dimensional stereoimages.

B. Hand-eye Calibration

The method of changing the position and orientation ofthe sensor and performing the calibration using the motionsobserved by each sensor is known as hand-eye calibration.Let A and B are the changes in position and orientationobserved by two fixed sensors respectively, and X be theunknown relative position and orientation between sensors.Then the expression AX = XB holds(Refer top of Fig. 1).By using this expression to solve X, extrinsic parameterbetween sensors can be obtained [16], [17]. Furthermore,due to the influence of noise on the sensor, Kalman Filterwas used in calibration methods for estimating the bias ofthe sensor simultaneously [18], [19].

In [20], a method to calibrate the transitions betweenfour cameras mounted on the car using visual odometryis proposed. In [7], Taylor and Nieto propose a methodto obtain the motion of a sensor in 2D-3D calibrationand estimate the extrinsic parameter and the time offsetof the motion between the sensors. In their method, ahighly accurate result is obtained by combining a multi-modal alignment method. However, since the method usesthe scaleless position transition estimated from the cameraimages, it is difficult to obtain an accurate camera motion.In particular, it is difficult to accurately obtain the translationparameters with a small number of motions.

III. METHODOLOGY

Figure 2 shows the overview of our method. This methodis roughly divided into two steps. In the initialization phase,we estimate the extrinsic parameter from the LiDAR motionsusing ICP alignment and camera motions using featurepoint matching. Then we alternatingly iterate estimating theextrinsic parameter and the camera motions through sensorfusion odometry.

A. Initial calibration parameter estimation

1) Sensor motion estimation: First, we explain about themotion estimation of each sensor in the initial extrinsicparameter estimation.

LiDARFor the estimation of the LiDAR motion, we use a high-

speed registration method through ICP algorithm whichsearches correspondence points in gaze directions [21]. Wecreate meshes on point clouds in advance using a sequenceof points and project these points onto two dimension anduse Voronoi splitting. When aligning scans, initially, thethreshold value of the distance between corresponding pointsis set to be large, and the outlier correspondences areeliminated while gradually decreasing the threshold value.

CameraFor the initial motion estimation of the camera, a method

using standard feature point matching is used. First, we ex-tract feature points from two images using AKAZE algorithm[22], calculate descriptors, and make matchings. From thematchings, we calculate the initial relative position and posebetween camera frames using 5 point algorithm [23] andRANSAC [24]. After obtaining the initial relative position

Camera images

Initialization

Feature point detection & matching

Iteration phase

Range data ICP alignment

Extrinsic calibration

Camera motion estimation

Extrinsic calibration

Camera motion re-estimation

3D-2D Projection

Tracking

Camera motion estimation

Input Output

,{ , }

{ , }

{ , } { , }

,

Fig. 2. Overview

and orientation, optimization is performed by minimizing theprojection error using an angle error metric with the epipolarplane was used [25].

2) Initial extrinsic parameter calibration from sensor mo-tion: In order to obtain the relative position and orientationbetween the two sensors from the initially estimated motions,we use a method that extends the normal hand-eye calibrationto include the estimation of the camera motion’s scale. Letthe i th position and pose changes measured by the cameraand the LiDAR be 4 × 4 matrix Ai, Bi respectively andextrinsic parameter between two sensors be 4 × 4 matrixX. AiX = XiB can be established and the following twoequations hold by decomposing it [16],

RiaR = RRi

b (1)Ri

at+ tia = Rtib + t, (2)

where Ria and Ri

b is a 3×3 rotation matrix of Ai and Bi, tiaand tib represents the 3×1 vector of translational componentof Ai, Bi. Let ki

a and kib be rotational axis of rotation matrix

Ria and Ri

b. When Eq. 1 holds, following equation holds,

kia = Rki

b. (3)

Since the absolute scale of translational movement betweencamera frames can not be calculated, Eq. 2 is written asfollowing using the scale factor si,

Riat+ sitia = Rtib + t. (4)

R is linearly solved by using SVD from series of kia,k

ib.

However, to solve the rotation, more than two positionand pose transitions are required and rotations in differentdirections must be included in the series of transition. Innonlinear optimization, R is optimized by minimizing thefollowing cost function derived from Eq. 1,

R = arg minR

∑i

∣∣RiaR−RRi

b

∣∣ (5)

After optimizing R, t and si is obtained by constructing asimultaneous equation and solving it linearly using the leastsquares method.

B. Iteration of Camera motion estimation and Calibrationfrom sensor motion

Initial extrinsic parameter can be obtained using estimatedLiDAR motions and initial camera motions. However, scaleinformation of camera motion cannot be obtained fromcamera images alone. The rotation component of the extrinsicparameter is independent of the scale information and canbe accurately estimated in the initial parameter estimationphase. On the other hand, the translation of the extrinsicparameter can be calculated from the difference betweenthe movement amount of the camera and the LiDAR whenrotating sensors as indicated in Eq. 2. Therefore, since theprecision of the translational component of the extrinsicparameter is deeply related to the accuracy of camera motionestimation, it is difficult to accurately estimate the extrinsicparameter from the scaleless camera motion.

On the other hand, the motion estimation using the sensorfusion system can solve the scaled motion with high accuracy[1], [2]. Once the extrinsic parameter is estimated, we canestimate the camera motion Ai with scale by using the givenextrinsic parameter X and the range data scanned by theLiDAR. After motion estimation, the extrinsic parameter Xis re-estimated using the series of AiandBi. Since there is noneed to estimate the translation component of Ai at the sametime, it is possible to estimate the translation component ofX more accurately than the initial estimation. Estimatingthe camera motion Ai again using the re-estimated X andthe range data increases the accuracy of Ai. The extrinsicparameter is then optimized by repeating the estimationof the camera motion and the estimation of the extrinsicparameter alternately until convergence.

1) Camera motion estimation with range data: Figure 3shows that the schematic diagram of constructing 2D-3D cor-respondence. Inputs are point cloud in the world coordinates,two camera images taken at the position of camera 1, 2,and extrinsic parameter to localize camera 1 into the worldcoordinates. First, a certain point p in the point cloud ontothe image in Camera 1 using projection function Proj(pc)

v

3D surface

Camera 1

p

Camera 22D image

v’

Fig. 3. Schematic diagram of how to obtain 2D-3D correspondences

which project 3D points pc in camera coordinates ontocamera image and extrinsic parameter by following equation,

v = Proj(Rp+ t), (6)

where v is the vector heading from the center of camera 1 tothe corresponding pixel. Then we track the pixel on whichp is projected from camera image 1 to image 2 using KLTtracker [26]. Let v′ be the vector heading from the centerof camera 2 to the tracked pixel. Now the point p and thevector v′ of 2D-3D correspondence is constructed.

After constructing 2D-3D correspondences, it is possibleto optimize the relative position and orientation of the camera1 and the camera 2 by minimizing the projection error. Let(v′j ,pj) be the j th 2D-3D correspondence in i th motion,the position and pose transition between cameras Ri

a, tia can

be optimized by minimizing the following angle metric costfunction.

Ria, t

ia = arg min

Ra,ta

∑j

∣∣v′j × Proj(Ra(Rpj + t) + ta)∣∣ (7)

For the initial values of Ra and ta, generalized perspective3 point algorithm [27] is used in the first iteration. After thesecond iteration, the estimation result in the previous iterationis used.

2) Parameter Calibration: Once the position and posetransition of the camera is recalculated, the extrinsic param-eter is optimized again using the motion of the camera andthe LiDAR. In each iteration, R and t is solved linearly andnon-linearly. In non-linear optimization, R is optimized byEq. 5 and t is optimized by following,

t = arg mint

∑i

∣∣(Riat+ tia)− (Rtib + t)

∣∣ (8)

IV. OPTIMAL MOTION FOR 2D-3D CALIBRATION

We consider the motion suitable for the calibration takinginto consideration the influence each other’s error has onthe mutual parameter estimation. During the alternatingestimation of the extrinsic parameter between the sensors

βv

Camera 1

p

Camera 2v’αv

EstimatedCamera 1

p’

a

a EstimatedCamera 2

estv’

Projection error

Fig. 4. Schematic diagram when there is an error in the localized positionof the camera 1

and the motion of the camera, it is inevitable to estimatethe position and pose of each other with the error. Sincemotion estimation and extrinsic parameter estimation are alsodependent on the measured environment and the numberof motions, it is difficult to obtain precise convergenceconditions. However, it is possible to consider the motionthat is likely to converge the estimation.

A. Camera motion estimation

First, we consider the influence of the extrinsic parametererror on the localization of the camera and the conditionsunder which the estimation is successful. We suppose thecase where there is an error in the extrinsic parameter givenin the section as shown in Fig. 4. Let ”Estimated Camera 1”be the estimated position of camera 1 with respect the actualcamera position (Camera 1, 2 in Fig. 4). The rotation errorbetween Camera 1 and Estimated Camera 1 is considered tobe negligibly small.

Next, consider the operation of creating 2D-3D correspon-dence. In the projection step, a certain point p′ on the pointcloud is projected onto the Estimated camera 1. However,a difference is caused between the projected pixel and the3D point due to the error of the extrinsic parameter. Thenthe pixel on which the point p′ is projected is tracked on theCamera 2 image. Let v be the vector heading from Estimatedcamera 1 to the point p′. Point p is actually correspondsof the vector v. Ignoring the error of pixel tracking fromCamera 1 to Camera 2, the direction vector from Camera 2to the pixel corresponding to v is v′. On the computer, thethree-dimensional point p′ and v′ correspond to each other.

For the projection error, assuming that there is no rotationerror in estimating the position and orientation of the camera

2, the projection error when the estimated position is aest is

e(aest) =

∣∣∣∣v′ × αv − aest|αv − aest|

∣∣∣∣ . (9)

From Fig. 4, let a be the vector directing from Camera 1 toCamera 2, v′ is expressed as following,

v′ =βv − a

|βv − a|. (10)

Substitute Eq. 10 for Eq. 9,

e(aest) =

∣∣∣∣ βv − a

|βv − a|× αv − aest|αv − aest|

∣∣∣∣ . (11)

When optimizing aest, we compute the projection error forv in all directions and aest approach to the point where thesum of the projection error is minimized.

Now, in order to estimate the extrinsic parameter ac-curately, ideal camera motion should be estimated as realcamera actually moves. In other words, ideally estimatedmotion becomes aest → a. In order for aest to approach a,the projection error e(a) when Estimated Camera 2 is locatedaway from Estimated Camera 1 by a becomes small. e(a)is represented by the following equation:

e(a) =

∣∣∣∣ (α− β)(a× v)

|βv − a||αv − a|

∣∣∣∣ . (12)

From Eq. 12, the followings can be said.• The smaller the a, the smaller the projection error

even if the extrinsic parameter contains error. That is,the smaller the moving distance of the camera is, themore accurate the estimation becomes. In other words,when a is small, the existence probability of EstimatedCamera 2 appears to the periphery of the ideal placeacutely. Therefore in the subsequent extrinsic param-eter estimation, the existence probability of EstimatedCamera 1 also appears sharply around the true value.

• The smaller the value of (α − β), the smaller theprojection error. The cases the difference of (α − β)comes out are, for example, when there is a large stepunder environments or when the incident angle fromthe camera is shallow. Therefore it is preferable thatthe calibration environment is, for example, surroundedby smooth walls.

B. Extrinsic parameter estimation

Next, we consider the influence of the error in the esti-mated motions on extrinsic parameter calibration. Ignoringthe rotation error for the simplification, consider the casewhere there is the error in the translation of the cameramotion and the translation of the extrinsic parameter forEq. 2. Let ea and e be the error with respect to ta andt,

Ra(t+ e) + ta + ea = Rtb + t+ e. (13)

Taking the difference between Eq. 13 and Eq. 2,

ea = (I−Ra)e. (14)

Fig. 5. Indoor and outdoor calibration scene taken by Ladybug 3

In the case seeing Eq. 14 as a single unit, when the rotationamount of the Ra is small, the translation errors become|ea| < |e|. This indicates that the error of the cameramotion propagates to the extrinsic parameter in the divergingdirection to the error.

If the amount of error propagation in estimating theextrinsic parameter from the camera motions does not exceedthe amount of the error reduction in the camera motion esti-mation using the distance image, the accuracy of the relativeposition and orientation is improved by the proposed method.Therefore, in order to reduce the propagation amount oferror in relative position and pose estimation, increasing therotation amount of the camera motion is effective. It is alsoeffective to sample a plurality of motions as much as possiblefor robust extrinsic parameter estimation. Regarding theamount of rotation of the camera motion, if the appearance ofthe image changes significantly, it might affect the accuracyof the motion estimation. Therefore this also needs to betaken into account. Although the proposed method can beapplied to perspective camera, an omnidirectional camera hasadvantage because it is possible to secure a common field ofview even when the camera rotates significantly.

V. EXPERIMENTAL RESULTS

In the experiments, we conduct calibrations in indoorand outdoor environments shown in Fig. 5 using panoramicLiDAR, multi-beam LiDAR, and omnidirectional camera.One of the compared methods is image-based calibrationusing the camera motions with no scale, which is used asthe base in [7]. In Fig. 2, this is the initialization outputand hereinafter referred to it as ”Scaleless”. In addition toScaleless, we used calibration by Manual correspondenceacquisition and calibration with alignment using MI [11] asthe other compared methods.

Fig. 6. Sensor configuration. (a)Imager 5010C and Ladybug 3, (b)HDL-64E and Ladybug 3

0

0.01

0.02

0.03

0.04

0.05

0.06

1 2 3 4 5Rot

atio

n er

ror

(rad

)

Number of motion

Fig. 7. Transition graph of rotation error when changing the number ofmotion. Blue line: Scaleless, Red line: Ours.

A. Evaluation with colored range data

First, the results using the evaluation datasets are shown.To measure the data set, two range sensors Focus S 150 byFARO Inc.1 and Imager 5010C by Zollar+Flohlich Inc.2 areused. Three-dimensional panoramic point clouds are scannedwith both range sensors. In the data measured by FocusS 150, a colored point cloud is obtained using a phototexture function of it. In the evaluation, inputs are a pseudopanorama rendered image obtained from colored point cloudscanned by Focus S 150 and a point cloud scanned by Imager5010C. Ground truth is computed through registration of twopoint clouds scanned by the two sensors.

In the indoor scene dataset, motions are obtained byrotating sensors 5 times in the vertical direction and 5 timesin the horizontal direction to measure the data. The result ofcalibration with changing the number of motion in the indoorscene is shown in Fig. 7 and Fig. 8. The horizontal axis ofthe graphs indicates the number of horizontal and verticalmotions used for calibration. For example, when the numberis one, it indicates that the calibration is performed usingtwo motions in total with one horizontal and one verticalmotion. Evaluation is performed by conducting calibration10 times in each motion number sampling motions randomly.

1https://www.faro.com2http://www.zf-laser.com

00.10.20.30.40.50.60.7

1 2 3 4 5Tran

slat

ion

erro

r (m

)

Number of motion

00.0020.0040.0060.008

0.010.0120.0140.0160.018

0.02

1 2 3 4 5

Tran

slatio

n er

ror

(m)

Number of motion

Fig. 8. Transition graph of translation error when changing the numberof motion. Blue line: Scaleless, Red line: Ours. Bottom shows the result ofour method only.

0.0100 0.0280

0.2382

0.0063 0

0.050.1

0.150.2

0.250.3

manual MI Scaleless Ours

Tra

nsla

tion

erro

r(m

)

0.0027

0.0110

0.0150

0.0087

0

0.005

0.01

0.015

0.02

manual MI Scaleless Ours

Rot

atio

n er

ror(

rad)

Fig. 9. Error from the ground truth of the calibration result by each methodin indoor scene

Figure 7 and Fig. 8 show the graphs plotting the average andstandard deviation of the error of rotation and translation ofextrinsic parameter. The blue line indicates the error of theScaleless and the red line indicates the error of the extrinsicparameter estimated by ours. In terms of rotation, there is nogreat improvement in accuracy. However, for the translationerror, the accuracy improves dramatically using the proposedmethod as shown in Fig. 8. It is also shown that the error isgradually decreasing by increasing the number of motions.

The results compared with other methods (Manual, MI)are shown in Fig. 9. In the registration by maximizingMI, only 1 scan is used, and the initial point was shiftedfrom the ground truth by a fixed distance (0.1m) in therandom direction only for the translation parameter. In theManual calibration, calibration is carried out by acquiring30 correspondings as much as possible from all directions.From the Fig. 9, the rotational error is less than 1 degree inany method. While, in translational error, ours achieves theleast error compared to other methods.

Evaluation results using the outdoor environment datasetare also shown in Fig. 10. Motions are obtained by rotatingsensors by 3 times in the vertical direction and 3 times in thehorizontal direction to measure the data. For the rotation, ourmethod obtained the best result. However, in any method, theerrors are less than 0.5 degrees and no significant differenceis seen. On the other hand, ours has the best estimationresult for the translation. Considering that the accuracy is lessthan 1 cm with the same number of motions in the indoor

0.0473 0.0475

0.3365

0.0250 0

0.1

0.2

0.3

0.4

manual MI Scaleless Ours

Tra

nsla

tion

erro

r(m

)

0.0072

0.0041 0.0055

0.0033

0

0.002

0.004

0.006

0.008

manual MI Scaleless Ours

Rot

atio

n er

ror(

rad)

Fig. 10. Error from the ground truth of the calibration result by eachmethod in outdoor scene

Fig. 11. The pictures in which panorama images taken by Ladybug 3and panorama rendered reflectance image scanned by Imager 5010C arealternately arranged like a checker. When extrinsic parameter is correct,consistency is established between the two images. We set stitching distanceof panoramic image to 4m in indoor scene and 7m in outdoor scene.

environment, we can say that the indoor scene is good forcalibration.

B. Ladybug 3 and Imager 5010C

Next, we show the results of calibration using omnidirec-tional camera Ladybug 3 by FLIR Inc.3 and Imager 5010C.The appearance of the sensor configuration is shown inFig. 6 (a).

For the evaluation, as shown in Fig. 11, the evaluationis performed by overlaying the image of Ladybug 3 andthe reflectance image obtained by panorama rendering fromthe center of the estimated camera position in the pointcloud. Images are displayed alternately in the checker pat-tern. We then confirm the consistency on the two images.From Fig. 11, Scaleless before optimization does not haveconsistency between RGB image and reflectance image, butthe results of the proposed method have consistency betweentwo images.

C. Ladybug 3 and Velodyne HDL-64E

We show the result of extrinsic calibration of HDL-64E by Velodyne Inc. 4 which is multi-beam LiDAR andLadybug 3 in the indoor scene. The appearance of the sensorconfiguration is shown in Fig. 6 (b). In the measurement of

3https://www.ptgrey.com/4http://velodynelidar.com/

0.1152

0.4218 0.4547

0.0262 0

0.1

0.2

0.3

0.4

0.5

manual MI Scaleless Ours

Tra

nsla

tion

erro

r(m

)

0.0184

0.2265

0.0108 0.0105 0

0.05

0.1

0.15

0.2

0.25

manual MI Scaleless Ours

Rot

atio

n er

ror(

rad)

Fig. 12. Error from the ground truth of the calibration result by each methodusing HDL-64E and Ladybug 3. In MI, Scaleless, and ours, calibrationperformed with 16 scans

data, the rover loading the sensors is operated to generate therotation motion in the vertical direction and the horizontaldirection. For the rotation in the vertical direction, motionsare generated by raising and lowering only the front wheelby about 4 cm step. When acquiring data, we stopped therover and measured it with stop-and-scan. To obtain rangedata and camera image scanned at the same position, wevisually check the timing when the LiDAR and the camerawere stationary.

The ground truth of extrinsic parameter between HDL-64Eand Ladybug 3 is indirectly obtained by using the point cloudmeasured by Imager 5010C under the same environment andcomputing the relative positions and orientations with Imager5010C. Regarding the HDL-64E and Imager 5010C, the po-sition and orientation are obtained by aligning the range datascanned by each sensor. For Ladybug 3 and Imager 5010C,the extrinsic parameter is obtained by manually specifyingthe correspondence point between the panorama image andthe three-dimensional reflectance image and computing theextrinsic parameter using the correspondences. In Scalelessand the proposed method, eight horizontal rotation motionsand eight vertical rotation motions are randomly sampledeach time, and an average error of 10 times is recorded.In Manual calibration, calibration is carried out using onlyone scan, taking corresponding points between the panoramicimage of Ladybug 3 and the 3D reflectance image of HDL-64E. For MI, MI is calculated with 16 scan sets of image-point cloud scanned at each position.

From Fig. 12, Manual calibration fails to obtain accurateresults because the narrow scan range and the sparse pointcloud of HDL-64E make correspondence construction dif-ficult. Also in MI, since HDL-64E has low resolution andinformation of reflectance is not clear, optimization cannotbe completed with this dataset. On the other hand, in themotion-based methods, accuracies are less than 1 degree inthe rotation in both Scaleless and ours. However, in Scaleless,translation is significantly different from the ground truth.In contrast, in ours, highly accurate translation results areobtained.

The proposed method can also work with motions acquiredby operating rover. To obtain all the extrinsic parameter of6-DoF by hand-eye calibration based method, it is necessaryto rotate in two or more directions. However, regarding therotation motion in the vertical direction, a part of the platformon which the sensors are mounted must be raised. Although

this operation is more difficult than the horizontal rota-tion, this experiment demonstrates that the proposed methodworks well with the vertical rotational motions obtained bya reasonable mobile platform operation such as raising andlowering the small step.

VI. CONCLUSION

In this paper, we present targetless automatic 2D-3Dcalibration based on hand-eye calibration using sensor fu-sion odometry for camera motion estimation. The proposedmethod can be fully utilized with less translation and largerrotation camera motions. It is also preferable to carry out themeasurement for calibration in the place surrounded by flatterrains as much as possible.

It is necessary to rotate the sensor in multi directionsto carry out the hand-eye calibration. However, in manycases it is more difficult to make rotation in the verticaldirection than in the horizontal direction. Although theproposed method also requires satisfying this conditions, itis enough for carrying out the calibration to use verticalrotation obtained through reasonable movement by usingmobile platform. Therefore this method is highly practicaland it is possible to calibrate dynamically during scanningby choosing appropriate motions.

ACKNOWLEDGMENT

This work was partially supported by the social corporateprogram (Base Technologies for Future Robots) sponsoredby NIDEC corporation and also supported by JSPS ResearchFellow Grant No.16J09277.

REFERENCES

[1] Y. Bok, Y. Jeong, D. Choi, and I. Kweon, “Capturing Village-levelHeritages with a Hand-held Camera-Laser Fusion Sensor,” Interna-tional Journal of Computer Vision, vol. 94, pp. 36–53, 2011.

[2] B. Zheng, X. Huang, R. Ishikawa, T. Oishi, and K. Ikeuchi, “ ANew Flying Range Sensor: Aerial Scan in Omini-directions ,” inInternational Conference on 3D Vision, 2015.

[3] R. Ishikawa, M. Roxas, Y. Sato, T. Oishi, T. Masuda, and K. Ikeuchi,“A 3d reconstruction with high density and accuracy using laserprofiler and camera fusion system on a rover,” in 3D Vision (3DV),2016 Fourth International Conference on. IEEE, 2016, pp. 620–628.

[4] J. Zhang, M. Kaess, and S. Singh, “A real-time method for depthenhanced visual odometry,” Autonomous Robots, vol. 41, no. 1, pp.31–43, 2017.

[5] Q. Zhang and R. Pless, “Extrinsic calibration of a camera and laserrange finder (improves camera calibration),” in Intelligent Robots andSystems, 2004.(IROS 2004). Proceedings. 2004 IEEE/RSJ Interna-tional Conference on, vol. 3. IEEE, 2004, pp. 2301–2306.

[6] V. Fremont, P. Bonnifait, et al., “Extrinsic calibration between a multi-layer lidar and a camera,” in Multisensor Fusion and Integration forIntelligent Systems, 2008. MFI 2008. IEEE International Conferenceon. IEEE, 2008, pp. 214–219.

[7] Z. Taylor and J. Nieto, “Motion-based calibration of multimodalsensor extrinsics and timing offset estimation,” IEEE Transactions onRobotics, vol. 32, no. 5, pp. 1215–1229, 2016.

[8] J. Levinson and S. Thrun, “Automatic online calibration of camerasand lasers.” in Robotics: Science and Systems, vol. 2, 2013.

[9] T. Cui, S. Ji, J. Shan, J. Gong, and K. Liu, “Line-based registrationof panoramic images and lidar point clouds for mobile mapping,”Sensors, vol. 17, no. 1, p. 70, 2016.

[10] P. Viola and W. Wells III, “Alignment by Maximization of MutualInformation,” International Journal of Computer Vision, vol. 24, no. 2,pp. 137–154, 1997.

[11] G. Pandey, J. R. McBride, S. Savarese, and R. M. Eustice, “Auto-matic extrinsic calibration of vision and lidar by maximizing mutualinformation,” Journal of Field Robotics, vol. 32, no. 5, pp. 696–722,2015.

[12] Z. Taylor and J. Nieto, “A mutual information approach to automaticcalibration of camera and lidar in natural environments,” in AustralianConference on Robotics and Automation, 2012, pp. 3–5.

[13] K. Irie, M. Sugiyama, and M. Tomono, “Target-less camera-lidarextrinsic calibration using a bagged dependence estimator,” in Au-tomation Science and Engineering (CASE), 2016 IEEE InternationalConference on. IEEE, 2016, pp. 1340–1347.

[14] Z. Taylor, J. Nieto, and D. Johnson, “Multi-Modal Sensor CalibrationUsing a Gradient Orientation Measure,” Journal of Field Robotics,vol. 32, no. 5, pp. 675–695, 2015.

[15] A. Banno and K. Ikeuchi, “Omnidirectional texturing based on robust3d registration through euclidean reconstruction from two sphericalimages,” Computer Vision and Image Understanding, vol. 114, no. 4,pp. 491–499, 2010.

[16] Y. C. Shiu and S. Ahmad, “Calibration of wrist-mounted roboticsensors by solving homogeneous transform equations of the form AX=XB,” ieee Transactions on Robotics and Automation, vol. 5, no. 1, pp.16–29, 1989.

[17] I. Fassi and G. Legnani, “Hand to sensor calibration: A geometricalinterpretation of the matrix equation AX= XB,” Journal of FieldRobotics, vol. 22, no. 9, pp. 497–506, 2005.

[18] J. D. Hol, T. B. Schon, and F. Gustafsson, “Modeling and calibrationof inertial and vision sensors,” The international journal of roboticsresearch, vol. 29, no. 2-3, pp. 231–244, 2010.

[19] J. Kelly and G. S. Sukhatme, “Visual-inertial sensor fusion: Localiza-tion, mapping and sensor-to-sensor self-calibration,” The InternationalJournal of Robotics Research, vol. 30, no. 1, pp. 56–79, 2011.

[20] L. Heng, B. Li, and M. Pollefeys, “Camodocal: Automatic intrinsicand extrinsic calibration of a rig with multiple generic cameras andodometry,” in Intelligent Robots and Systems (IROS), 2013 IEEE/RSJInternational Conference on. IEEE, 2013, pp. 1793–1800.

[21] T. Oishi, A. Nakazawa, R. Kurazume, and K. Ikeuchi, “Fast Simul-taneous Alignment of Multiple Range Images using Index Images,”Proc. of 5th International Conference on 3-D Digital Imaging andModeling (3DIM2005), pp. 476–483, 2005.

[22] P. F. Alcantarilla and T. Solutions, “Fast explicit diffusion for acceler-ated features in nonlinear scale spaces,” IEEE Trans. Patt. Anal. Mach.Intell, vol. 34, no. 7, pp. 1281–1298, 2011.

[23] D. Nister, O. Naroditsky, and J. Bergen, “Visual odometry for groundvehicle applications,” Journal of Field Robotics, vol. 23, no. 1, pp.3–20, 2006.

[24] M. A. Fischler and R. C. Bolles, “Random Sample Consensus: AParadigm for Model Fitting with Applications to Image Analysis andAutomated Cartography,” Communications of the ACM, vol. 24, no. 6,pp. 381–395, 1981.

[25] A. Pagani and D. Stricker, “Structure from motion using full sphericalpanoramic cameras,” in Computer Vision Workshops (ICCV Work-shops), 2011 IEEE International Conference on. IEEE, 2011, pp.375–382.

[26] B. D. Lucas and T. Kanade, “An Iterative Image Registration Tech-nique with an Application to Stereo Vision,” in International JointConference on Artificial Intelligence, 1981, pp. 674–679.

[27] L. Kneip, D. Scaramuzza, and R. Siegwart, “A novel parametrizationof the perspective-three-point problem for a direct computation ofabsolute camera position and orientation,” in Computer Vision andPattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 2011,pp. 2969–2976.