isexpressedasthefirstreflec/onofon.Alinearconstraintonisholdasfollow,

A Linear Extrinsic Calibration of Kaleidoscopic Imaging System from Single 3D Point Kosuke Takahashi, Akihiro Miyata, Shohei Nobuhara, Takashi Matsuyama (Kyoto Univ.)

Background

•  Bioinformatics •  The marine products industry

3D reconstruction and motion analysis of micro-scale objects in water.

Finalgoal:3DReconstruc/onofmicro-scaleobjectsOur goal is 3D reconstruction of

Ø Object ：micro-scale object 　　　 　　　　　　 (An embryo，a water flea, etc) Ø Scale ：mm, nm, µm Ø Device ：microscope + camera.

In this work, we utilize a virtual multi-view system with planar mirrors ( KIS : Kaleidoscopic Imaging System )

Microscope

Water

Mirror

Vritual microscope

Mirror

Outlineofthiswork

This paper is aimed at proposing a new method of extrinsic calibration of KIS, i.e. estimation of mirror normals and distances .

Zhang utilizes a reference object of known geometry such as a chessboard.

It is difficult to prepare such reference object of known geometry in micro-scale.

This paper proposes a new linear extrinsic calibration method from a single 3D point of unknown geometry.

KIS : Kaleidoscopic Imaging System

ni di

Itisdifficulttocapturemul/-viewimageswithmul/plemicroscopesduetophysicalconstraints

Purpose

Proposedmethod

Es/ma/onofmirrornormals

Es/ma/onofmirrordistances

BundleAdjustment

INPUTProjec/onsofsingle3Dpoint

anditsreflec/ons

OUTPUTMirrorparameters

2

4y0 � y1 x1 � x0 x0y1 � x1y0

y2 � y12 x12 � x2 x2y12 � x12y2

y3 � y13 x13 � x3 x3y13 � x13y3

3

5n1 = 03⇥1

p

p0

⇡

nd

q

q0

Camera

A

,A

�1q = (x, y, 1)>

has 2 DOF

q0

q1

q2

q12q3

q13

p0

p1

⇡1 ⇡2

⇡3

Mirror Mirror

Mirror

p12

p2

p13

p3

= S1p2

Reflection on

p12 p2

, provide similar constraints as well.p3 p13

can be computed from single 3D point and its reflections.n

Secondreflec/onNon-linear

Linear

p

q

⇡i

⇡ij

di

dij

qi

qijni

nijpi

pij

[First reflection](A�1qi)⇥ pi = xi ⇥ pi = 03⇥1

(A�1qij)⇥ pij = xij ⇥ pij = 03⇥1

[Second reflections]

K[p0 d1 d2 d3]> = 030⇥1 Linear

Minimizing nonlinearly over . E(n1,n2,n3, d1, d2, d3) = [q0 � q̂0, e1, e2, e3, e

01,2, e

02,3, e

03,2, e

03,1, e

01,3]

p12 = S12p1

= S12S1p0 = S1S2p0

can be estimated by using more than or equal two reference points.

⇡1

||E(·)||2 n1,n2,n3, d1, d2, d3

(y2 � y12 x12 � x2 x2y12 � x12y2)n1 = 0

Epipolargeometrywithaplanarmirror

n

n

(n⇥ p)>p0 = 0

q>A>[n]>⇥A�1q0 = 0

(y � y

0x

0 � x xy

0 � x

0y)n = 0,

is intrinsic parameters

KeyIdea:U/lizing2Dprojec/onsofmul/plereflec/onstoformalinearsystemonmirrorparameters

Conven/onalmethod

Problem

Contribu/on

AquaVision Manyapplica/ons

Colinearityconstraint

p0 p1 p12⇡1 Reflection on ⇡12

Firstreflec/onLinear

⇡1

Implicitlyrequeststobeconsistentwiththeothermirrors.

Experiments:SimulaCon

Evaluation of our proposed method under the observation noise. [Baseline] Zhang’s method with a chessboard.

0 0.5 1 1.5 20

0.2

0.4

0.6

0.8

1

1.2

1.4#10-3

0 0.5 1 1.5 20

1

2

3

4

5

6 #10-3

0 0.5 1 1.5 20

0.5

1

1.5

2

2.5

Error of normals (rad) Error of distances (mm) Reprojection error (pixel)

Horizontal axis：The standard deviation of Gaussian noise (pixel). Vertical axis：Estimation error of each parameter.

Proposed method (5 points) + BA

Baseline (5 points) + BA

Proposed method (5 points)

Proposed method (1 point) + BA

Proposed method (1 point)

CameraSetup

ApplicaCon:3DreconstrucCon

Results

Mirrornormalsanddistances

Proposed method (5 points) + BA

Baseline (5 points) + BA

We can conclude that the proposed method (1) outperforms baseline method even without final BA. (2) can achieve comparable estimation linearly even with a single point.

While the estimated mirror parameters look close to each other, the reprojection error of the proposed method is smaller than that of the baseline method.

Experiments:Realdata

Result

Calibration

Method ReprojecConError(pixel)Proposed+BA 3.37Baseline 4.75

CameraProjector

Setup

Right figure shows a 3D rendering of the estimated 3D shape using the mirror parameters estimated by the proposed method. These results show the proposed method provides a sufficiently accurate calibration for 3D shape reconstruction.

CodeAvailable http://vision.kuee.kyoto-u.ac.jp/~nob/proj/kaleidoscope/

Evaluation of our proposed method with real data. [Baseline] Zhang’s method with a chessboard.

CameraMirror

Virtual camera n1n2

n3

d1 d2

d3

A reference point (unknown)

Reflections of a reference point ni di

Summary This paper proposes a new linear extrinsic calibration method of kaleidoscopic imaging system from a single 3D point of unknown geometry. INPUT

Projec/onsofsingle3Dpointanditsreflec/onsOUTPUT

Mirrornormalsanddistances.

di

ni

ni di

ProposedMethod

(Image size: 6000x4000 pixel)