Registration for Augmented Reality Neil Birkbeck 3/27/2006.

Registration for Registration for Augmented RealityAugmented Reality

Neil BirkbeckNeil Birkbeck3/27/20063/27/2006

OutlineOutline

►What is AR?What is AR?►ApplicationsApplications►Research AreasResearch Areas►Single Plane-Based CalibrationSingle Plane-Based Calibration►Multiple Plane-Based CalibrationMultiple Plane-Based Calibration►Other TechniquesOther Techniques►SummarySummary

Augmented Reality (AR)Augmented Reality (AR)

►Definition (according to Azuma):Definition (according to Azuma): A system which combines virtual objects A system which combines virtual objects

with the real world in real-time. The with the real world in real-time. The virtual objects should be correctly virtual objects should be correctly registered with the real world.registered with the real world.

►Display MethodsDisplay Methods Head Mounted Display (HMD)Head Mounted Display (HMD)

►Video HMDVideo HMD►Optical HMDOptical HMD

Monitor/TVMonitor/TV

AR ApplicationsAR Applications

► AdvertisingAdvertising► MilitaryMilitary► Surgical/MedicalSurgical/Medical► MaintenanceMaintenance► Entertainment Entertainment ► CommerceCommerce► ……


► AdvertisingAdvertising

► MilitaryMilitary► Surgical/MedicalSurgical/Medical► MaintenanceMaintenance► Entertainment Entertainment ► CommerceCommerce► ……

http://www.informationinplace.com/Solutions/CaseStudies/case_RDECOM/Demos/Blast/lBlast.html


► AdvertisingAdvertising► MilitaryMilitary

► Surgical/MedicalSurgical/Medical► MaintenanceMaintenance► Entertainment Entertainment ► CommerceCommerce► ……

http://jama.ama-assn.org/cgi/content/full/292/18/2214-b/JLD40609F1


► AdvertisingAdvertising► MilitaryMilitary► Surgical/MedicalSurgical/Medical

► MaintenanceMaintenance► Entertainment Entertainment ► CommerceCommerce► ……

http://hci.rsc.rockwell.com/AutomationFair_2003/


► AdvertisingAdvertising► MilitaryMilitary► Surgical/MedicalSurgical/Medical► MaintenanceMaintenance► Entertainment Entertainment ► CommerceCommerce► ……

http://wearables.unisa.edu.au/projects/ARQuake/www/


► AdvertisingAdvertising► MilitaryMilitary► Surgical/MedicalSurgical/Medical► MaintenanceMaintenance► Entertainment Entertainment

► CommerceCommerce► ……

http://virtual.vtt.fi/multimedia/clipvf.html

Research AreasResearch Areas

►Display TechnologiesDisplay Technologies improving the wearable HMD devicesimproving the wearable HMD devices

►Mobile ComputingMobile Computing► Calibration/RegistrationCalibration/Registration

hardwarehardware►GPS, magnetic trackingGPS, magnetic tracking

vision-based/range-basedvision-based/range-based hybrid methodshybrid methods

► Rendering IssuesRendering Issues Calibration of illumination for correct/realistic Calibration of illumination for correct/realistic

shadingshading

Vision-Based RegistrationVision-Based Registration

►MotivationMotivation inexpensive/ubiquitousinexpensive/ubiquitous

►video HMD already uses imagesvideo HMD already uses images AccurateAccurate

►pixel/sub-pixel precision possiblepixel/sub-pixel precision possible Images useful for recovering Images useful for recovering

shading/occlusionsshading/occlusions►Potential DownsidesPotential Downsides

computational requirementscomputational requirements feasibility over wide-rangesfeasibility over wide-ranges

Vision-Based RegistrationVision-Based Registration

► Image features are used to register position Image features are used to register position of viewer (camera)of viewer (camera)

► Existing methods can be categorized using Existing methods can be categorized using the following characteristics:the following characteristics: Feature type: Feature type:

►natural - corners, lines, image patches.natural - corners, lines, image patches.►synthetic – patterns, LEDssynthetic – patterns, LEDs

3D Position of features: 3D Position of features: ►known – the 3D positions of the features are knownknown – the 3D positions of the features are known►unknown – the 3D positions of features are unknown unknown – the 3D positions of features are unknown

(e.g., similar to SFM)(e.g., similar to SFM)

Vision-Based Registration Vision-Based Registration OverviewOverview

►When 3D positions of features are When 3D positions of features are known, known, XXii=(x=(xii,y,yii,z,zii), find camera ), find camera parameters that align the projection of parameters that align the projection of the 3D feature points with the observed the 3D feature points with the observed 2D feature points, 2D feature points, xxii=(u=(uii,v,vii). This is an ). This is an optimization problem over the space of optimization problem over the space of camera parameters camera parameters pp::

argminargminpp ∑ ∑ii|f(|f(pp,,XXii)-)-xxii||22

The Planar Case:The Planar Case:A registration method for features on a A registration method for features on a

planeplane►A specific implementation for features A specific implementation for features

on a single plane using the typical on a single plane using the typical perspective projection model.perspective projection model.

►Why planes?Why planes? Easy to make planar calibration patternsEasy to make planar calibration patterns

►Measurement of relative 2D positions of Measurement of relative 2D positions of feature points on planes is straightforward.feature points on planes is straightforward.

Occur frequently on man-made structuresOccur frequently on man-made structures►Rooftops, building walls, etc.Rooftops, building walls, etc.

Zhang’s Planar Calibration Zhang’s Planar Calibration MethodMethod

► Calibrate the camera using correspondences Calibrate the camera using correspondences between (at least) four planar points to their between (at least) four planar points to their 3D reference positions3D reference positions

► The calibration method is based on The calibration method is based on determining the 2D projective Homography determining the 2D projective Homography between the observed points and their between the observed points and their reference position.reference position.

► A 2D projective Homography is a 3x3 matrix A 2D projective Homography is a 3x3 matrix that operates on 2D homogeneous points:that operates on 2D homogeneous points:


The 3x3 Homography defines the motion from the image coordinates of the pattern to the reference coordinates

The camera calibration parameters are extracted from the recovered Homography


►The method is based on the following The method is based on the following observation:observation:

►Where R is a 3x3 rotation matrix, t is a Where R is a 3x3 rotation matrix, t is a 3x1 translation vector, and K is the 3x1 translation vector, and K is the internal parameters of the camera.internal parameters of the camera.

Zhang’s Planar Calibration Zhang’s Planar Calibration Method…Method…

Assuming intrinsics are known:Assuming intrinsics are known:

Planar Calibration ExamplePlanar Calibration Example

►Augmented Reality with simple Augmented Reality with simple pattern:pattern: Planar pattern detected each frame by thresholding Planar pattern detected each frame by thresholding

and finding connected componentsand finding connected components Quadrilateral shapes are warped to squares and correct Quadrilateral shapes are warped to squares and correct

orientation is foundorientation is found Homography is recovered and used to calibrate cameraHomography is recovered and used to calibrate camera

Region-based AlternativeRegion-based Alternative

►On initialization, a user On initialization, a user selects a plane of interestselects a plane of interest

► The rectifying The rectifying Homography and rectified Homography and rectified template image are template image are retainedretained

Template

Input Image at time t

H

Region-based AlternativeRegion-based Alternative

►When new When new image arrives, image arrives, use image use image intensities to intensities to refine the refine the Homography Homography

Template

Image at time t Image at time t+1

HH

►Mathematically, given:Mathematically, given: a template, T, which is indexed by a set of a template, T, which is indexed by a set of

2D points, 2D points, Define the warp, which in the most Define the warp, which in the most

general case is a homography:general case is a homography:

The parameters of the warp are:The parameters of the warp are:

Region-Based SSD TrackingRegion-Based SSD Tracking(Lucas-Kanade Tracking)(Lucas-Kanade Tracking)

Region-Based SSD TrackingRegion-Based SSD Tracking

► Find the parameter update that minimizes the sum Find the parameter update that minimizes the sum of squared differences (SSD):of squared differences (SSD):

► To minimize, first perform Taylor series Expansion:To minimize, first perform Taylor series Expansion:

Region-Based SSD TrackingRegion-Based SSD Tracking

►The minimization problem is The minimization problem is equivalent to a least-squares problem, equivalent to a least-squares problem, with m equations, one for each xwith m equations, one for each x ii::

►Giving, the following:Giving, the following:

Region-Based SSD ExampleRegion-Based SSD Example

Image at time t Image at time t+1

Template Warped Image at t+1

Diff. Between template1


h11 h13h12 h21

h22 h23 h31 h32

Image derivatives w.r.t the homography parameters:

Image Diff

Update is Essentially a linear combination of Partial Derivative Images


► Successive improvement after several Successive improvement after several iterationsiterations

Rectified

Diff. FromTemplate

SSD score 6867 2809 1799 583

Region-Based AR ExampleRegion-Based AR Example

►A single planar region was identified, A single planar region was identified, tracked, and used to register the world tracked, and used to register the world coordinate frame with the cameracoordinate frame with the camera

Planar Calibration for ARPlanar Calibration for AR

►Problems with plane-based approaches:Problems with plane-based approaches: Poor registration for objects far from the Poor registration for objects far from the

planeplane Registration degrades at grazing viewsRegistration degrades at grazing views

►Limited viewing range with single planar Limited viewing range with single planar pattern/regionpattern/region

►Potential Solutions:Potential Solutions: use several planes (Buenaposada et al.) use several planes (Buenaposada et al.) use other feature types (Marchand et al.)use other feature types (Marchand et al.)

Multiple Region-Based Multiple Region-Based Registration Registration

► Use multiple planar regions that are registered Use multiple planar regions that are registered with respect to one another (3D model)with respect to one another (3D model)

► Mathematical formulation is similar to the single Mathematical formulation is similar to the single plane-based SSD trackingplane-based SSD tracking

► Update in camera parameters is influenced by all Update in camera parameters is influenced by all planar regionsplanar regions

Approximate 3D model

Other ApproachesOther Approaches

► Compute 3D model and register camera Compute 3D model and register camera (Davison et al.)(Davison et al.) Camera/User state is modeled with a position, Camera/User state is modeled with a position,

orientation, 3D velocity, and angular velocityorientation, 3D velocity, and angular velocity Salient image features are detected and Salient image features are detected and

matched in subsequent frames to initialize matched in subsequent frames to initialize uncertain 3D features uncertain 3D features

Extended Kalman Filter (EKF) is used to update Extended Kalman Filter (EKF) is used to update camera state, 3D feature position, and their camera state, 3D feature position, and their covariance matrices covariance matrices

http://www.doc.ic.ac.uk/~ajd/

SummarySummary

►Vision-based registration useful for ARVision-based registration useful for AR High accuracy is possibleHigh accuracy is possible

►Plane-based techniques simple and Plane-based techniques simple and efficientefficient

►Use of non-planar features more stable Use of non-planar features more stable through wider rangesthrough wider ranges

ReferencesReferences► Azuma, Ronald T. "A Survey of Augmented Reality." Azuma, Ronald T. "A Survey of Augmented Reality." Presence: Teleoperators and Presence: Teleoperators and

Virtual EnvironmentsVirtual Environments 6, 4 (August 1997), 355 - 385 6, 4 (August 1997), 355 - 385► Z. Zhang. Z. Zhang. A flexible new technique for camera calibrationA flexible new technique for camera calibration. . IEEE Transactions on IEEE Transactions on

Pattern Analysis and Machine Intelligence,Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000. 22(11):1330-1334, 2000. ► José Miguel Buenaposada, Enrique Muñoz, Luis Baumela. José Miguel Buenaposada, Enrique Muñoz, Luis Baumela. Tracking heads using Tracking heads using

piecewise planar modelspiecewise planar models.. Proc. of .. Proc. of Iberian Conference on Pattern Recognition and Iberian Conference on Pattern Recognition and Image AnalysisImage Analysis, IbPRIA 2003. LNCS 2652, pp. 126-133 (, IbPRIA 2003. LNCS 2652, pp. 126-133 (ISBN 3-540-40217-9)ISBN 3-540-40217-9), (c) , (c) Springer-Verlag. Palma de Mallorca, Spain, June 2003.Springer-Verlag. Palma de Mallorca, Spain, June 2003.

► Dana Cobzas and Peter Sturm, Dana Cobzas and Peter Sturm, 3D SSD Tracking with Estimated 3D Planes3D SSD Tracking with Estimated 3D Planes,,In proceedings of In proceedings of Computer Robot Vision (CRV05Computer Robot Vision (CRV05),), Pages 129-134, 2005. Pages 129-134, 2005.

► José Miguel Buenaposada Biencinto, Luis Baumela Molina. José Miguel Buenaposada Biencinto, Luis Baumela Molina. Real-time tracking and Real-time tracking and estimation of plane poseestimation of plane pose., In Proc. of ., In Proc. of International Conference on Pattern International Conference on Pattern RecognitionRecognition, ICPR 2002. Vol. II, pp. 697-700. (c) IEEE. Quebec, Canada, August , ICPR 2002. Vol. II, pp. 697-700. (c) IEEE. Quebec, Canada, August 2002. 2002.

► E. Marchand, F. Chaumette. E. Marchand, F. Chaumette. Virtual Visual Servoing: a framework for real-time Virtual Visual Servoing: a framework for real-time augmented realityaugmented reality. In . In EUROGRAPHICS 2002 Conference ProceedingEUROGRAPHICS 2002 Conference Proceeding, G. Drettakis, , G. Drettakis, H.-P. Seidel (eds.), Computer Graphics Forum, Volume 21(3), Pages 289-298, H.-P. Seidel (eds.), Computer Graphics Forum, Volume 21(3), Pages 289-298, Sarrebruck, Germany, September 2002.Sarrebruck, Germany, September 2002.

► Simon Baker and Iain Matthews. Simon Baker and Iain Matthews. Lucas-Kanade 20 Years On: A Unifying FrameworkLucas-Kanade 20 Years On: A Unifying Framework. . IInternational Journal of Computer Visionnternational Journal of Computer Vision, Vol. 56, No. 3, March, 2004, pp. 221 - 25 , Vol. 56, No. 3, March, 2004, pp. 221 - 25

Registration for Augmented Reality Neil Birkbeck 3/27/2006.

Documents

Transcript of Registration for Augmented Reality Neil Birkbeck 3/27/2006.