Post on 22-Feb-2016
description
Supporting Beyond-surface Interaction for Tabletop Systems by Integrating IR Projections
Hui-Shan Kao
2
Outline Introduction Related Work System Design System Calibration Beyond Surface Implementation Working with multi-touch Application Conclusion
3
Outline Introduction Related Work System Design System Calibration Beyond Surface Implementation Working with multi-touch Application Conclusion
4
Introduction Regular interactive surface only support
Multi-Touch Tangible input
Extend more possibility on interactive surface Add another display Enable intuitive 3D manipulation
Introduction5
Beyond-Surface Interactions Base on regular interactive surface Embedded IR markers
Invisible 6DOF of IR camera
pico projector Tablet PC
IR Projecto
r
Location & Orientation
6
Outline Introduction Related Work
Beyond-surface Interaction Localization of Device Invisible Projection
System Design System Calibration Beyond Surface Implementation Working with multi-touch Application Conclusion
7
Beyond-surface Interaction Second Light
Use an electronically switchable diffuser Turn any translucent sheet above the
surface into a mobile display via the second projection
With the camera that sees through the surface, it can localize a mobile panel in six-degrees.
Izadi, S.etc Going beyond the display: a surface technology with an electronically switchable diffuser. In Proc UIST’08
8
Localization of Device Enable 3D interaction on tabletop display
Need to recognize the 6DOF of device The way to know 6DOF
Magnetic tracker Penlight
Vision based tracker Handheld projector JanusVF
Marker based trackerVisible
9
Invisible projection Invisible projection
Spectrum IR/Color polarization
Time high frequency
Synchronization of camera and projector Encoding in content
Embedded code in color channel
10
Invisible projection Hybrid Infrared and Visible Light Projection for
Location Tracking A projector capable of projecting visible images and infrared images Using gray-coded pattern to locate the sensors.
Dynamic adaptation of projected imperceptible codes Using high frequent temporal image modulation to
project an invisible pattern
Johnny Lee, etc. Hybrid Infrared and Visible Light Projection for Location Tracking In Proc UIST’07 A. Grundh¨ofer, etc.“Dynamic adaptation of projected imperceptible codes,” In Proc. ISMAR ’07
11
Outline Introduction Related Work System Design
Hardware Configuration Markers System
System Calibration Beyond Surface Implementation Working with multi-touch Application Conclusion
12
System Design Goal:
Support multi-touch and multi-user DI based touch detection
Add beyond surface interaction Using invisible marker
13
Hardware Configuration
IR ProjectionColor Projection
IR Projector
IR Camera IR
Camera
Color Projector
Mirror
Pico projector + IR Camera
Tablet PC + IR Camera
14
Hardware Design The order of glass layer and diffuse layer
diffuse layer should on top Not to degrade the luminance of pico projector The reflection of pico projector may offending
the user
15
Hardware Configuration Problem:
IR rays will be reflected by the touch-glass and resulting in IR spot regions in camera views
Use two cameras to reduce the IR spot
16
Markers System ARToolkitPlus
Fiducial marker Self-identify
Enable error correct bit Localization
Camera and projector calibration Camera pose estimation
17
Outline Introduction Related Work System Design System Calibration
Tabletop System Calibration Projector and Camera System Calibration
Beyond Surface Implementation Working with multi-touch Application Conclusion
18
Tabletop System Calibration Original tabletop system calibration
Finding the homographs between table, camera, and projector
The corners are manually specified by the users Time-consuming Human intervention Pixel-level accuracy
Tabletop System Calibration19
Only need four points to be manually specified Adding an additional IR-Color camera Project predefined markers for calibration
IR Camera
IR Projector
Color Projector
IR-Color Camera
20
Pico projector and camera calibration
The projector as an inverse camera Mapping pixel from a 2D image into 3D rays Using standard camera calibration procedure
Find the 3D points of the projected pattern and the 2D points of the image projected
Rc , Tc
Rp , Tp
Rcp , Tcp
21
Outline Introduction Related Work System Design System Calibration Beyond Surface Implementation
3D posture estimation Multi-Resolution Markers
Working with multi-touch Application Conclusion
22
Beyond Surface Implementation
3D posture?
23
3D Posture Estimation ARToolKitPlus for 3D estimation
Camera looks within image for markers Encode identity Allow recovery of camera pose relative to
marker
24
Multi-Resolution Markers Uni-resolution marker
Camera could observe the markers too small or big The marker with unfit size will not be
recognized
Multi-resolution marker System resizes the IR markers according to
the camera’s posture
25
Marker Split and Merge Marker Split
Not enough : split the markers into smaller size Marker Merge
Too much : merge the markers for higher accuracy
How to re-arrange the layout of IR makers? Ensure that camera will see at least 4 markers Only re-arrange the layout in camera’s view field The nearest camera will have high priority
27
Outline Introduction Related Work System Design System Calibration Beyond Surface Implementation Working with multi-touch
Foreground detection Background Simulation Software Synchronization
Application Conclusion
28
Working with multi-touch Traditional DI process
Take few frames for building background subtract the background obtain the foreground
IR markers projection will interrupt the traditional detection of multi-touch Foreground can not be recognized
3131
IR Projecto
r
IR Cameras
IR Camera of Mobile Device
Simulated Background
Observed Image
Foregrounds
Layout Manager
Color Project
or
ROI
Real scene
ROI Generation
34
IR Projecto
r
IR Cameras
IR Camera of Mobile Device
Smoothing
Simulated Background
Observed Image
Foregrounds
Tangible Objects
Finger Touches
Layout Manager
Prediction
Color Project
or
ROI
Applications
KalmanFiltering
Real scene
Foreground Detection
35
Capture Image
Simulate BG
Real BG
Software Synchronization The camera and projectors are two
independent systems The simulated background will not
synchronize to the capture image Some of the markers will be treat as
foreground
36
Software Synchronization Keep the simulated backgrounds in a
buffer by time Find the most similar background by
subtraction
Background Candidate queue
Real BG
Capture Image
BG Candidate
Real BG
37
Outline Introduction Related Work System Design System Calibration Beyond Surface Implementation Working with multi-touch Application Conclusion
38
Application 3 types of the application provide
intuitive and natural manipulation.
iLamp
iFlash
iView
iLamp
Goal: Project high-resolution content, bring
more detailed and fine-grained information
Combine a Pico projector and an IR camera
iFlashlight A mobile version of iLamp, can be
moved easily.
iView Tablet PC + IR Camera An intuitive tool to see 3D content or
augmented information of the 2D map from different perspectives.
42
iView Problem in iView
The user will lose the connection with the surface.
Adding the boundary of surface Instruct user to manipulate the surface for
farther information.
43
Outline Introduction Related Work System Design System Calibration Beyond Surface Implementation Working with multi-touch Application Conclusion
44
Conclusion A new interactive surface based on the
programmable invisible markers. Supporting both on-surface and above-
surface interaction for any device outfitted with an IR camera.
Bring another level of information on interactive surface.
45