Embedded Media Markers: Marks on Paper that Signify Associated Media
Qiong Liu, Chunyuan Liao, Lynn Wilcox, Anthony Dunnigan, Bee Liew
FX Palo Alto Laboratory, Palo Alto, CA, USA
Motivation & Vision• Motivation: Combine advantages of
paper and cell phones
• Vision: Use Embedded Media Markers (EMMs) to link multimedia to paper patch •EMMs are meaningful marks on paper•Use visual features inside EMM to identify document patch
Slide 2
User takes a picture of a document patch
Multimedia and digital content displayed on cell phone
What is EMM• Embedded Media Markers (EMM) are optical-filter-like overlaid marks printed on paper documents.
• Printed marks (EMMs) that improve usability and recognition accuracy
• Computationally efficient features (FIT) that identify document patches
Slide 3
EMM-signified document patch
EMM-signified document location
Media type (video)
Slide 4
EMM Patch Identification•Patch identification is based on image local feature (FIT) • Similar to SIFT features. More efficient.• Robust to scaling, rotation and partial occlusion
•FIT features work on a broad range of document types• Western/Asian text, images, photos
EMM Operation Flowchart
Take a snapshotof EMM
Match feature points against the document database
Compute feature points
Find the matched document patch
Retrieve the linked videoPlay the video on
the phone
• Marker-based Technologies• Recognizable Visual Markers (e.g.
Barcode, QR-Code)• Nearly Invisible Fiduciary Markers (e.g.
DataGlyphs, Anoto)
• Content-based Technologies• Text-words-based Features (e.g. BWC)• Image-based Features (e.g. SIFT, FIT)
• EMM is a hybrid approach• Provide users multimedia cues and
interaction guidance• Save computation and
improve performance• Minimize the interference with
document content
Document Identification Methods
Barcode QR-code
Data Glyphs
Anoto Pattern
BWC
EMM
SIFT
EMM Design Guidelines• User Friendly – guide
users for media finding and query.
• Document Friendly –minimize changes to document content and format.
• Machine Friendly – save computation resources for document identification process.Major Entities
EMM Components’ Functions
• Feature boundary mark – framing feature capture
• Media type mark
• Anchor point – associate media with an accurate location.
• Graphical effects – attract attention and help separate content and marker.
feature-boundary-circle
media-type-icon
Anchor point
Top View of a 3-level EMM Model
•Semi-automatically arrange EMM based on feature distribution• Criterion 1 – Minimize author’s effort.
Author only selects an EMM anchor point.
• Criterion 2 – Minimize machine resources used for patch query. Find a small feature boundary, and index the small number of keypoints inside the feature boundary.
• Criterion 3 – Minimize the EMM interference to document. Minimize the number of keypoints overlaid by an EMM.
EMM Authoring Tool
Parameters for EMM Placement
•Anchor point location (m,n) – provided by a user
•Feature-boundary-circle center (X,Y)
•Feature-boundary-circle radius R
•Media-type-circle center (x,y)
Parameter Optimization• EMM boundary optimization• Find the center (X,Y) and the minimum radius R such that
• The number of keypoints in the circle exceeds the threshold to ensure patch identification accuracy
• The user selected anchor point is contained in the boundary
• Media type icon placement• Select the media-type-icon center (x,y) so that the icon covers the minimum number of keypoints
•Counting the number of keypoints in a circle is time consuming • O(2N). N is the number of
keypoints in a page.•Speedup
• Inscribed square to estimate the number of keypoint in a circle
• Use the cumulative feature-point histogram
• IABCD =IC-IB-ID+IA
•Optimization of media type circle is similar
Parameter Optimization Speedup
A B
CD
A
C
B
D
IPP
Boundary Circle
EMM Evaluation in Real Life
• 2188 EMMs in the database• At randomly selected 2188 document locations of2188
pages• 110 EMM Overlaid Page Printouts
• 90 1-up + 20 4-up on letter-size paper• 110 Queries
• Either in an office or open patio w/natural lighting conditions
The Missed One• Include many less
distinctive keypoints
• Correctly identified after position adjustment
Slide 15
Related Work
Ricoh “HotPaper”Link personal media to text.Use BWC (Brick Wall Coding)Only works on Western textACM Multimedia 08
University of Oldenberg “Bookmarkr”Link photobook picture to digital photo.Use SIFT featuresACM Multimedia 08
Document ID Technology Comparison
Marker-based Technologies Content-based Technologies Hybrid Technology
TypeVisual Markers
Fiduciary Markers
Text-based Features
Image-based Features
EMMs
ExampleBar Code, QR Code, SP Code
Dataglyph, Anoto, MIST, MIG
Ricoh BWC SIFT, FITMarker overlay plus FIT
Capture Guidance
Good (no indication of media type)
NoneFair (Scan to find patches with media)
None
Excellent (shows existence, accurate association location, and type of media)
MachineReadability
Excellent ExcellentFair (depends on content)
Fair (depends on content)
Good (optimized for content)
DocumentFidelity
Poor (visually obtrusive marks need extra space)
Fair (slight modifications of document appearance)
Excellent ExcellentGood (nearly transparent and attractive markers)
Technology Comparison - cont
Marker-based TechnologiesContent-based Technologies
Hybrid Technology
Type Visual MarkersFiduciary Markers
Text-based Features
Image-based Features
EMMs
ExampleBar Code, QR Code, SP Code
Dataglyph, Anoto, MIST, MIG
Ricoh BWC SIFT, FITMarker overlay plus FIT
Document Restrictions
None NoneYes (Western Text only)
Some (depends on content)
Some (depends on content)
Marking Resolution
Low (Page)High (Patch, Pixel-level)
Medium (Block of ~100 words)
High (Patch, Pixel)
High (Patch, Pixel)
Scan/print Restrictions
None* (Regular printer and camera)
Yes (High-res printer and scanner)
None (Regular printer and camera)
None (Regular printer and camera)
None (Regular printer and camera)
Accuracy Control
Full Full None None Partial
* SP-code requires high-res printer and scanner
Summary
• EMMs are nearly transparent iconic marks printed on paper document to link various multimedia to specific document locations
• EMMs are robust to scaling, rotation and partial occlusion, and applicable to a broad range of document types
• Unlike barcodes, EMMs do not change document layouts and are less visually obtrusive
• Unlike other feature-based approaches, EMMs explicitly signify the media-augmented document regions and guide user interaction
• EMM can improve the efficiency of computation
Thank You
Top Related