Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous...
Transcript of Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous...
![Page 1: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/1.jpg)
1
Context-aware Intelligent Assistant
Ubiquitous Computing Seminar Sabir Akhadov 2015
![Page 2: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/2.jpg)
2
Assistance
Ubiquitous Computing Seminar Sabir Akhadov 2015
Servant Valet Secretary
Introduction
![Page 3: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/3.jpg)
3
Intelligent Assistant
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
[15]
Siri S Voice
![Page 4: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/4.jpg)
4
Intelligent Assisting Devices
Ubiquitous Computing Seminar Sabir Akhadov 2015
Amazon Echo WIMM One
Introduction
[16]
![Page 5: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/5.jpg)
5
AR Intelligent Assisting Devices
Ubiquitous Computing Seminar Sabir Akhadov 2015
Google Glass
CastAR
Moverio BT-200
Introduction
[15], [16], [17]
![Page 6: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/6.jpg)
6
AR Intelligent Assisting Devices
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
![Page 7: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/7.jpg)
7
Privacy concerns
Ubiquitous Computing Seminar Sabir Akhadov 2015
● What is recorded?
● Where does the data go?
● What is done with it?
● Who can see it?
Introduction
[18]
![Page 8: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/8.jpg)
8
Context
Ubiquitous Computing Seminar Sabir Akhadov 2015
What resources are nearby
Where you are
Who you are with
Introduction
[4], [6]
![Page 9: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/9.jpg)
9
Context-aware Intelligent Assistant
● Recognize current context of the situation
● Use context to help with a task
● Make predictions about the future
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
![Page 10: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/10.jpg)
10
Road map
Ubiquitous Computing Seminar Sabir Akhadov 2015
Tracking of manual workflows Guiding Block Assembly Situation Awareness for In-CarRecommendations
Introduction
enactive proactive
![Page 11: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/11.jpg)
11
Real-time Modeling and Tracking Manual Workflowsfrom First-Person Vision
Ubiquitous Computing Seminar Sabir Akhadov 2015
German Research Center for Artificial Intelligence (DFKI)
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
[1]
![Page 12: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/12.jpg)
12
Goal
● Augmented Reality manuals
● Follow the progress of a user
● Show the next steps
● Indicate errors
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
![Page 13: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/13.jpg)
13
Ikea Catalog
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
See furniture installedin your apartment
![Page 14: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/14.jpg)
14
Ikea Manual
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
![Page 15: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/15.jpg)
15
AR manual
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
![Page 16: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/16.jpg)
16
Approach
● Record with head-worn camera
● Analyze video sequences
● Record hands position during training
● Show the hand position for enactive feedback*
● Provide optical validation*enactive feedback – there is an ongoing interaction between user action and the system
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
![Page 17: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/17.jpg)
17
Process diagram
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
Record Workflow Camera trackingSequence into
subtasks
Estimate hand position
Hand position map
Project hand map on user view field
![Page 18: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/18.jpg)
18
Camera tracking
● Initialization:
– Select arbitrary corner features.
– Find correspondences in the next frame with KLT
– Use RANSAC to find the largest subset P which can be described using a homography H
– The subset P is used at a later step
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
![Page 19: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/19.jpg)
19
RANSAC
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
LS
RANSAC
[5], [8]
![Page 20: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/20.jpg)
20
Camera tracking
● Work step
– P as input from previous frame
– Use KLT and RANSAC to find next H
– Find new corner features across the entire image
– Find correspondences in the next frame and reject
points that cannot be described with H
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
![Page 21: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/21.jpg)
21
Task segmentation
● Image difference function:
● Strong camera movement:– Translation
– Rotation
Ubiquitous Computing Seminar Sabir Akhadov 2015
d ( I 1 , I 2)→ R
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
![Page 22: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/22.jpg)
22
Hand position
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
![Page 23: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/23.jpg)
23
Application
Enactive feedback
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
![Page 24: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/24.jpg)
24
Application
Optical validation
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
![Page 25: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/25.jpg)
25
Conclusion
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
● Depth inferred from video sequences
● This approach works only in static environment
● Different lighting may be a problem
![Page 26: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/26.jpg)
26
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
[2]
![Page 27: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/27.jpg)
27
Goal
● Produce assembly manuals
● Track user actions
● Show next step
● Detect mistakes
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 28: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/28.jpg)
28
Traditional manuals
Figure-based:– Difficult to
create
– No motion
cues
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 29: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/29.jpg)
29
Traditional manuals
Video based
+ Pause, repeat
+ Motion cues
– Possibly different
view point
– Still no feedback
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 30: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/30.jpg)
30
Augmented Reality
● Expensive equipment
needed● Static models● Motion cues from
head motion only
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
[19]
![Page 31: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/31.jpg)
31Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Overview
Authoring Guiding
![Page 32: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/32.jpg)
32
DuploTrack Demonstration
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 33: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/33.jpg)
33
Processing pipeline
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 34: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/34.jpg)
34
Tracking the model
● User may rotate and shift model in the Play area
● Lose of tracking if turned upside down
● The point cloud is aligned with virtual model– Iterative Closest Point (ICP)
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 35: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/35.jpg)
35
Model tracking at the beginning
● Poor tracking for models with under 5 blocks
– Noise from Kinect
– Outliers overwhelm the points before an update
● Solution: Place the model on the table before reaching 5 blocks
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 36: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/36.jpg)
36
User study
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Baseline Track
Test guidance system
![Page 37: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/37.jpg)
37
User study, Two Tasks
Single block and multiple blocks addition
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 38: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/38.jpg)
38
Results, one block
● 21.8 seconds for Baseline
● 18.9 seconds for Track
● 14% of improvement
● 3 mistakes with Baseline ● 0 with Track
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 39: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/39.jpg)
39
Results, one block
Ubiquitous Computing Seminar Sabir Akhadov 2015
speedup=100∗(Time (Baseline)Time (Track )
−1)
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 40: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/40.jpg)
40
Results, multiple blocks
Times to add blocks one after the other
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 41: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/41.jpg)
41
Results, multiple blocks
● Model A
● 11.6s Track
● 17.3s Baseline
● 7 mistakes Baseline
● 0 mistakes TrackUbiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 42: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/42.jpg)
42
Results, multiple blocks
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 43: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/43.jpg)
43
Results, multiple blocks
● Model B
● 10.03s for Track
● 10.22s for Baseline
● 0 mistakes for both interfaces
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 44: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/44.jpg)
44
Results, multiple blocks
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 45: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/45.jpg)
45
Results, qualitative feedback
● 11 of 16 participants preferred Track
● 3 participants preferred Baseline
● All said Track was more enjoyable
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 46: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/46.jpg)
46
Results summary
● Track increased on average the speed and accuracy
● For some participants negative speedup
● For more complicated models the results may be even better
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 47: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/47.jpg)
47
Future work
● Extend system to handle smaller blocks
● Different shapes
● Furniture assembly
● Home repairs
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
![Page 48: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/48.jpg)
48
Situation Awareness for Proactive In-CarRecommendations of Points-Of-Interest
● Recommendation system for in-car context● Fuel stations, restaurants, parking lots
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI)
[3]
![Page 49: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/49.jpg)
49
Driver's attention
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI)
![Page 50: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/50.jpg)
50
Proactive system
User above the interaction loop
Resource: Driver's attention
Focus: Relevance of information - The right information at the right time to the right user
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI)
![Page 51: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/51.jpg)
51
Three levels of situation awareness
● Level 1 context sensing
● Level 2 situation comprehension
● Level 3 projection into the future
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI)
![Page 52: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/52.jpg)
52
Model for Situation Awareness in Proactive Systems
● Fuzzy logic values between 0..1
– Certainty expression
– No abrupt behavior
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI)
[7], [20]
![Page 53: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/53.jpg)
53Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI)
Model for Situation Awareness in Proactive Systems
![Page 54: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/54.jpg)
54Ubiquitous Computing Seminar Sabir Akhadov 2015
IF fuel_level == empty THEN relevance_fueling = high
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI)
Model for Situation Awareness in Proactive Systems
![Page 55: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/55.jpg)
55Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI)
Model for Situation Awareness in Proactive Systems
![Page 56: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/56.jpg)
56Ubiquitous Computing Seminar Sabir Akhadov 2015
IF fuel_level == empty AND distance == close
THEN relevance_fueling = high
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI)
Model for Situation Awareness in Proactive Systems
![Page 57: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/57.jpg)
57
Evaluation
Ubiquitous Computing Seminar Sabir Akhadov 2015
● Fuel level and station reachability
● Connection fuzzy variable
● Low fuel stations coverage area
● 40 liters gas, 20 reachable stations
● 800km
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI)
![Page 58: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/58.jpg)
58
Evaluation
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI)
![Page 59: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/59.jpg)
59
Future work
Ubiquitous Computing Seminar Sabir Akhadov 2015
● More complex scenarios
● User study
● Comparison to other models
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI)
![Page 60: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/60.jpg)
60
Summary
● Assistant
Helps with some task, trust issues
● Intelligent assistant
Privacy issues
● Context
Characterize a situation
Ubiquitous Computing Seminar Sabir Akhadov 2015
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI)
Summary
![Page 61: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/61.jpg)
61
Summary
Ubiquitous Computing Seminar Sabir Akhadov 2015
Tracking of manual workflows Guiding Block Assembly Situation Awareness for In-CarRecommendations
Guiding systems Recommender system
Introduction
Real-time Modeling and Tracking Manual Workflows from First-Person Vision
DuploTrack: A Real-time System for Authoring and Guiding Duplo Block Assembly
Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI)
Summary
![Page 62: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/62.jpg)
62
References
Papers:● [1] Petersen, Nils, Alain Pagani, and Didier Stricker. "Real-time Modeling and Tracking Manual Workflows from First-
person Vision." 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (2013): n. pag. Web.
● [2] Gupta, Ankit, Dieter Fox, Brian Curless, and Michael Cohen. "DuploTrack." Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology - UIST '12 (2012): n. pag. Web.
● [3] Roland Bader, Wolfgang Woerndl, Vivian Prinz: Situation Awareness for Proactive In-Car Recommendations of Points-Of-Interest (POI). Proc. Workshop Context Aware Intelligent Assistance (CAIA 2010), 33rd Annual German Conference on Artificial Intelligence (KI 2010), Karlsruhe, Germany, Sep. 2010
● [4] Dey, A.K., Abowd, G.D.: Towards a Better Understanding of Context and Context-
Awareness. In: 1st international symposium on Handheld and Ubiquitous Comput-
ing. pp. 304 – 307. Georgia Institute of Technology, Georgia (1999)
● [5] Martin A. Fischler and Robert C. Bolles (June 1981). "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography" (PDF). Comm. of the ACM 24 (6): 381–395.
● [6] A. Zimmermann, A. Lorenz, and R. Oppermann, “An operational definition of context,” in CONTEXT’07: Proceedings of the 6th international and interdisciplinary conference on Modeling and using context, (Berlin, Heidelberg), pp. 558–571, Springer-Verlag, 2007.
Ubiquitous Computing Seminar Sabir Akhadov 2015
![Page 63: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/63.jpg)
63
References(2)
Articles:● [7] Fuzzy Math http://blog.peltarion.com/2006/10/25/fuzzy-math-part-1-the-theory/ (12.05.2015)
● [8] Overview of the RANSAC Algorithm http://www.cse.yorku.ca/~kosta/CompVis_Notes/ransac.pdf (12.05.2015)
● [9] How private is Amazon Echo? http://www.slashgear.com/how-private-is-amazon-echo-07354486/ (12.05.2015)
Images:● [13] 8 hilarious IKEA memes, http://blog.ted.com/8-ikea-memes-and-homages/, (12.05.2015)
● [14] Siri-Chan: http://ikiwed.deviantart.com/art/Siri-Siri-chan-266625034 (12.05.2015)
● [15] Google Now: http://commons.wikimedia.org/wiki/File:GoogleNow_logo.jpg (12.05.2015)
● [16] "Amazon Echo" by Frmorrison - Take a picture. Licensed under CC BY-SA 3.0 via Wikipedia - http://en.wikipedia.org/wiki/File:Amazon_Echo.jpg#/media/File:Amazon_Echo.jpg (12.05.2015)
● [17] CastAR: http://media.bestofmicro.com/N/M/433714/original/castar3.png (12.05.2015)
● [18] Smartphone: http://pixabay.com/static/uploads/photo/2013/07/12/18/39/smartphone-153650_640.png (12.05.2015)
Ubiquitous Computing Seminar Sabir Akhadov 2015
![Page 64: Context-aware Intelligent Assistant - ETH Z · 1 Context-aware Intelligent Assistant Ubiquitous Computing Seminar Sabir Akhadov 2015](https://reader031.fdocuments.net/reader031/viewer/2022021513/5b19ba5c7f8b9a41258ce9f5/html5/thumbnails/64.jpg)
64
References(3)
Images:● [19] New Display Technology:
http://i.livescience.com/images/i/000/018/892/original/IIP-47_0848519_Sun_Innovations_Inc.jpg?1313013442 (12.05.2015)
● [20] Fuzzy Logic: http://upload.wikimedia.org/wikipedia/commons/1/1b/Fuzzy_control_-_definition_of_input_temperature_states_using_membership_functions.png (12.05.2015)
Ubiquitous Computing Seminar Sabir Akhadov 2015