Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of...
-
Upload
camilla-scott -
Category
Documents
-
view
218 -
download
1
Transcript of Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of...
![Page 1: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/1.jpg)
Making Sense of Sensors
Henry KautzDepartment of Computer Science & EngineeringUniversity of Washington, Seattle, WA
Funding for this research is provided in part by IISI and AFRL/IF
![Page 2: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/2.jpg)
Making Sense of Sensors
or … Climbing the Data Interpretation Food-Chain
![Page 3: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/3.jpg)
The Ubiquitous Future
Rapidly declining size and cost of sensing and networking technology makes it practical to rapidly deploy systems that monitor large environments in great detail– factories, airports, hospitals, homes– oceanic regions, cities, countryside
Problem: it is easier to collect data than make to sense of it!
![Page 4: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/4.jpg)
Data Fusion
Traditional work in data-fusion attacks problem of recovering specific physical phenomena from the readings of homogeneous networks of noisy sensorsE.g.: given readings from underwater microphone array, determine the position of a submarine
![Page 5: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/5.jpg)
Current Trends
Heterogeneous sensors– Instrumented environment: motion detectors, weight
detectors, video, audio, …– Instrumented personnel: smart badges, GPS phones,
metabolic sensors. …
Goal: high-level understanding– What actions are being performed?– What are the goals of the subjects?– Do we need to intervene?
![Page 6: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/6.jpg)
Example: Security
System monitors activity in a post officeTracks common tasks performed by individuals– Mailing packages– Getting mail from PO boxes– Buying stamps
Alerts operator when abnormalities noted– Person leaves package on floor and exits– Loitering (but not waiting in line!)
![Page 7: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/7.jpg)
Example: Guiding
Activity Compass: GPS system that– Learns daily patterns of travel – Understands walking, car, bus, bike– Integrates external information
• Real-time bus data
Predicts problems– Will user miss appointment?– Is user on the wrong bus?
Offer proactive help– E.g., suggest alternative travel plan
![Page 8: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/8.jpg)
Triple-Use Technology
Plan-AwareComputing
Military
surveillanceaugmented cognition
CommercialSoftware
intelligent user interfaces
PatientCare
aging in placeassisted cognition
![Page 9: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/9.jpg)
Key Issue
How to go from noisy and incomplete sensor measurements toA meaningful description of what a person is doing
• “Waiting to mail package”• “Trying to get home”
A decision by the system about whether or not to intervene … in a principled and scalable manner!
![Page 10: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/10.jpg)
Data Interpretation Food Chain
Movement
Intentions
Behavior
Interventions
![Page 11: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/11.jpg)
Model-Based Interpretation
General approach: build a probabilistic model of– Common user goals– Plans (complex behaviors) that achieve those goals
• Feasibility constraints • Temporal constraints• Failure (abnormality) modes
– How simple behaviors are sensed
Run model “backwards” to interpret sensed data
![Page 12: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/12.jpg)
Million-Mile View
In principal we know how to estimate the state of the system under observation:
To make this practical, we must take advantage of the regular structure of the domain
1 1 1Bel( ) Pr( | ) Pr( | ) Bel( )t t t t t t tx z x x x x dx state at time t
observation at time t
system dynamics
![Page 13: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/13.jpg)
Technical Foundations
Hidden Markov models– Mathematical framework for describing processes
with hidden state that must be inferred from observations
Hierarchical plan networks– Represents how a task can be broken down into
subtasksHierarchical hidden Markov models*– Key to climbing food-chain!
* Precisely speaking: factorial hierarchical hidden semi-Markov models
![Page 14: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/14.jpg)
Video Door Sensor Motion
Location
Example
Enter PO
Wait in line
Let go package
Pay cashier
Exit PO
Mail Package
![Page 15: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/15.jpg)
Video Door Sensor Motion
Location
Enter PO
Go to PO
boxes
Open PO box
Pick up mail
Exit PO
Retrieve Mail
Example
![Page 16: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/16.jpg)
Video Door Sensor Motion
Location
Mail Package
PO Patron
Retrieve Mail
Outside PO
Example
![Page 17: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/17.jpg)
Inexplicable Observations
Enter PO
Wait in line
Let go package
Pay cashier
Exit PO
Mail Package
Enter PO
Go to PO
boxes
Open PO box
Pick up mail
Exit PO
Retrieve Mail
Enter PO
Let gopackage
Exit PO
![Page 18: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/18.jpg)
Absolute Timing Constraints
Mail Package active [9 am – 4 pm]
Enter PO
Retrieve Mail active [6 am – 8 pm]
Enter PO
![Page 19: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/19.jpg)
Relative Timing Constraints
Go to PO
boxes
Open PO box
Retrieve Mail
Timeout
seconds
seconds
Forgot combo?Safecracking?
![Page 20: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/20.jpg)
Summary
Commonsense knowledge base of “significant” behaviors– Hierarchically organized– Probabilistic at all levels– Many parallel ongoing activities possible– Absolute and relative timing constraints– Probabilities “tuned” by machine learning techniques for
individual users– Inexplicable observations and failure modes – points of
possible intervention
![Page 21: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/21.jpg)
Interventions
Framework allows system to predict when an anomalous situation is likelyDifferent anomalies have different costs– Confused patron– Deliberate loitering– Planting bomb
Must avoid:
![Page 22: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/22.jpg)
Deciding When to Intervene
(Horvitz 98)G = prediction that help is needed
![Page 23: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/23.jpg)
Common Architecture
![Page 24: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/24.jpg)
![Page 25: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/25.jpg)
Activity Compass
Palm-based wireless GPS– No explicit programming – learns pattern of
transportation plans – Accesses user’s calendar, real-time bus information– Constantly tries to predict where user will go next, and
whether problems will arise– Proactive help:
• “Walk faster or you’ll miss the 9:15 bus!”• “Green St bus is late, suggest you take Elm St bus instead”
![Page 26: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/26.jpg)
Substeps
Cleaning up GPS data– 3 meter accuracy– frequent signal loss– determine most likely path
Infer mode of transportationPredict when and where transitions in mode of travel will occurPredict points of possible failure
indoors
walk
bus
bikecar
![Page 27: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/27.jpg)
Gathering Data
![Page 28: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/28.jpg)
On Foot: Across Campus
![Page 29: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/29.jpg)
By Bus: Across Seattle
![Page 30: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/30.jpg)
Transition Prediction
Training Data:– 20,000 GPS readings gathered over 3 weeks
Inferring current mode– Input: current location, time, velocity– 98% accuracy (10 FCV)
Predicting next transition– Input: current mode, location, time, velocity– 97% accuracy (10 FCV)*
* Don is a very organized guy. Your accuracy may vary.
![Page 31: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/31.jpg)
Predicting Transition Location
![Page 32: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/32.jpg)
User Interface
![Page 33: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/33.jpg)
Assisted Cognition
“Plan aware” systems to help people with cognitive disabilitiesNew project based at University of Washington – Computer Science & Engineering– UW Medical Center, ADRC– Collaborators: Intel, OGI, Elite Care
http://assistcog.cs.washington.edu/
![Page 34: Making Sense of Sensors Henry Kautz Department of Computer Science & Engineering University of Washington, Seattle, WA Funding for this research is provided.](https://reader036.fdocuments.net/reader036/viewer/2022062517/56649e7f5503460f94b82bb3/html5/thumbnails/34.jpg)
Summary
Potential of widespread sensor networks just beginning to be tappedKey issue: interpreting data in terms of human behavior, plans, and goalsResearchers in data fusion, AI, and “ubicomp” coming together around a core set of representations and algorithms