Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T....
-
Upload
austen-offord -
Category
Documents
-
view
215 -
download
0
Transcript of Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T....
![Page 1: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/1.jpg)
Darwin Phones: the Evolution of Sensing and Inference on
Mobile Phones
Emiliano Miluzzo*, Cory T. Cornelius*, Ashwin Ramaswamy*, Tanzeem Choudhury*, Zhigang Liu**,
Andrew T. Campbell*
* CS Department – Dartmouth College** Nokia Research Center – Palo Alto
![Page 2: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/2.jpg)
[email protected] Miluzzo
![Page 5: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/5.jpg)
[email protected] Miluzzo
![Page 6: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/6.jpg)
[email protected] Miluzzo
![Page 7: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/7.jpg)
[email protected] Miluzzo
![Page 8: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/8.jpg)
[email protected] Miluzzo
![Page 9: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/9.jpg)
[email protected] Miluzzo
![Page 12: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/12.jpg)
[email protected] Miluzzo
accelerometer
digital compass
microphone
WiFi/bluetooth GPS
….
light sensor/camera
sensing
![Page 13: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/13.jpg)
[email protected] Miluzzo
accelerometer
digital compass
microphone
WiFi/bluetooth GPS
light sensor/camera
gyroscope
air quality /pollution sensor
sensing….
![Page 15: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/15.jpg)
[email protected] Miluzzo
- 600 MHz CPU
- up to 1GB application memory
hardware
computation capability is increasing
![Page 17: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/17.jpg)
[email protected] Miluzzo
application distribution
deploy apps onto millions of phones at
the blink of an eye
![Page 18: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/18.jpg)
[email protected] Miluzzo
application distribution
collect huge amount of data for research
purposes
deploy apps onto millions of phones at
the blink of an eye
![Page 21: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/21.jpg)
[email protected] Miluzzo
cloud infrastructure
cloud - backend support
we want to push intelligence to the
phone
![Page 22: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/22.jpg)
[email protected] Miluzzo
cloud infrastructure
cloud - backend support
preserve the phone user experience
(battery lifetime, ability to make calls, etc.)
![Page 23: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/23.jpg)
[email protected] Miluzzo
cloud infrastructure
cloud - backend support
- sensing
- run machine learning algorithms locally
(feature extraction + inference)
![Page 24: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/24.jpg)
[email protected] Miluzzo
cloud infrastructure
cloud - backend support
- sensing
- run machine learning algorithms locally
(feature extraction + inference)
run machine learningalgorithms (learning)
![Page 25: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/25.jpg)
[email protected] Miluzzo
cloud infrastructure
cloud - backend support
store and crunch big data(fusion)
run machine learningalgorithms (learning)
- sensing
- run machine learning algorithms locally
(feature extraction + inference)
![Page 26: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/26.jpg)
[email protected] Miluzzo
cloud infrastructure
cloud - backend support
run machine learningalgorithms (learning)
store and crunch big data(fusion)
3 to 5 years from now our phones will be as powerful as a - sensing
- run machine learning algorithms locally
(feature extraction + inference)
![Page 27: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/27.jpg)
[email protected] Miluzzo
cloud infrastructure
cloud - backend support
run machine learningalgorithms (learning)
store and crunch big data(fusion)
3 to 5 years from now our phones will be as powerful as a - sensing
- run machine learning algorithms locally
(feature extraction + inference)
![Page 28: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/28.jpg)
[email protected] Miluzzo
cloud infrastructure
cloud - backend support
run machine learningalgorithms (learning)
store and crunch big data(fusion)
3 to 5 years from now our phones will be as powerful as a - sensing
- run machine learning algorithms locally
(feature extraction + inference)
![Page 29: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/29.jpg)
[email protected] Miluzzo
cloud infrastructure
cloud - backend support
- Sensing
- run machine learning algorithms locally
(feature extraction + learning + inference)
run machine learningalgorithms (learning)
store and crunch big data(fusion)
3 to 5 years from now our phones will be as powerful as a
![Page 32: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/32.jpg)
[email protected] Miluzzo
societal scale sensing
global mobile sensor network
reality mining using mobile phones
will play a big role in the future
![Page 36: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/36.jpg)
[email protected] Miluzzo
evolution of sensing and inferenceon mobile phones
from motes to mobile phones
![Page 37: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/37.jpg)
[email protected] Miluzzo
evolution of sensing and inferenceon mobile phones
from motes to mobile phones
darwin
- classification model evolution
- classification model pooling
- collaborative inference
![Page 38: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/38.jpg)
[email protected] Miluzzo
microphone
camera
GPS/WiFi/cellular
air quality pollution
sensing apps
social context
audio / pollution / RF fingerprinting
image / video manipulation
darwin applies distributed computing and collaborative inference concepts to
mobile sensing systems
darwin
- classification model evolution
- classification model pooling
- collaborative inference
![Page 40: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/40.jpg)
why darwin?
[email protected] Miluzzo
train classification model X in the lab
mobile phone sensing today
![Page 41: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/41.jpg)
why darwin?
[email protected] Miluzzo
deploy classifier X
mobile phone sensing today
train classification model X in the lab
![Page 42: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/42.jpg)
why darwin?
[email protected] Miluzzo
train classification model X in the lab deploy classifier X
train classification model X’ in the lab
mobile phone sensing today
![Page 43: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/43.jpg)
why darwin?
[email protected] Miluzzo
deploy classifier X
deploy classifier X’
mobile phone sensing today
train classification model X’ in the lab
train classification model X in the lab
![Page 44: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/44.jpg)
why darwin?
[email protected] Miluzzo
train classification model X in the lab deploy classifier X
deploy classifier X’
a fully supervised approach doesn’t
scale!
mobile phone sensing today
train classification model X’ in the lab
![Page 45: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/45.jpg)
why darwin? a same classifier does not scale to multiple
environments (e.g., quiet and noisy env)
[email protected] Miluzzo
![Page 46: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/46.jpg)
why darwin? a same classifier does not scale to multiple
environments (e.g., quiet and noisy env)
[email protected] Miluzzo
![Page 47: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/47.jpg)
why darwin? a same classifier does not scale to multiple
environments (e.g., quiet and noisy env)
[email protected] Miluzzo
![Page 48: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/48.jpg)
why darwin? a same classifier does not scale to multiple
environments (e.g., quiet and noisy env)
[email protected] Miluzzo
darwin creates new classification models transparently from the user
(classification model evolution)
![Page 50: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/50.jpg)
[email protected] Miluzzo
why darwin?
ability for an application torapidly scale to many devices
darwin re-uses classification models when possible
(classification model pooling)
![Page 52: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/52.jpg)
[email protected] Miluzzo
why darwin?
leverage the large ensemble of in-situ resources
darwin exploits spatial diversity and co-operate to alleviate the “sensing context”
problem(collaborative inference)
![Page 57: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/57.jpg)
[email protected] Miluzzo
darwin phases
initial training (derive model seed)
classification model evolution
supervised
unsupervised
![Page 58: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/58.jpg)
[email protected] Miluzzo
darwin phases
initial training (derive model seed)
classification model evolution
classification model pooling
supervised
unsupervised
![Page 59: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/59.jpg)
[email protected] Miluzzo
darwin phases
initial training (derive model seed)
classification model evolution
classification model pooling
collaborative inference
supervised
unsupervised
![Page 61: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/61.jpg)
[email protected] Miluzzo
classification model training
sensed event
filtering (silence suppression +
voicing)
![Page 62: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/62.jpg)
[email protected] Miluzzo
classification model training
sensed event
filtering (silence suppression +
voicing)
featureextraction(MFCC)
![Page 63: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/63.jpg)
[email protected] Miluzzo
classification model training
filtering (silence suppression +
voicing)
featureextraction(MFCC)
modeltraining(GMM)
model
baseline
sensed event
send model + baseline back to phone
send MFCC tobackend to train the model
backend
![Page 64: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/64.jpg)
[email protected] Miluzzo
classification model training
phone: feature extraction(low
computation)
backend
backend: model training (high
computation)
![Page 67: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/67.jpg)
[email protected] Miluzzo
classification model evolution
phone: determines when to evolve
training sampled
![Page 68: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/68.jpg)
[email protected] Miluzzo
classification model evolution
phone: determines when to evolve
match?
YES
do not evolve
![Page 69: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/69.jpg)
[email protected] Miluzzo
classification model evolution
phone: determines when to evolve
match?
NO
evolve(train new model using
backend as before)
![Page 74: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/74.jpg)
[email protected] Miluzzo
classification model pooling
Speaker A’s model
Phone A Phone B
Phone C
Speaker C’s model
Speaker B’s modelSpeaker B’s model
Speaker C’s model
![Page 75: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/75.jpg)
[email protected] Miluzzo
classification model pooling
Speaker A’s model
Phone A Phone B
Phone C
Speaker C’s model
Speaker B’s modelSpeaker B’s model
Speaker C’s model
we have two options
1. train a new classifier for each speaker (costly for power, inference delay)
![Page 76: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/76.jpg)
[email protected] Miluzzo
classification model pooling
Speaker A’s model
Phone A Phone B
Phone C
Speaker C’s model
Speaker B’s modelSpeaker B’s model
Speaker C’s model
we have two options
1. train a new classifier for each speaker (costly for power, inference delay)
2. re-use already available classifiers
![Page 77: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/77.jpg)
[email protected] Miluzzo
classification model pooling
Speaker A’s model
Phone A Phone B
Phone C
Speaker C’s model
Speaker B’s modelSpeaker B’s model
Speaker C’s model
we have two options
1. train a new classifier for each speaker (costly for power, inference delay)
2. re-use already available classifiers
![Page 78: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/78.jpg)
[email protected] Miluzzo
classification model pooling
Speaker A’s model
Phone A Phone B
Phone C
Speaker C’s model
Speaker B’s modelSpeaker B’s model
Speaker C’s model
![Page 79: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/79.jpg)
[email protected] Miluzzo
classification model pooling
Speaker A’s model
Phone A Phone B
Phone C
Speaker B’s model
Speaker C’s model
Speaker C’s model
Speaker B’s model
Speaker A’s model
Speaker C’s model
![Page 80: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/80.jpg)
[email protected] Miluzzo
classification model pooling
Speaker A’s model
Phone A Phone B
Phone C
Speaker B’s model
Speaker C’s model
Speaker C’s model
Speaker B’s model
Speaker A’s model
Speaker C’s model
Speaker A’s model
Speaker B’s model
![Page 81: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/81.jpg)
[email protected] Miluzzo
classification model pooling
Speaker A’s model
Phone A Phone B
Phone C
Speaker B’s model
Speaker C’s model
Speaker C’s model
Speaker A’s model
Speaker B’s model
Speaker B’s model
Speaker A’s model
Speaker C’s model
![Page 82: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/82.jpg)
[email protected] Miluzzo
classification model pooling
Speaker A’s model
Phone A Phone B
Phone C
Speaker B’s model
Speaker C’s model
Speaker C’s model
Speaker A’s model
Speaker B’s model
Speaker B’s model
Speaker A’s model
Speaker C’s model
ready to run the collaborative inference algorithm
- local inference first- final inference later
![Page 84: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/84.jpg)
[email protected] Miluzzo
collaborative inference
1. local inference (running independently in parallel on each mobile phone)
two phases
![Page 85: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/85.jpg)
[email protected] Miluzzo
collaborative inference
1. local inference (running independently in parallel on each mobile phone)
two phases
2. final inference (after collecting Local Inference results, to get better confidence about the final classification result)
![Page 87: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/87.jpg)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
speaker A speaking!!!local inference (LI)
![Page 88: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/88.jpg)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
speaker A speaking!!!local inference (LI)
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
![Page 89: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/89.jpg)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
speaker A speaking!!!local inference (LI)
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
![Page 90: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/90.jpg)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
speaker A speaking!!!
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
local inference (LI)
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
![Page 91: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/91.jpg)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
speaker A speaking!!!
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
local inference (LI)
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
![Page 92: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/92.jpg)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
speaker A speaking!!!local inference (LI)
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
![Page 93: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/93.jpg)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
speaker A speaking!!!local inference (LI)
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
individual classification can be misleading!
![Page 94: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/94.jpg)
final inference (FI)
[email protected] Miluzzo
collaborative inference
Phone A Phone B
Phone C
each phone gathers LI results
A’s LI results
C’s LI results
B’s LI results
A’s LI results A’s LI results
C’s LI results C’s LI results
B’s LI resultsB’s LI results
![Page 95: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/95.jpg)
final inference (FI)
[email protected] Miluzzo
collaborative inferenceon each phone
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
![Page 96: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/96.jpg)
[email protected] Miluzzo
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
xxx
xxx
final inference (FI)
collaborative inferenceon each phone
![Page 97: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/97.jpg)
[email protected] Miluzzo
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
xxx
xxx
FI results (normalized):Confidence (A speaking) = 1 Confidence (B speaking) =
0.12Confidence (C speaking) =
0.002
=
final inference (FI)
collaborative inferenceon each phone
![Page 98: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/98.jpg)
[email protected] Miluzzo
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
xxx
xxx
=FI results (normalized):Confidence (A speaking) = 1 Confidence (B speaking) =
0.12Confidence (C speaking) =
0.002
final inference (FI)
collaborative inferenceon each phone
![Page 99: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/99.jpg)
[email protected] Miluzzo
A’s LI results:Prob(A speaking) = 0.65Prob(B speaking) = 0.25Prob(C speaking) = 0.10
C’s LI results:Prob(A speaking) = 0.30Prob(B speaking) = 0.67Prob(C speaking) = 0.03
B’s LI results:Prob(A speaking) = 0.79Prob(B speaking) = 0.11Prob(C speaking) = 0.10
xxx
xxx
=
collaborative inference compensates the inaccuracies of individual
inferences
FI results (normalized):Confidence (A speaking) = 1 Confidence (B speaking) =
0.12Confidence (C speaking) =
0.002
final inference (FI)
collaborative inferenceon each phone
![Page 101: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/101.jpg)
[email protected] Miluzzo
evaluation
C/C++ &
implemented on Nokia N97 andiPhone in support of a speaker
recognition app
![Page 102: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/102.jpg)
[email protected] Miluzzo
evaluation
C/C++ &
unix server
implemented on Nokia N97 andiPhone in support of a speaker
recognition app
![Page 103: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/103.jpg)
[email protected] Miluzzo
evaluation
C/C++ &
unix server
lightweight reliable protocol to transfer models from the server
and between phones
implemented on Nokia N97 andiPhone in support of a speaker
recognition app
![Page 104: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/104.jpg)
[email protected] Miluzzo
evaluation
C/C++ &
UDP multicast protocol to distribute
local inference results between phones
implemented on Nokia N97 andiPhone in support of a speaker
recognition app
![Page 105: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/105.jpg)
[email protected] Miluzzo
experimental scenarios
up to eight people in conversation in three different scenarios (quiet indoor, down the
street, in a restaurant)
![Page 115: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/115.jpg)
[email protected] Miluzzo
indoor quiet scenario
8 people talking around a table
collaborative inference + classification model evolution
boost the performance of a mobile sensing app
![Page 120: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/120.jpg)
[email protected] Miluzzo
impact of the number of mobile phones
the larger the number of mobile phones collaborating, the better the final inference result
![Page 127: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/127.jpg)
[email protected] Miluzzo
battery lifetime Vs inference responsiveness
smart duty-cycling techniques and machine learning algorithms with better performance in
terms of energy usage on mobile phones need to be identified
![Page 128: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/128.jpg)
[email protected] Miluzzo
a quick recap
smartphone’s are everywhere, let’s exploit their collective sensing and
computation capabilities
![Page 129: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/129.jpg)
[email protected] Miluzzo
a quick recapsmartphone’s are everywhere – let’s exploit
their collective sensing and computation capabilities
smartphone sensing opens up new frontiers: applications can be spread and
big data collected at unprecedented scale enabling endless research
opportunities
![Page 130: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/130.jpg)
[email protected] Miluzzo
a quick recapsmartphone’s are everywhere – let’s exploit
their collective sensing and computation capabilities
continuous sensing is still challenging; efficient mobile sensing requires to
preserve the phone user experience (need for energy efficient ML algorithms
and smart duty-cycling techniques)
smartphone sensing opens up new frontiers: applications can be spread and big data collected at
unprecedented scale enabling endless research opportunities
![Page 131: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/131.jpg)
[email protected] Miluzzo
a quick recapsmartphone’s are everywhere – let’s exploit
their collective sensing and computation capabilities
continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user
experience (need for energy efficient ML algorithms and smart duty-cycling techniques)
ML algorithms should perform reliably in the wild
smartphone sensing opens up new frontiers: applications can be spread and big data collected at
unprecedented scale enabling endless research opportunities
![Page 132: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/132.jpg)
[email protected] Miluzzo
a quick recapsmartphone’s are everywhere – let’s exploit
their collective sensing and computation capabilities
continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user
experience (need for energy efficient ML algorithms and smart duty-cycling techniques)
ML algorithms should perform reliably in the wild
smartphone sensing opens up new frontiers: applications can be spread and big data collected at
unprecedented scale enabling endless research opportunitiesok I think I’m done…
![Page 133: Darwin Phones: the Evolution of Sensing and Inference on Mobile Phones Emiliano Miluzzo *, Cory T. Cornelius *, Ashwin Ramaswamy *, Tanzeem Choudhury *,](https://reader038.fdocuments.net/reader038/viewer/2022110319/56649c3e5503460f948e8990/html5/thumbnails/133.jpg)
[email protected] Miluzzo
a quick recapsmartphone’s are everywhere – let’s exploit
their collective sensing and computation capabilities
continuous sensing is still challenging; efficient mobile sensing requires to preserve the phone user
experience (need for energy efficient ML algorithms and smart duty-cycling techniques)
ML algorithms should perform reliably in the wild
smartphone sensing opens up new frontiers: applications can be spread and big data collected at
unprecedented scale enabling endless research opportunitiesbut please bear in mind…