Gary M. Weiss Comp & Info Science Dept Fordham University [email protected] or...

Smart Phone-Based

Sensor Mining

Gary M. WeissComp & Info Science Dept

Fordham [email protected]

www.cis.fordham.edu/wisdm or wisdmproject.com

Gary M. Weiss ICCS 2012 2

What is Smart Phone Sensor Mining?

Data Mining: Extraction of knowledge from data via

automated methods

Smart phone sensor mining: Extraction of useful knowledge from the

data generated by smart phone sensors

1/11/2012


Smart Phone Sensors

What sensors are found on smart phones? Audio sensor (microphone) Image sensor (camera, video recorder) Tri-Axial Accelerometer Location sensor (GPS, cell tower, WiFi) Infrared proximity sensor; Light sensor Magnetic compass; Temperature sensor; Touch

sensor Virtual/calculated sensors: ▪ Proximity (via light), gravity, orientation, gyroscope

1/11/2012


How Does this Topic Relate to ICCS?

Learning about smart phone users Security requires understanding how devices

used Main focus of talk not on security but on what

can be learned about smart phone users

Smart phone based biometric identification Can be considered a security application

Many news stories about abuses Apps to spy on your spouse; iPhone location

fiasco1/11/2012


WISDM Research Areas

Activity recognition (what are you doing)? Are you walking, jogging, sitting, standing,

etc?

Biometric Identification (who are you)? Are you John Smith?

Trait Identification (who are you at diff. level)? Are you male? Are you tall? What do you

weigh?1/11/2012


Why Learn Everything About You?

Data miners want to learn everything about you Somehow that info will be useful Develop useful apps, marketing leads,

etc. Many positive uses▪ That is why NSF provided WISDM with funding

for activity recognition from “Health and Well Being” program

But obviously issues with privacy and abuse1/11/2012


Data Mining: Basic Approach

Approach to Predictive Data Mining

1. Collect labeled (sensor) training data

2. Apply data mining method to build predictive model

3. Apply predictive model to future unlabelled data

1/11/2012


Activity Recognition

1/11/2012


Activity Recognition

Why is it useful? Context-sensitive applications▪ Context influences handling of phone calls or

music to play Health applications▪ Track activity levels or detect falls in elderly

Approaches to activity recognition Uses multiple accelerometers Use custom devices (pedometer, FitBit) Our approach: use existing smart phones

1/11/2012


Sample Accelerometer Data

Accelerometer data from Android phone Walking Jogging Climbing Stairs Lying Down Sitting StandingGravity included

1/11/2012


Accelerometer Data for “Walking”

1/11/2012


Accelerometer Data for “Jogging”

1/11/2012


Accelerometer Data for “Up Stairs”

1/11/2012


Accelerometer Data for “Standing”

1/11/2012


Activity Recognition Results: Impersonal

1/11/2012

Impersonal (Universal) Model Single Model trained and used for everyone

Data Mining Method: Instance Based Learning (WEKA IB3)

72.4%Accuracy

Predicted Class

Walking

Jogging

Stairs

Sitting

Standing

LyingDown

Actual Class

Walking 2209 46 789 2 4 0

Jogging 45 1656 148 1 0 0

Stairs 412 54 869 3 1 0

Sitting 10 0 47 553 30 241

Standing 8 0 57 6 448 3

Lying Down 5 1 7 301 13 131


Activity Recognition Results: Personal

1/11/2012

Personal Model: Model Build per UserData Mining Method: Instance Based Learning (WEKA IB3)

98.4%accuracy

Predicted Class

Walking Jogging Stairs

Sitting

Standing

LyingDown

Actual Class

Walking 3033 1 24 0 0 0

Jogging 4 1788 4 0 0 0

Stairs 42 4 1292 1 0 0

Sitting 0 0 4 870 2 6

Standing 5 0 11 1 509 0

Lying Down 4 0 8 7 0 442


Biometric Identification

1/11/2012


Biometric Identification

Identification based on physical/behavioral traits Fingerprints, DNA, iris, gait, etc.

Biometrics for everyone Equipment smaller & cheaper (sensors + processing)▪ Laptops currently perform face recognition

Gait-based recognition Most work is camera-based

Some applications device security, customization & personalization

1/11/2012


WISDM Biometrics

Used for identification and authentication Identification means predicting identity from pool

of users (36 in initial study and 200 in recent study)

Authentication is a binary class prediction▪ Is it you or an imposter?

We evaluate walking and other activities as well as unclassified activities

Predictions made on individual 10 sec. samples but also combine “votes” to exploit larger samples

1/11/2012


WISDM Biometric Prediction Results

Unclassified

Walk Jog Up Dow

n

J48 72.2 84.0 83.0

65.8

61.0

Neural Net

69.5 90.9 92.2

63.3

54.5

Straw Man

4.3 4.2 5.0 6.5 4.7

1/11/2012

Unclassified

Walk

Jog Up Down

J48 36/36 36/36

31/32

31/31 28/31

Neural Net

36/36 36/36

32/32

28.5/31

25/31

Based on 10 second test samples

Based on most frequent prediction for 5-10 minutes of data

Recent unpublished results demonstrate 100% accuracy with 200 users!

Authentication results even better (~90% with 10 sec samples)


Trait Identification

1/11/2012


Trait Identification Applications

Soft biometrics: traits can aid with biometrics

As data miners we want to know everything about a person Marketing applications: ads based on sex Inferred weight to predict calories

burned

1/11/2012


Expanding the Definition of Trait Normally think about traits as being:

Unchanging: race, skin color, eye color, etc. Slow changing: Height, weight, etc.

But want to know everything about a person: What they wear, how they feel, if they are

tired, etc. Have never seen this goal for mobile sensor

mining

1/11/2012


WISDM Trait Identification Work in early stages Data initially collected from ~70 people,

now 200 Accelerometer and survey data Survey data includes anything we could think of

that might somehow be predictable▪ Sex, height, weight, age, race, handedness, disability▪ Shoe size, footwear type, size of heels, type of

clothing▪ # hours academic work , # hours exercise

Too few subjects investigate all factors▪ Many were not predictable (maybe with more data)

1/11/2012


WISDM Trait Identification Results

Accuracy

71.2%

Male

Female

Male 31 7

Female 12 16

1/11/2012

Accuracy

83.3%

Short

Tall

Short 15 5

Tall 2 20

Accuracy

78.9%

Light Heavy

Light 13 7

Heavy 2 17Results for IB3 classifier. For height and weight middle categories removed.


Security & Privacy

1/11/2012


Security and Privacy

Security policies vary widely by OS & platform Symbian requires properly signed keys to

remove restrictions on using certain APIs iPhone apps have relatively strict oversight Android OS has few restrictions and

Marketplace has essentially no oversight or restrictions▪ WISDM project has had no problem tapping into

sensors and transmitting results. Just pay $25 for account.

1/11/2012


Android Notifications

Android notifies user of services SYSTEM PERMISSIONS FOR WISDM SensorCollector▪ Coarse location, fine location, internet access, keep from

sleeping, modify/delete USB storage

Applications routinely access sensitive services Fandango : fine GPS location, read phone state &

identity, modify/delete USB storage, internet access

Angry Birds: identical permissions! Notifications probably next to useless given this!

1/11/2012



Even legitimate applications have to be concerned with privacy & security WISDM will encrypt data in transit,

encrypt on phone, include secure accounts & passwords, etc.

Need to ensure than any aggregated info is made public only if cannot be traced to individual

1/11/2012



Good Policies: Make it clear what you are monitoring and storing Provide application level control for the user▪ Allow user to turn on/off monitoring of specific sensors▪ If they use an option to upload the information to

Facebook then little privacy!

Since legitimate and illegitimate apps function alike, no easy way to distinguish them Could try to use only certified apps, but quite

limiting

1/11/2012


Available Soon: Actitracker

WISDM is building & deploying the actitracker service to track your activities real-time and display them via a web-based interface Useful health information and thus

supported by NSF Grant & Google faculty research award

Actitracker.com online and should have basic functionality shortly

1/11/2012


Special Thanks To …

WISDM research group Current Members▪ Anthony Alcaro, Alex Armero, Shaun Gallagher,

Andrew Grosner, Margo Flynn, Jeff Lockhart, Paul McHugh, Luigi Patruno, Tony Pulickal, Greg Rivas, Priscilla Twum, Bethany Wolff, Zach Wyhowanec, Jack Xue

Key Former Members▪ Jennifer Kwapisz, Sam Moore, Shane Skowron,

Alvan Wong Funders: NSF, Google, and Fordham

1/11/2012


WISDM References

1. J.R. Kwapisz, G.M. Weiss, and S.A. Moore. 2010. Activity recognition using cell phone accelerometers, in Proceedings of the Fourth International Workshop on Knowledge Discovery from Sensor Data, 10-18.

2. J. R. Kwapisz, G.M. Weiss, and S.A. Moore, 2010.Cell phone-based biometric identification, in Proceedings of the IEEE Fourth International Conference on Biometrics: Theory, Applications and Systems.

3. J.W. Lockhart, G.M. Weiss, J.C. Xue, S.T. Gallagher, A.B. Grosner, T.T. Pulickal. 2011. Design considerations for the WISDM smart phone-based sensor mining architecture, in Proceedings of the Fifth International Workshop on Knowledge Discovery from Sensor Data, San Diego, CA.

4. G.M. Weiss, and J.W. Lockhart, 2011.Identifying user traits by mining smart phone accelerometer data, in Proceedings of the 5th International Workshop on Knowledge Discovery from Sensor Data., San Diego, CA.

1/11/2012


Thank youFor more information go to wisdmproject.com Gary Weiss

[email protected]

1/11/2012

Gary M. Weiss Comp & Info Science Dept Fordham University [email protected] or...

Documents

Transcript of Gary M. Weiss Comp & Info Science Dept Fordham University [email protected] or...