Wearable Computing - Part II: Sensors

1. Daniel Roggen 2011 Wearable Computing Part II Sensors

2. Daniel Roggen www.danielroggen.net [email protected] Physical Social Mental Mental Emotion awareness Own/Others Sadness, joy Depression Cognitive awareness Cognitive load Attention, concentration Stress Social Social interactions Detect known people Social network Information exchange Optimization of organizations Crowd / collective behavior Dimensions of context Physical User location Absolute, relative User activity Manipulative gestures, pointing movements, modes of locomotion, posture, composite and hierarchical activities Environment Map of surrounding services Environment characteristics Temperature, light, humidity Radio fingerprints Environment 3. Daniel Roggen www.danielroggen.net [email protected] Sensing context Sensors are needed to infer the users context Software sensors SMS on mobile phones: e.g. used to infer friend network Email reception: e.g. work / home presence Calendar: e.g. location, work activities Hardware sensors (the typical definition of sensors in EE ) Accelerometers: e.g. gesture recognition RFID tags: e.g. object detection Reed switch e.g. door/window open/closed GPS: e.g. outdoor location WIFI fingerprints: e.g. indoor location 4. Daniel Roggen www.danielroggen.net [email protected] There is no Drink Sensor Simple sensors (e.g. RFID) can provide a "binary" information Presence (e.g. RFID, Proximity infrared sensors) Movement (e.g. ADXL345 accelerometer activity/inactivity pin) Fall (e.g. ADXL345 accelerometer freefall pin) But in general activity-X sensor does not exist Sensor data must be interpreted Multiple sensors must be correlated (data fusion) Several factors influence the sensor data Drinking while standing: the arm reaches the object then the mouth Drinking while walking: the arm moves, and also the whole body Context is inferring from the sensor data with Signal processing Machine learning Reasoning Can be integrated into a sensor node or smart sensor Sensor chip + data processing in a device 5. Daniel Roggen www.danielroggen.net [email protected] et al., Collecting complex activity datasets in highly rich networked sensor environments, Proc INSS, 2010 6. Daniel Roggen www.danielroggen.net [email protected] Accelerometer The most common sensor in activity recognition Used in mobile phones to rotate screen Nintendoo Wii Highly miniaturized with MEMS technology [1] ADXL335 datasheet, Analog devices [2] http://www.silicondesigns.com/tech.html [2] ADXL335 (analog output) 3D accelerometer 4x4 mm Vin: 1.8-3.6V I: 350 uA Analog output: 300mV/g 30'000 interaction primitives (object, environment) Annotation length = 230% of dataset length Highly collaborative effort 11 days recording session, 10 experimenters Subjects instrumented for 5-7 hours ~10 hours of annotation effort for 30mn of data 55. Daniel Roggen www.danielroggen.net [email protected] Challenge 1: Data recording with heterogeneous sensor networks Integration at system level + Central control & monitoring + Synchronized data acquisition - Internals of sensor systems - Fixed real-time merge Integration at data level + Independent data recorders + Robustness, flexibility - Complex control & monitoring - Offline synchronization Obtain synchronized data streams for further processing 7 computers recording sensor data Store data and data reception time Coarse NTP synchronisation Fine synchronisation with specific gestures (jump and clap) 56. Daniel Roggen www.danielroggen.net [email protected] Challenge 2: data handling after recording Burst equalization w/ streaming sensors Missing data represented as NaNs Stream alignment to video footage 57. Daniel Roggen www.danielroggen.net [email protected] Challenge 3: flexible activity annotations, at all levels ~60 instances ~30'000 instances Solution: annotation on multiple tracks, hand-action-object representation 58. Daniel Roggen www.danielroggen.net [email protected] Some lessons learned and best practices Complex recordings will see heterogeneous WSN System level integration is desired however data level integration may be sufficient Developing custom tools can be beneficial Signal/video alignment, annotation Now: students can do the work Plan for a repository for raw data Required already before data can be stored in a database Can be huge! 59. Daniel Roggen www.danielroggen.net [email protected] Some lessons learned and best practices Test infrastructure locally Placement of antennas Integration & optimization of the wireless links Expect, plan for, and accept data loss or failures Independent systems Strategy: ignore problem, fix&restart, fix&continue Data integrity checks Tradeoff: frequency / time of test-subjects Develop a rapid on-body sensor deployment solution Clothing attached/integrated sensors Reproducible placement More convenient 60. Daniel Roggen www.danielroggen.net [email protected] Some lessons learned and best practices Avoid wireless links altogether :) Use local data storage Wireless only for synchronization Do not underestimate logistics 10 people during 11 days Room rented for 11 days, 3 days setup Setup: 3 days 61. Daniel Roggen www.danielroggen.net [email protected] http://www.intersense.com Nintendo WiiMote Open source BlueSense acceleration/gyro sensors (http://www.wearable.ethz. ch/resources/educationkit) Some readily available sensors http://www.phidgets.com/ http://www.sparkfun.com/ LilyPad (http://hlt.media.mit.edu/? p=34) http://www.xsens.com/ 62. Daniel Roggen www.danielroggen.net [email protected] Summary User requirements Small / unobtrusive Invisible to the outside Comfortable Privacy issues Context-recognition requirements Must be discriminative of the context to recognize Minimize subsequent processing (simpler modalities favored) Hardware requirements Low-power Low-computational complexity Usually multimodal approaches Multiple activities affect a sensor Several sensors can disambiguate the activity 63. Daniel Roggen www.danielroggen.net [email protected] For further reading Context Dey, Understanding and Using Context, Personal and Ubiquitous Computing,2001 Activity-aware computing Lukowicz et al., On-Body Sensing: From Gesture-Based Input to Activity-Driven Interaction, IEEE Computer, 43(10), pp. 92-96, 2010 N. Davies, D. Siewiorek, R. Sukthankar, Special Issue: Activity-Based Computing, IEEE Pervasive Computing, 7(2), pp. 20-21, 2008 Stiefmeier et al, Wearable Activity Tracking in Car Manufacturing, PCM, 2008 Cognitive-affective computing R. Picard, Affective computing, MIT Press, 1997 R. Picard, E. Vyzas, J. Healey, Toward Machine Emotional Intelligence: Analysis of Affective Physiological State, IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(10), pp. 1175-1191, 2001 Bulling et al., Whats in the Eyes for Context-Awareness? Pervasive computing magazine, 2011 Location-awareness Hightower, Borriello, Location systems for ubiquitous computing, IEEE Computer, pp. 57-66, 2001 Kjaergaard, Blunck, Godsk, Toftkjaer, Christensen, Gro}nb{ae}k, Indoor Positioning Using GPS Revisited, Pervasive, 2010 Koo, Cha, Autonomous Construction of a WiFi Access Point Map Using Multidimensional Scaling, Pervasive, 2011 Social context Wirz, Roggen, Trster, Decentralized Detection of Group Formations from Wearable Acceleration Sensors, Int. Conf. Social Computing, 2009 Datasets Roggen et al., Collecting complex activity data sets in highly rich networked sensor environments, INSS, 2010 Context and Sensing frameworks Roggen et al., Titan: An Enabling Framework for Activity-Aware ``PervasiveApps'' in Opportunistic Personal Area Networks, EURASIP Journal on Wireless Communications and Networking, 2011 Kukkonen, Lagerspetz, Nurmi, Andersson, BeTelGeuse: A Platform for Gathering and Processing Situational Data, IEEE Pervasive Computing Magazine, 8(2): 49-56, 2009 Fortino, Guerrieri, Bellifemine, Giannantonio, SPINE2: Developing BSN Applications on Heterogeneous Sensor Nodes, Proc. IEEE Symposium on Industrial Embedded Systems SIES2009, 2009 Bannach et al., Rapid Prototyping of Activity Recognition Applications, IEEE Pervasive Computing Magazine, 7(2):22-31, 2008 Kurz et al., The OPPORTUNITY Framework and Data Processing Ecosystem for Opportunistic Activity and Context Recognition, International Journal of Sensors, Wireless Communications and Control, 2011 64. Daniel Roggen www.danielroggen.net [email protected]

Wearable Computing - Part II: Sensors

Technology

Transcript of Wearable Computing - Part II: Sensors