Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different...
Transcript of Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different...
![Page 1: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/1.jpg)
Touché: Enhancing Touch Interaction on Humans, Screens,
Liquids, and Everyday ObjectsMark Howard
![Page 2: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/2.jpg)
Who?
• Munehiko Sato, Ivan Poupyrev, Chris Harrison
• Disney Research Pittsburgh
• Graduate School of Engineering at the University of Tokyo
• HCI Institute at Carnegie Mellon University
![Page 3: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/3.jpg)
Purpose of the Research
• Demonstrate a variety of applications and interactions enabled by Touché
• Demonstrate immediate feasibility of Touché
• Explore the potential richness of gesture vocabularies supported by the system
• Establish the baseline performance of the recognition engine.
![Page 4: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/4.jpg)
Overview• Touché is a scalable capacitive touch sensing and gesture sensitive
technology- only a single electrode is required for sensing purposes (conductive materials serve as intrinsic electrodes)
• Scalable- may be used for touch/gesture detection on many different objects
• Door knob
• Water
• Table
• Not limited to inanimate objects- People!
• Safe, low power, inexpensive, compact
![Page 5: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/5.jpg)
Overview Cont.’d
• Traditional Touch Detection
• Conductive object is excited by an electrical signal at a fixed frequency
• Sensing circuit monitors return signal and uses changes in this signal to determine touch events (human body as a conductor)
• Swept Frequency Capacitive Sensing (SFCS)
• Return signal is monitored over a range of frequencies
• Objects excited by an electrical signal respond differently at differing frequencies; return signal also changes
• Allows measurement of multiple data points at different frequencies to classify gesture/touch using machine learning
![Page 6: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/6.jpg)
Overview Cont.’d
• SFCS can determine HOW a touch event occurred
• Signal frequency sweeping has been used for wireless comm., but not for touch interaction
• Touché is proven as feasible for immediate real-world applications with this research
![Page 7: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/7.jpg)
Related Work• Touch sensing with respect to Human-Computer Interaction (HCI) is central
to touch interaction (via cameras, pressure sensors, acoustics, etc.)
• Like Humantenna
• Capacitive Touch Sensing- family of sensing techniques
• Based on Capacitive Coupling
• Electrical signal in an electrode forms an oscillating electrical field that is altered as a user’s hand nears it
• Degree of signal change is used for classification (signal phase or signal amplitude)
![Page 8: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/8.jpg)
Limitations of Capacitive Sensing
• Not very expressive- can only detect whether something is touched; maybe some proximity info
• Matrices of electrodes (more data collection) can result in more meaningful results (rough 3D shapes, spatial gestures)
• Resulting increased complexity limits potential applications
![Page 9: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/9.jpg)
Swept Frequency Capacitive Sensing
• In this work, single electrode analyzed at varying frequencies – frequency multiplexing
• Data collected at different frequencies used to construct a capacitive profile
• Human skin has high resistance (~1M Ohm), so weak DC signals would be blocked.
• AC signals pass through the body; the resistive qualities of the human body changes the phase and amplitude of an AC signal
• This AC-human interaction has been used since the 60’s for medical applications
![Page 10: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/10.jpg)
Signal Change in SFCS• How the electrode is touched (amount of contact), body connection to
ground (shoes vs bare feet)
• Strongly dependent on signal frequency
• Different frequencies = different paths through the body (path of least impedence)
• Different anatomical parts of the body become more or less impeding depending on the frequency of the AC signal
• Generates information about: how the object is being touched, how the user is touching the ground, and the human body configuration/individual body properties
![Page 11: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/11.jpg)
Touché Implementation
![Page 12: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/12.jpg)
Implementation Cont.’d
• ARM Cortex-M3 microprocessor on a custom board• Runs at 120MHz, AD5932 wave generator (1KHz to 3.5MHz)
• Signal generator on the board excites an electrode and measure return signal at varying frequencies- generate a capacitive profile• Single sweep takes 33ms; sampling rate of ADC is limiting speed factor
• No measurement of phase changes for now
• Capacitive profile sent to computer via Bluetooth for classification (could classify on single board, computer allowed more rapid development)
• Classification- Support Vector Machine (SVM) on a conventional computer
![Page 13: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/13.jpg)
![Page 14: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/14.jpg)
Sensing Configurations
• Sensor touching one (a and c) and two (b and d) locations of the user’s body
![Page 15: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/15.jpg)
Touché Applications
• Broad applications warrant categorization
• Making everyday objects touch gesture sensitive
• Sensing human bimanual hand gestures
• Sensing human body configuration (i.e. pose)
• Enhancing traditional touch interfaces
• Sensing interaction with unique materials (i.e. liquids)
![Page 16: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/16.jpg)
Objects Sensitive to Touch and Grasp
• Doorknobs!
• Already in usual path, not currently computational
• Already conductive; sensor could be connected with single wire
• Features
• tight grasp = lock, pinch = away message, “grasp password”
![Page 17: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/17.jpg)
Sensing Body Configuration• Sense configuration of entire human body without instrumenting the user
• Tables, chairs, etc. may be conductive in nature; otherwise a single electrode is needed
• Applications in gaming, smart offices, rehabilitation, etc.
• Touché Table
![Page 18: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/18.jpg)
Enhanced Touchscreen Interaction
• Sensing hand posture (hand is a fist vs open palm, single finger vs five)
• “right click” functionality on touch interfaces like traditional computers
• Future applications include 3D drawing/sculpting and music composition/performance
• Improvement upon mobile touchscreen interaction seems viable
![Page 19: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/19.jpg)
Sensing On-Body Gestures• Human body is “input device”- must be minimally invasive instrumentally
• Human body is conductive
• Source is placed near the hands, and the other electrode can be placed many locations
• As users touch their body differently, impedence between the electrodes will vary, resulting in differing capacitive profiles
• Example application is making “shh” gesture to silence a phone
![Page 20: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/20.jpg)
Gesture Sensing in Liquids
• Can determine user interaction with regards to liquids
• Not touchscreens in liquids
• Touching the surface of water vs dipping a finger or hand
• Track indirect interactions (touching water via a conductive object)
![Page 21: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/21.jpg)
Study Participants
• 2 groups of 12 participants
• First group- first four studies
• Second group- liquid study
• Studies were independently run to allow real-world environmental variation
![Page 22: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/22.jpg)
Procedure of Studies• Participants shown gesture set to be performed sequentially
• Training
• 10 gesture instances repeated 3 times for 30 instances per gesture per user
• Useful for data analysis and capturing environmental variability
• Testing
• Participants perform random gesture and the classified gesture is compared to the intended gesture
• Five trials for each gesture
![Page 23: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/23.jpg)
Accuracy• Per-user classification (classifier trained on user specific data)
• Accuracy improves as the gesture set decreases
• Strive for gesture sets with at least 95% accuracy
• General Classifier
• Classifier trained with data from 11 participants; 12th participant was tested
• More difficult with regards to classifier accuracy because of differences in the way participants make gestures
• Per-user classification isn’t always feasible
![Page 24: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/24.jpg)
Results
• Study 1- Doorknob:
• Real-time per-user: 96.7% accuracy (dropping circle gesture => 98.6% accuracy)
• Walk-up: 76.8% accuracy (dropping circle => 95.8% accuracy)
• Study 2- Table to Sense Body Configuration:
• Table with thin copper plating
• Real-time per-user: 92.6% accuracy (dropping two elbows gesture => 96% accuracy)
• Walk-up: 81.2% accuracy (dropping two gestures => 91.6%; dropping 4 => 100%)
![Page 25: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/25.jpg)
Results Cont.’d• Study 3- Improved Touchscreen Interaction
• Mobile device used for testing; two electrodes on front and back of device
• Impedance measured between user’s hand; 5 gestures
• Per-user: 93.3% accuracy (removing two finger pinch => 97.7 % accuracy
• Walk-up: 76.1% (reducing to 3 gestures => 100% accuracy)
• Study 4- On-body Gesture Sensing
• 5 gestures
• Per-user: 84% accuracy (94% accuracy with 4 gestures)
• Walk-up: 52.9% accuracy (87.1% with only 3 gestures)
![Page 26: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/26.jpg)
Results Cont.’d
• Study 5- Touching Liquids
• Best of the 5 studies
• Per-user: 99.8% accuracy
• Walk-up: 99.3% accuracy (removing three finger tips gesture => 99.9% accuracy)
• Post Hoc Tests were conducted to ensure no linkage between accuracy and user weight, height , or gender
![Page 27: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/27.jpg)
![Page 28: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/28.jpg)
![Page 29: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/29.jpg)
Conclusions
• Touché works with 200 samples between 1KHz and 3.5 MHz • Good tradeoff between speed and accuracy
• Decreasing sweep resolution would make classification faster while sacrificing accuracy
• Future work is optimizing Swept Frequency Capacitive Sensing for specific applications (how many samples are needed and what frequency bands should be sampled)
• Difficult to determine which frequencies are ideal for specific interactions, users, applications, materials, etc.
• Mark Weiser- “the most profound technologies are those that disappear”• How do we plan for the future of interacting with invisible computers?
![Page 30: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/30.jpg)
Strengths
• Explored broad range of Touché applications (scalability)
• Immediate feasible applications (phone detecting camera mode by hand placement)
• Opens the door for many future research opportunities
• Decent real-world testing
• Walk-up classification
![Page 31: Touché: Enhancing Touch Interaction on Humans, Screens ...€¦ · Data collected at different frequencies used to construct a capacitive profile • Human skin has high resistance](https://reader034.fdocuments.net/reader034/viewer/2022043011/5fa47d41370d2e4be579e50b/html5/thumbnails/31.jpg)
Critiques
• Discussed decreasing gesture set to improve accuracy
• How do smaller gesture sets limit potential applications?
• Does there seem to be a limit for the amount of gestures/options for classifiers?
• Security Issues- Location/body configuration information
• As these computers become more “invisible,” will data collection become more widespread? What does this mean for personal privacy?
• Discussed ideal sample sizes and frequencies depending on application
• Extrapolate sample size/frequency for potential application realms:
• Why is this extrapolation difficult?