Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative...
Transcript of Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative...
![Page 1: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/1.jpg)
![Page 2: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/2.jpg)
Shaun KaneUniversity of Colorado Boulder
AI Support for Communication Disabilities
![Page 3: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/3.jpg)
Augmentative and Alternative Communication (AAC)
• AAC ➡ Any tool to replace or supplement spoken and written communication [ASHA]
Input: symbolic or typing
Output: synthesized speech, text (SMS, email);may be used for computer access
• About 2 million current AAC users in US
(~0.8% of population) [Beukelman]
![Page 4: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/4.jpg)
Who benefits from AAC?
• Physical disabilities (e.g. cerebral palsy, ALS)
• Linguistic disabilities (e.g., aphasia)
• Cognitive and intellectual disabilities
• Autism spectrum disorders…
• Typically functioning adults (e.g. learning new languages, texting)
![Page 5: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/5.jpg)
How AAC impacts people’s lives
• Support independent access to work, school, travel
• Maintain ties with family and friends
• Support expression of independent social identityaka.ms/expressiveaac
![Page 6: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/6.jpg)
“Until there is a medical
treatment or cure [for ALS],
technology can be a cure”
–Steve Gleason
Impact
![Page 7: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/7.jpg)
The potential of AI for communication support
• AAC (and other communication support tools) present exciting opportunities for AI+HCI
• Underexplored research area with large potential impact
![Page 8: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/8.jpg)
Human-computer interaction challenges for AAC• Support real-time communication about any possible
topic
• Enable rapid communication despite slow input (20 wpm eye-typing vs. 180wpm spoken conversation)
• Authentic communication: do I sound like myself? (phrasing, pronunciation, emoting)
• Key is understanding context and predicting what the user might wish to talk about (and how they would say it)
![Page 9: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/9.jpg)
Contextual factors in communication
• Speaker: current and recent activities, conversational goals, personal communication style(s)
• Conversation partner: relationship, interests, prior communications
• Environment: location, activity, conversation partner, conversation flow, nearby objects
![Page 10: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/10.jpg)
Can we use humans in the loop?
• Private/sensitive information (talking to spouse, discussing health information)
• Low latency is key (conversational speech is ~180wpm)
• Use technology to support independence
![Page 11: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/11.jpg)
Context-aware communication support
• TalkAbout: user-defined contextual communication
• SceneTalk: Leveraging computer vision for context detection
![Page 12: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/12.jpg)
Location &group context
![Page 13: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/13.jpg)
TalkAbout (2012) shaun.cat/aphasia
• Tablet-based communication aid for people with aphasia
• Developed through participatory design
• Allow users and caregivers to set up conversational contexts based on location & speaking partner
• Limitations: Requires manual configuration;assumes conversations will match context
![Page 14: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/14.jpg)
![Page 15: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/15.jpg)
Local environment as context
![Page 16: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/16.jpg)
SceneTalk (2017) aka.ms/scenetalk
• Conversation support for eye-gazetyping
• Developed for (and with) peoplewith ALS
• Use image recognition to identifynearby objects and suggestrelevant conversation topics(using Microsoft Cognitive Services)
![Page 17: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/17.jpg)
![Page 18: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/18.jpg)
Current state of the art
• Research shows promise for integrating more context into communication aids
Significant challenges remain
• Matching environmental understanding to appropriate language model
• Data collection – slow output, very few phrases
• Assessment – small improvements could have a big impact over time, but can be difficult to measure
![Page 19: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/19.jpg)
Opportunities for AI research
• Increasing awareness of location and actvity
• Fusion of contextual data sourcese.g. nearby objects + location
• Authentic communication: how would this person comment about the current context?
• Assessing interventions with limited data
![Page 20: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/20.jpg)
![Page 21: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/21.jpg)
![Page 22: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/22.jpg)
![Page 23: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/23.jpg)
Human-computer interaction challenges for AAC
• Support rapid, real-time communication
• Enable communication about a wide range of topics while minimizing input requirements
• Word & phrase prediction
• Understanding context (conversation, location, event)
• Supporting error-prone input
• Combining pre-prepared with live content
• Expressivity
![Page 24: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/24.jpg)
Shaun Kane
University of Colorado Boulder
AI Support for Communication Disabilities
![Page 25: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/25.jpg)
Challenges for end users
• Speaking and writing (for those who cannot speak or write independently)
• Vocabulary support for unknown or forgotten words
• Supporting real-time communication with a variety of conversational partners
![Page 26: Microsoft Research Academic Services GTM Draft · 2018-01-04 · Augmentative and Alternative Communication (AAC) • AAC Any tool to replace or supplement spoken and written communication](https://reader034.fdocuments.net/reader034/viewer/2022050203/5f570481a8600420a36bdc25/html5/thumbnails/26.jpg)