Amul&’modaltrainingapproach · PDF fileIntroduc&on" •...

A mul&-‐modal training approach to improve cochlear implant users' ability to handle simultaneous talk

Amy Beeston and Emina Kur&ć Mul&modality in Language Research

Leeds ·∙ 26 June 2014

University of Sheffield •  Prof Bill Wells, Human Communica&on Sciences •  Prof Guy Brown, Computer Science •  Dr Emina Kur&ć and Amy Beeston, Research Associates, Computer Science •  Ella Page, project student, Human Communica&on Sciences Sheffield Teaching Hospitals NHS Trust •  Dr Harriet Crook, Clinical Scien&st, Neurotology •  Erica Bradley, Speech & Language Therapist, Neurotology Cochlear implant (CI) user panel •  5 cochlear-‐implanted par&cipants recruited from Sheffield Cochlear Implant Service

Project team

!"#$!%&'(&)&(&"%&*

%+,-$.&('"+./&(*

0111234524

/&&106255

-78'9$!"-:;7-& <1:16:<1=0.!,,'7.>'/&&,-$"=36'%&.&-&(>'($7;,?&))!&@;,=='3)A

!"#$!%&

;&-7!@,'$)'%?7(B& %?7(B& #7- -$-7@@!"&'"$C

!

97>.&"-';+&';7-& =5:14:<1=0

)$('-?&'(&%$#&(>'$)')&&'(&*;/,'%?&%D

'03C02 '1C11 '03C0211= !"'&#&"-'$)'7"'!"#$!%&'A+&(>%$"-7%-'($!,!"'.%)&&@>'$"1==0'216'<611

!

-$-7@' '03C02

!EFGHIJ'(JKJLJEIJ'0111234524'M/&&106255N'KGL'O'03C02'HP'EGQ'RSJ'TER'UTVTWXJ'WV'GEJ'GK'YZJ'[JYZGRP'WJXGQ*\'/V'/TE]'-LTEPKJL':'/7%,':'?G[J'/TE]HE^_',GLY'%GRJ'132211`'/TE]'7IIGSEY'=<<a0663C'!/7"'B/0<%!-!132211=<<a0663`',b!)-'%!-!B/<@\'/V';JWHY:'%LJRHY'ITLR'WV'YJXJUZGEHE^'GE'1==0'<a=<141'GL'1==0'<a=05<aC\'!E'UJLPGE'YG'YZJ'%TPZHJLP'$KKHIJ`'JHYZJL'TY'YZJ'(GVTX'?TXXT[PZHLJ'?GPUHYTX`'BXGPPGU'(GTR`',ZJKKHJXR',=1'<c)'GL'TY'YZJ'"GLYZJLE'BJEJLTX'?GPUHYTX`'

?JLLHJP'(GTR`',ZJKKHJXR',6'a7+\'/V'UGPY'd'%ZJeSJP`'UGPYTX'GLRJLP'JYIC'PZGSXR'WJ'ILGPPJR'f7:%'9TVJJ'$EXVf'TER'[TRJ'UTVTWXJ'YG',ZJKKHJXR'-JTIZHE^'?GPUHYTX'"?,')GSERTYHGE'-LSPY

9XJTPJ'JEIXGPJ'T'PYT[UJR'TRRLJPPJR'JEFJXGUJ'HK'T'LJIJHUY'HP'LJeSHLJR

9XJTPJ'RJYTIZ'YZHP'PJIYHGE'TER'KGLQTLR'QHYZ'VGSL'LJ[HYYTEIJ'YG*%TPZHJLP'$KKHIJ`'(GVTX'?TXXT[PZHLJ'?GPUHYTX`'BXGPPGU'(GTR`',ZJKKHJXR',=1'<c)'GL%TPZHJLP'$KKHIJ`'"GLYZJLE'BJEJLTX'?GPUHYTX`'?JLLHJP'(GTR`',ZJKKHJXR',6'a7+C

)HETEIHTX',JLFHIJP`',TXJP'@JR^JL`'%XGI]'-GQJL`'"GLYZJLE'BJEJLTX'?GPUHYTX`'?JLLHJP'(GTR`',ZJKKHJXR`',6'a7+-JXC'1==0'<<441a6'')Tg'1==0'<<441a0

#7-'(&B'"GC'460'011='46

'03C02 '1C11

Mul&modality in language research 26 June 2014 ·∙ Leeds ·∙ Amy Beeston and Emina Kur&ć

Introduc&on

•  Losing your hearing is not simply the absence of sound – for individual: loss of capacity to take part in social life – addi&onal societal s&gma: being treated differently*

•  Cochlear implanta&on is not a cure for deafness – it is a prosthe&c subs&tute.

•  Approx 324,200 people worldwide implanted by end of 2012† – 10,000 cases in UK† – c. 113 per month in UK‡

•  Insufficient evidence/training materials to guide clinicians to help cochlear implant (CI) users deal with simultaneous talk.


* Lamb and Archibold (2013), Adult Cochlear Implanta&on: Evidence and experience. Tech rep, The Ear Founda&on. † NIH Publica&on No. 11-‐4798 (2013-‐11-‐01). ‡ Bri&sh Cochlear Implant Group, Annual Update on UK Cochlear Implant Provision (2011-‐2012).

Electric hearing


www.mrc-‐cbu.cam.ac.uk

•  Acous&cal vibra&ons are delivered to brain as electrical signals

Challenges for CI users

•  Cochlear implant (CI) users need op&mum conversa&onal sejngs (quiet, one partner, awareness to avoid overlap).

•  Q: “When no background noise is present, can you effortlessly par&cipate in conversa&ons with friends or family members (e.g., aner dinner)?”


•  Mean response for 26 adults = 5.7

MED-‐EL Hearing Implant Sound Quality Index (HISQUI) v23015_2.0, English.

always never

“almost always”

“frequently”

Challenges for CI users

•  Q: “When mul&ple people are talking simultaneously, can you effortlessly follow discussions of friends and family members?”


MED-‐EL Hearing Implant Sound Quality Index (HISQUI) v23015_2.0, English.

•  Mean response for 26 adults = 2.8 •  Unfamiliar voices are s&ll more difficult to follow •  Limits of the device? Conversa&onal/social experience?

always never

“occasionally” “rarely”

Social signals and behaviours

•  Mul&ple behavioural cues combine to produce a social signal


Vinciarelli et al. (2009). Social signal processing: Survey of an emerging domain. Image and Vision Compu&ng 27, 1743–1759.

– Physical appearance – Gesture and posture – Face/eyes behaviour – Space and environment – Linguis'c content – Vocal behaviour *

* prosodic features, e.g., pitch, loudness, dura&on, silence, …

•  Stronger reliance for CI users on non-‐auditory cues? •  Is vocal non-‐verbal behaviour informa&ve or distrac&ng?

CI sound encoding

•  22 electrodes vs. 30,000 hair cells. •  Low spectral resolu&on: hard to dis&nguish one talker from

another (especially concurrent talkers). •  Poor representa&on of voice F0: hard to iden&fy social ac&ons

realised by prosodic cues (e.g., turn compe&&on).


www.cochlear.com

Scene analysis problem


•  Parse single-‐channel stream without cues from pitch or loca&on

Kur&ć et al. (2012). A corpus of spontaneous mul&-‐party conversa&on in Bosnian Serbo-‐Croa&an and Bri&sh English. Proc. LREC, Istanbul.

Conversa&onal dataset


•  3 hours of informal talk, na&ve English (UK)

•  16% of total talk &me is overlapping talk

•  41% of speaker turns are overlapped by another speaker

Kur&ć et al. (2012). A corpus of spontaneous mul&-‐party conversa&on in Bosnian Serbo-‐Croa&an and Bri&sh English. Proc. LREC, Istanbul.

•  Lip-‐reading impossible; Gaze, gesture, posture, … •  Segmenta&on by turn construc&on unit (TCU), transcrip&on,

overlap compe&&veness, xml data in ELAN annota&on format

CI-‐user feedback


•  Web-‐based ques&onnaire – hearing conversa&ons – taking part in conversa&ons – age, gender, and hearing status (implant type) per ear

•  CI-‐user panel

– 5 expert CI-‐users recruited for dura&on of project – conversa&onal experiences, awareness of overlap – grading speech material – grading user tasks – evaluate training sonware – listening skills – (later, speaking skills)

Listening skill development

•  Objec&ve – Understand mul&-‐party conversa&on •  Visualisa&on of acous&c informa&on •  CI-‐user difficulty ra&ngs for speech material and tasks


Reported difficul&es – iden&fy talker – recognise speech – understand ac&on

i. Speech material

•  Graded exposure to mul&-‐party conversa&on – Incremental learning through repe&&on

•  Classify by number/gender of talkers, turn transi&ons


ii. User tasks

•  Widen conversa&onal experience – Rate difficulty for CI user

•  Focus on non-‐linguis&c aspects – iden&fy number of talkers – iden&fy same/different gender – iden&fy clear/overlap


N G T speech collec'on 0 0 0 1 talker 1 0 0 2 talkers, same gender, clear 1 0 1 2 talkers, same gender, overlap 1 1 0 2 talkers, diff. gender, clear 1 1 1 2 talkers, diff. gender, overlap

Number of talkers, N (single = 0, mul&ple = 1)

Gender, G (same = 0, different = 1)

Turn exchange, T (clear = 0, in overlap = 1)

(0, 0, 0) T G

E

Speaking skill development


•  Objec&ve: par&cipate in mul&-‐party conversa&on •  1. gather voice norms (pitch, intensity, tempo) •  2. use overlaps appropriately

– Iden&fy entry &mes (especially in overlap) – Control pitch/loudness (show compe&&veness)

Conclusions and further work


•  Some CI-‐users keen to develop mul&-‐party conversa&onal skills – varied popula&on – different experiences/strategies

•  Informal talk is extremely challenging – missing some turn-‐taking cues – mul&modal presenta&on of acous&c cues

•  Learning interac&onal strategies for handling overlapping talk – complement technological advances – increase social par&cipa&on

•  Sonware may be more widely relevant – hearing or communica&ve difficul&es

Thank you for your ayen&on

•  Any ques&ons?


Amul&’modaltrainingapproach · PDF fileIntroduc&on" •...

Documents

Transcript of Amul&’modaltrainingapproach · PDF fileIntroduc&on" •...