Fluidity: Real-time feedback for speaking fluency …...and pitch. Fluidity: Real-time fluency...

２０１５年３月７・８日日本英語教育学会第45回年次研究集会 1

Fluidity: Real-time feedback for Fluidity: Real-time feedback for speaking fluency developmentspeaking fluency development

Ralph ROSERalph ROSECenter for English Language Education (CELESE)Center for English Language Education (CELESE)

Faculty of Science and EngineeringFaculty of Science and EngineeringWaseda UniversityWaseda University

AcknowledgmentsAcknowledgmentsHiroaki Suzuki, Junichi Inagaki, MasayukiHiroaki Suzuki, Junichi Inagaki, MasayukiMotoori, Yukikatsu Fukuda, TatsuhiroMotoori, Yukikatsu Fukuda, TatsuhiroNomaguchi, Aiko Ooe, Maiko SerizawaNomaguchi, Aiko Ooe, Maiko Serizawa

This research is partially supported by a Waseda University This research is partially supported by a Waseda University Grant for Special Research Projects (#2011B-152) and a Grant for Special Research Projects (#2011B-152) and a

Japanese Ministry of Education, Culture, Sports, Science, Japanese Ministry of Education, Culture, Sports, Science, and Technology (MEXT) Grant-in-Aid (#24520661)and Technology (MEXT) Grant-in-Aid (#24520661)


Fluidity: Real-time fluency feedbackFluidity: Real-time fluency feedback R. Rose, Waseda Univ.R. Rose, Waseda Univ.

OverviewOverview● BackgroundBackground

– Automated assessment and feedbackAutomated assessment and feedback– Fluency in L2 speechFluency in L2 speech

● Fluidity overviewFluidity overview● Usability evaluationUsability evaluation● SummarySummary



Automated assessment of L2 speechAutomated assessment of L2 speech● Pronunciation (with visual feedback*)Pronunciation (with visual feedback*)

– Segmental: Cucchiarini et al 2009; Patten and Edmonds Segmental: Cucchiarini et al 2009; Patten and Edmonds 2013*2013*

– Supra-segmental: Anderson-Hsieh 1992*; Taniguchi and Supra-segmental: Anderson-Hsieh 1992*; Taniguchi and Abberton 1999*; de Wet et al 2009Abberton 1999*; de Wet et al 2009

● FluencyFluency– ETS SpeechRater (Zechner et al 2009)ETS SpeechRater (Zechner et al 2009)– Versant (Pearson, Ordinate; Bernstein 1999)Versant (Pearson, Ordinate; Bernstein 1999)– CASEC (Hayashi et al 2004)CASEC (Hayashi et al 2004)

Useful overviews: Eskenazi 2009; Gamper and Knapp 2010Useful overviews: Eskenazi 2009; Gamper and Knapp 2010



Feedback to learnerFeedback to learner● Eskenazi (1999) - “Learners must receive pertinent Eskenazi (1999) - “Learners must receive pertinent

corrective feedback”corrective feedback”● Most systems provide rapid feedback.Most systems provide rapid feedback.● In human-human communication, some feedback is in real-In human-human communication, some feedback is in real-

timetime– Back-channeling (uh-huh)Back-channeling (uh-huh)– Head movements (nodding, shaking)Head movements (nodding, shaking)– Facial expressionsFacial expressions

● Is it possible to provide real-time feedback on fluency-Is it possible to provide real-time feedback on fluency-related matters in human-computer interaction?related matters in human-computer interaction?



Fluency in L2 speechFluency in L2 speech● ScopeScope

– Broad: speak a language proficientlyBroad: speak a language proficiently– Narrow: speak smoothly with minimal but natural hesitationNarrow: speak smoothly with minimal but natural hesitation

● Segalowitz (2010): levels of fluencySegalowitz (2010): levels of fluency– Cognitive fluency: ease of mental preparationCognitive fluency: ease of mental preparation– Utterance fluency: smoothness of articulationUtterance fluency: smoothness of articulation– Perceptual fluency: hearer's view of smoothnessPerceptual fluency: hearer's view of smoothness

Focus of the present workFocus of the present work



Fluency in L2 speechFluency in L2 speech● Crosslinguistic Corpus of Hesitation Phenomena (Rose 2013: Crosslinguistic Corpus of Hesitation Phenomena (Rose 2013:

http://filledpause.com/chp/cchp)http://filledpause.com/chp/cchp)– Speech recordings of Japanese (L1) and English (L2)Speech recordings of Japanese (L1) and English (L2)– Parallel elicitation tasks: read/spontaneous speechParallel elicitation tasks: read/spontaneous speech– Intra-speaker comparisons possibleIntra-speaker comparisons possible

● Fluency factors that correlate with L2 proficiencyFluency factors that correlate with L2 proficiency– Speech rateSpeech rate– Silent pause rateSilent pause rate– Silent pause durationSilent pause duration– Filled pause rateFilled pause rate

Correlated with L1 speechCorrelated with L1 speech

Independent of L1 speechIndependent of L1 speech



Fluidity: fundamental aimsFluidity: fundamental aims● Measure various utterance fluency characteristicsMeasure various utterance fluency characteristics

and update them in real-time.and update them in real-time.● Provide real-time feedback to learner on utteranceProvide real-time feedback to learner on utterance

fluency measures.fluency measures.● Provide opportunity for learner to review theirProvide opportunity for learner to review their

production together with visual representationproduction together with visual representationof fluency measures.of fluency measures.

● Provide feedback in a manner that emulatesProvide feedback in a manner that emulateshuman-human communication.human-human communication.

Curr

ent f

ocus

of d

evel

opm

ent

Curr

ent f

ocus

of d

evel

opm

ent

A work in progress!A work in progress!



Fluidity: fluency measuresFluidity: fluency measures● Phonation timePhonation time● Silence timeSilence time● Syllable count: energy peaksSyllable count: energy peaks

(cf., Bhat et al 2010)(cf., Bhat et al 2010)● Silent pause count: silence > 300msSilent pause count: silence > 300ms

(cf., De Jong and Bosker 2013)(cf., De Jong and Bosker 2013)● Filled pause count: stable formants and pitchFilled pause count: stable formants and pitch

(cf., Audhkhasi et al 2009)(cf., Audhkhasi et al 2009)



Fluidity: main windowFluidity: main window

audio levelaudio levelmetermeter

fluencyfluencymeasuremeasureindicatorsindicators

RequiresRequiresJava SE 6Java SE 6or greateror greater

Uses TarsosDSPUses TarsosDSP(Joren Six) and(Joren Six) andAudioInfo.javaAudioInfo.java

(Jonathan Simon)(Jonathan Simon)librarieslibraries

detectiondetectionsettingssettings

audioaudioinputinput

settingssettings



Fluidity: playback windowFluidity: playback window

playbackplaybackcontrolscontrols

waveformwaveformrepresentationrepresentation

fluencyfluencyvisualizationvisualization

silent pausessilent pausesspeechspeechfilled pausesfilled pauses



Fluidity: usability testingFluidity: usability testing● Participants (n=14, so far)Participants (n=14, so far)● ProcedureProcedure

– Practice speaking with Fluidity.Practice speaking with Fluidity.– Adjust settings to fit their production.Adjust settings to fit their production.– Respond to survey questions about the experience.Respond to survey questions about the experience.

● Still in progress...Still in progress...



Fluidity: user responseFluidity: user response



Fluidity: desirable featuresFluidity: desirable features



Fluidity: technical considerationsFluidity: technical considerations● Silence threshold often needs to be adjusted.Silence threshold often needs to be adjusted.

– Solution: Do a better job of auto-detecting the sound Solution: Do a better job of auto-detecting the sound configuration and microphone settings.configuration and microphone settings.

● Filled pause detection is difficult. Even after adjusting Filled pause detection is difficult. Even after adjusting sensitivity, many participants did not see their filled pauses sensitivity, many participants did not see their filled pauses detected accurately (or at all).detected accurately (or at all).– Solution: Try other algorithms for detecting stable formants Solution: Try other algorithms for detecting stable formants

and pitch.and pitch.


Fluidity: user commentsFluidity: user comments● 「語学を専攻していましたが，発音や文法にとらわれることが「語学を専攻していましたが，発音や文法にとらわれることが多く，流暢さを考えることがあまりなかったので，勉強になりま多く，流暢さを考えることがあまりなかったので，勉強になりました。」した。」

– Although I majored in languages, I have mostly studied about Although I majored in languages, I have mostly studied about pronunciation and grammar and have not studied much pronunciation and grammar and have not studied much about fluency. So, this was very educational.about fluency. So, this was very educational.

● 「具体的にどうすれば良いかは分かりませんが，この「具体的にどうすれば良いかは分かりませんが，この「「 FluidityFluidity 」を基板としたゲーム形式のアプリを使えれば，すご」を基板としたゲーム形式のアプリを使えれば，すごく楽しく使えるかと思います。」く楽しく使えるかと思います。」

– I wasn't really sure how to make use of Fluidity objectively, I wasn't really sure how to make use of Fluidity objectively, but if I could use it like a game application, I think it would be but if I could use it like a game application, I think it would be very enjoyable to use.very enjoyable to use.


SummarySummary● Fluidity is designed to give real-time feedback to L2 English Fluidity is designed to give real-time feedback to L2 English

speakers on their utterance fluency.speakers on their utterance fluency.● The application is still under development, but is capable of The application is still under development, but is capable of

providing real-time feedback on most fluency measures, providing real-time feedback on most fluency measures, plus visualizations for review.plus visualizations for review.

● Users find the application interesting, fun, and they are Users find the application interesting, fun, and they are motivated to think about how to improve fluency.motivated to think about how to improve fluency.

● Users noted that the interface is difficult to use, and filled Users noted that the interface is difficult to use, and filled pauses are not accurately detected.pauses are not accurately detected.

● Future development will include improving accuracy of Future development will include improving accuracy of fluency measurements and creating a user interface that fluency measurements and creating a user interface that better matches human-human interaction.better matches human-human interaction.



ReferencesReferencesAnderson-Hsieh, J. Using electronic visual feedback to teach supra-Anderson-Hsieh, J. Using electronic visual feedback to teach supra-

segmentals. segmentals. SystemSystem, Vol. 20, pp. 51-62, 1992., Vol. 20, pp. 51-62, 1992.

Audhkhasi, K.; Kandhway, K.; Deshmukh, O.D.; Verma, A., (2009) Audhkhasi, K.; Kandhway, K.; Deshmukh, O.D.; Verma, A., (2009) Formant-based technique for automatic filled-pause detection in Formant-based technique for automatic filled-pause detection in spontaneous spoken English. spontaneous spoken English. IEEE International Conference on IEEE International Conference on Acoustics, Speech and Signal Processing, 2009Acoustics, Speech and Signal Processing, 2009 , pp. 19-24 April , pp. 19-24 April 2009.2009.

Bernstein J (1999). PhonePassBernstein J (1999). PhonePassTMTM testing: Structure and construct. testing: Structure and construct. Menlo Park, CA: Ordinate.Menlo Park, CA: Ordinate.

Bhat, S., Hasegawa-Johnson, M., Sproat, R. Automatic Fluency AsBhat, S., Hasegawa-Johnson, M., Sproat, R. Automatic Fluency As--sessment by Signal-Level Measurement of Spontaneous Speech. sessment by Signal-Level Measurement of Spontaneous Speech. 2010 INTERSPEECH Satellite Workshop on Second Language Stud2010 INTERSPEECH Satellite Workshop on Second Language Stud--ies: Acquisition, Learning, Education and Technologyies: Acquisition, Learning, Education and Technology, Makuhari, , Makuhari, Japan, 2010.Japan, 2010.

Cucchiarini, C., Neri, A., Strik, H. (2009) Oral proficiency training in Cucchiarini, C., Neri, A., Strik, H. (2009) Oral proficiency training in Dutch L2: The contribution of ASR-based corrective feedback, Dutch L2: The contribution of ASR-based corrective feedback, Speech CommunicationSpeech Communication, 51(10):853-863., 51(10):853-863.

Eskénazi, M. (1999) Using Automatic Speech Processing for Foreign Eskénazi, M. (1999) Using Automatic Speech Processing for Foreign Language Pronunciation Tutoring: Some Issues and a Prototype. Language Pronunciation Tutoring: Some Issues and a Prototype. Language Learning & TechnologyLanguage Learning & Technology 2(2):62-76. 2(2):62-76.

Hayashi, N., Nogami, Y., Maeda, K., and Ikeda, H. (2004) Practical Hayashi, N., Nogami, Y., Maeda, K., and Ikeda, H. (2004) Practical use of computerized adaptive testing in Japan -Development and use of computerized adaptive testing in Japan -Development and operation of “CASEC”. ICP-2004 at Beijing, China. August 13th.operation of “CASEC”. ICP-2004 at Beijing, China. August 13th.

De Jong, N.H., Bosker, h.R. (2013) Choosing a threshold for silent De Jong, N.H., Bosker, h.R. (2013) Choosing a threshold for silent pauses to measure second language fluency. pauses to measure second language fluency. The 6th Workshop The 6th Workshop on Disfluency in Spontaneous Speechon Disfluency in Spontaneous Speech (DiSS), pp. 17-20. (DiSS), pp. 17-20.

Iomi Patten and Lisa A. Edmonds (2013) Effect of training Japanese Iomi Patten and Lisa A. Edmonds (2013) Effect of training Japanese L1 speakers in the production of American English /r/ using specL1 speakers in the production of American English /r/ using spec--trographic visual feedback. trographic visual feedback. Computer Assisted Language LearnComputer Assisted Language Learn--inging..

Rose, R.L. (2013) Crosslinguistic Corpus of Hesitation Phenomena: A Rose, R.L. (2013) Crosslinguistic Corpus of Hesitation Phenomena: A corpus for investigating first and second language speech percorpus for investigating first and second language speech per--formance. In: formance. In: Proceedings of the 14th Annual Conference of the Proceedings of the 14th Annual Conference of the International Speech Communication AssociationInternational Speech Communication Association (Interspeech (Interspeech 2013), pp. 992-996.2013), pp. 992-996.

Segalowitz, N. (2010) Segalowitz, N. (2010) Cognitive bases of second language fluencyCognitive bases of second language fluency. . Routledge.Routledge.

Simon, J. AudioInfo.java, audio info library. Retrieved from http:// Simon, J. AudioInfo.java, audio info library. Retrieved from http:// www.javafaq.nu/java-example-code-479.html on 25 December www.javafaq.nu/java-example-code-479.html on 25 December 2014.2014.

Six, J. TarsosDSP, Java library. Retrieved from https:// Six, J. TarsosDSP, Java library. Retrieved from https:// github.com/JorenSix/TarsosDSP on 25 December 2014.github.com/JorenSix/TarsosDSP on 25 December 2014.

Taniguchi, M. and Abberton, E. (1999) Effect of Interactive Visual Taniguchi, M. and Abberton, E. (1999) Effect of Interactive Visual Feedback on the Improvement of English Intonation of Japanese Feedback on the Improvement of English Intonation of Japanese EFL Learners. EFL Learners. Speech, Hearing and Language: Work in ProgressSpeech, Hearing and Language: Work in Progress No. 11 (pp.76-89) Department of Phonetics and Linguistics, UniNo. 11 (pp.76-89) Department of Phonetics and Linguistics, Uni--versity College London.versity College London.

de Wet, F., Van der Walt, C., and Niesler, T.R. (2009) Automatic asde Wet, F., Van der Walt, C., and Niesler, T.R. (2009) Automatic as--sessment of oral language proficiency and listening comprehensessment of oral language proficiency and listening comprehen--sion, sion, Speech CommunicationSpeech Communication, 51(10):864-874., 51(10):864-874.

Zechner, K., Higgins, D., Xi, X., and Williamson, D.M. (2009) AutoZechner, K., Higgins, D., Xi, X., and Williamson, D.M. (2009) Auto--matic scoring of non-native spontaneous speech in tests of spomatic scoring of non-native spontaneous speech in tests of spo--ken English, ken English, Speech CommunicationSpeech Communication, 51(10)883-895., 51(10)883-895.

Fluidity: Real-time feedback for speaking fluency …...and pitch. Fluidity: Real-time fluency...

Documents

Transcript of Fluidity: Real-time feedback for speaking fluency …...and pitch. Fluidity: Real-time fluency...