Speech and Language TechnologyFor Dialog-based CALL
Gary Geunbae Lee, POSTECH
Outline
Introduction1
Spoken Dialog Systems2
4 PESAA: Postech English Speaking As-sessment and Assistant
5 Field Study
3 DBCALL: Educational Error Han-dling
INTRODUC-TIONINTRODUC-TION
CHAPTER 1
English Tu-toring Meth-ods
English Tu-toring Meth-ods Tranditional Approches
CALL Approches
<CMC> <ICALL>
<Classroom> <Textbook> <Multimedia>
Socio-Economic Ef-fects
Socio-Economic Ef-fects
• Changing our current foreign language educa-tion system in public schools From vocabulary and grammar methodology To speaking ability
• Significant effect of decreasing private English education fee private English education fee in Korea, reaching up
to 16 trillion won annually
• Expect the effect of the overseas export Japan, China, etc.
Interdiciplinary ResearchInterdiciplinary Research
NLP
• Dialog Management• Error Detection• Corrective Feedback
• Comprehensible Input and Output• Corrective Feedback• Attitude & Motivation
SLA
Evaluation
• Cognitive Effect• Affective Effect
Second Language Acquisition Theory
Second Lan-guage Ac-quisition
• Input Enhancement• Comprehensible input• Provision of inputs with high
frequency
• Immersion• Authentic environment• Direct form-meaning map-
ping
• Noticing & Attention• Output hypothesis test • Corrective feedback• Affective factors
• Motivation• Goal achievement & rewards• Interest• Importance of L2
Dialog-Based CALL (DB-CALL)
Dialog-Based CALL (DB-CALL)
<Educational Robot>
<3D Educational Game>
Spoken Dialog System DB-CALL System
Existing DB-CALL Systems
Existing DB-CALL Systems
Alelo Tactical language & culture training system Learn Iraqi Arabic by playing a fun video game Dedicated to serving langauge and culture
learning needs of military
SPELL Learning English in functional situations such
as going to a restaurant, expressing (dis-)likes, etc.
The speech recogniser is programmed to recognise grammatical and some ungrammatical utter-ances
DEAL Learning Dutch in a flea market situation The model can also convey extra linguis-
tic signs such as lip-synching, frowning, nodding, and eyebrow movements
Video DemoVideo Demo
SPOKEN DIALOG SYS-TEMSSPOKEN DIALOG SYS-TEMS
CHAPTER 2
SPOKEN DIALOG SYSTEM (SDS)
Tele-serviceTele-service
Car-navigationCar-navigation Home networkingHome networking
Robot interfaceRobot interface
SDS APPLICATIONS
Automatic Speech Recognition (ASR)
FeatureExtraction
Decoding
AcousticModel
PronunciationModel
LanguageModel
버스 정류장이어디에 있나요 ?
Speech Signals Word Sequence
버스 정류장이어디에 있나요 ?
NetworkConstruction
SpeechDB
TextCorpora
HMMEstimation
G2P
LMEstimation
WO
)()|(maxargˆ WPWOPWLW
15
Spoken Language Understanding (SLU)
Dialog ActIdentificationDialog Act
IdentificationFrame-SlotExtraction
Frame-SlotExtraction
RelationExtractionRelation
Extraction
UnificationUnification
Feature Extraction / SelectionFeature Extraction / Selection
Info.SourceInfo.
Source
++
++
++
++ ++
Overall architecture for semantic analyzer
I like DisneyWorld.
Domain: ChatDialog Act: StatementMain Action: LikeObject.Location=DisneyWorld
Examples of semantic frame structure
Semantic Frame Extraction (~ Information Extrac-
tion Approach)1) Dialog act / Main action Identification ~ Classification
2) Frame-Slot Object Extraction ~ Named Entity Recognition
3) Object-Attribute Attachment ~ Relation ExtractionHow to get to DisneyWorld?Domain: NavigationDialog Act: WH-questionMain Action: SearchObject.Location.Destination=DisneyWorld
Named Entity ↔ Dialog Act
JOINT APPROACH
Joint Inference
Classification(Dialog Act / Intent)
Sequential Labeling
(Named Entity / Frame Slot)
Automatic Speech
Recognition
Dialog Management
Joint Model(e.g. TriCRFs)
x x,y,z
[Jeong and Lee, SLT2006][Jeong and Lee, IEEE TASLP2008]
HDP-HMM for Unsupervised Dialog Acts
β ~ GEM(α), ω ~ Dir(ω0)for each hidden state k [∈ 1,2,…] πk ~ DP(α',β) ϕk ~ Dir(ϕ0), θk ~ Dir(θ0)for each dialog d λd ~ Beta(λ0) for time stamp t zt ~ Multi(πzt-) for each entity e ei ~ Multi(θzt)
for each word w xi ~ Bern(λd) [select word type] if xi = 0: wi ~ Multi(ϕzt) else wi ~ Multi(ω) [background LM]
zt
wt,i
zt+1
et,i
N
V
ϕk
∞
πk
∞
ϕ0
α'
βα
θk
∞
θ0
zt-1
ωω0xt,i
Dλ0λd
Generative Story
CRF with Posterior Regularization for un-supervised NER Constraints for NER
Constraints Learning
Welcome to the New York City Bus Tour Center .I want to buy tickets for me and my child .What kind of tour would you like to take ?We would like to go on a tour dur-ing the day .We have two daytime tours: the Downtown Tour and the All Around Town Tour .Which tour goes to the Statue of Liberty ?…
BOARD_TYPE:Hop-onBOARD_TYPE:Hop-offPLACE:Times SquarePLACE:Empire State BuildingPLACE:ChinatownPLACE:Site of the World Trade CenterPLACE:Statue of LibertyPLACE:Rockefeller CenterPLACE:Central Park…
HeuristicMatch-
ing
DICT/DB/Web
UNLABELDCORPUS
# We would like to go on a tour during the day . # -> null0:1.000:We would like to go on a tour during the day . # We have two daytime tours # -> the Downtown Tour and the All Around Town Tour .0:1.000:We have two daytime tours # Which tour goes to the Statue of Liberty ? # -> null0:1.000:Which tour goes to the <PLACE>Statue of Liberty</PLACE> ? # You can visit the Statue of Lib-erty on either tour . # -> null0:1.000:You can visit the <PLACE>Statue of Liberty</PLACE> on either tour .…
HYPOTHE-SIS
Welcome O:1.000 W1=<s> O:0.997 PLACE-b:0.001 TOURS-b:0.002 GUIDE-b:0.001 W2=<s>,Welcome O:1.000 W3=_ O:0.997 PLACE-b:0.001 TOURS-b:0.002 GUIDE-b:0.001 W4=_ O:0.997 PLACE-b:0.001 TOURS-b:0.002 GUIDE-b:0.001 W5=_ O:0.997 PLACE-b:0.001 TOURS-b:0.002 GUIDE-b:0.001 W6=to O:1.000 W7=Welcome,to O:1.000 W8=the O:0.924 PLACE-b:0.005 PLACE-i:0.006 TOURS-b:0.001 TOURS-i:0.064 W9=Welcome,the O:1.000 …
LABELEDFEATURES
ExtractFeatures
CRFModel with PR
Vanilla EXAMPLE-BASED DM (EBDM) Example-based approaches
Dialog State Space
Domain = Building_GuidanceDialog Act = WH-QUESTIONMain Goal = SEARCH-LOCROOM-TYPE=1 (filled), ROOM-NAME=0 (unfilled)LOC-FLOOR=0, PER-NAME=0, PER-TITLE=0Previous Dialog Act = <s>, Previous Main Goal = <s> Discourse History Vector = [1,0,0,0,0]Lexico-semantic Pattern = ROOM_TYPE 이 어디 지 ?System Action = inform(Floor)
Dialog Corpus
USER: 회의 실 이 어디 지 ?[Dialog Act = WH-QUESTION][Main Goal = SEARCH-LOC][ROOM-TYPE = 회의실 ]SYSTEM: 3 층에 교수회의실 , 2 층에 대회의실 , 소회의실이 있습니다 . [System Action = inform(Floor)]
Turn #1 (Domain=Building_Guidance)
Dialog Example
Indexed by using semantic & discourse features
Having the simi-lar state
),(argmax* heSe iEei
[Lee et al., SPECOM2009]
Error handling and N-best support
To increase the robustness of EBDM with prior knowledge
1) Error Handling
If the system knows what the user will do next
Dynamic Help Generation
LOCATION
OFFICE PHONE NUMBER
ROOM ROLE
GUIDE
FOCUS NODE
NEXT_TASK
AgendaHelpS: Next, you can do the subtask 1) Asking the room's role, or 2)Asking the office phone num-ber, or 3) Selecting the desired room for navi-gation.
UtterHelpS: Next, you can say 1) “What is it?”, or 2) “What’s the phone number of [ROOM_NAME]?”, or 3) “ Let’s go there.
[Lee et al CSL2010]
Error handling and N-best support
To increase the robustness of EBDM with prior knowledge
2) N-best supportIf the system knows which subtask will be more probable next
Rescoring N-best hypotheses (h1~hn)
LOCATION
OFFICE PHONE NUMBER
FLOOR
ROOMNAME
h2
h1
h3
h4
Subtask System Utterance System Action
LOCATIONThe director’s room is Room No. 201.
Inform(RoomNumber)
N-best User Utterances Subtask P(hi|S)
U1 (h1)What are office rooms in this building?
ROOM NAME
0.2
U2 (h2) What is the floor? FLOOR 0.4
U3 (h3) Where is it? LOCATION 0.3
U4 (h4)What is the phone num-ber?
OFFICEPHONE NUMBER
0.5(More proba-
ble)
Misunderstanding handling by Confirma-tion
Dialog statehypotheses
ConfirmationAgent
(misunderstandingHandler)
EBDM
Multiple Dialog States
Representation
User Simulator
DEDB
ConfirmationStrategy
Confirmation
Task related system action
User
ASR
SLU
User’sActions
Executing Learning
[Kim et al SLT 2010]
The Framework of ranking-based EBDM
DiscourseSimilarity
Relative Position
Scoring Mod-uleDialog
Examples
Dialog ActFeatures
Entity Con-
straint
User Intention(system intention)
RankSVM
CalculatedScores
system Intention(user intention)
EBDM
[Noh et al IWSDS2011]
Dialog Simulation User Simulation for spoken dialog systems in-
volves four essential problems
User Intention Simulation
User Utterance Simulation
ASR Channel SimulationSpoken Dialog System Simulated Users
[Jung et al., CSL 2009]
Design Step
Annotation Step
LanguageSynchronization Step
Training Step
Running Step
Semantic Structure
Dialog Structure
KnowledgeStructure
ModelSLUModel
DialogModel
Knowledge
Model
ASRModel
CorpusSLU
CorpusDialogCorpus
Knowledge
Source
SemanticAnnotato
r
DialogAnnotato
r
KnowledgeAnnotator
DialogUtterance
Pool
KnowledgeImporter
KnowledgeBuilder
DMTrainer
SLUTrainer
ASRTrainer
SLU DMASR
ExternalComponen
tDialog Studio
Component
File
DIALOG STUDIO ARCHITECTURE
[Jung et al., SPECOM 2008]
humansubject Wizard
User speech
mic speaker
TTSText input
Wizard speech (Network RPC)
Architecture of WOZ
User Screen Wizard Screen
NPCsControl
User CharacterControl
[Lee et al SLATE2011]
User Screen (Mission)
DBCALL: EDUCATIONAL ERROR HANDLING
DBCALL: EDUCATIONAL ERROR HANDLING
CHAPTER 3
Global ErrorsGlobal Errors
• Global errors are errors that affect overall sen-tence organization. They are likely to have a marked effect on comprehension. [1]
What is the purpose of your trip?
It’s ... I ... purpose business
Sorry, I didn’t under-stand. What did you say?You can say “I am here on busi-ness”I am here on business
Intention: inform(trip-purpose)
Lee, S., Lee, C., Lee, J., Noh, H., & Lee, G. G. (2010). Intention-based Corrective Feedback Generation using Context-aware Model. Proceedings of International Conference on Computer Supported Education.
Hybrid ModelHybrid Model
Level 1Data
Learner’s Utterance
Dialog ContextModel
Level 2Utterance Model
Level NUtterance Model
Level 2Data
Level NData
Dialog State
Learner‘s Intention
Level 1Utterance Model
Dialog Manager
• Robust to learners’ errors– Hybrid model combining utterance-based model and dialog
context-based model
Formulating the prediction as probabilistic inference:Formulating the prediction as probabilistic inference:
Chain ruleBayes’ ruleIgnore invariants
Dialog-Context ModelUtterance ModelMaximum Entropy
Features: • Word• Part of speech
Enhanced K-Nearest Neighbors
Features: • Previous system intention• Previous user intention• Current system intention• A list of exchanged information• Number of database query results
Dialog State Space
Domain = Fruit_StorePrevious System Intention = Ask(Select_Item)Previous User Intention = Inform(Order_Fruit) System Intention = Ask(Order_Quantity)Exchanged Information State = [ITEM_NAME = ‘orange’ (C), ITEM_QUANTITY = 3 (U)]Number of DB query results = 0
Dialog Corpus
SYSTEM: Namsu, what would you like to buy today?[Intention = Ask(Select_Item)]USER: I’d like to buy some oranges[Intention = Inform(Order_Fruit), ITEM_NAME = orange]SYSTEM: How many oranges do you need?[Intention = Ask(Order_Quantity)]USER: I need three oranges[Intention = Inform(Order_Quantity), NUM = three]
Segment #2 (Domain = Fruit Store)
Dialog State
Indexed by using semantic & discourse features
User Intention = Inform(Order_Quantity)User Intention
Dialog-Context ModelDialog-Context Model
Recast Feedback GenerationRecast Feedback Generation
ExampleExpresssion DB
Example Search
Example Ex-pressions
Pattern Matching
Feedback
IntentionRecognition
User’sUtterance
> θ No Feedback
Y
N
What is the purpose of your trip?
I am here at business
On business
I am here on business
ErrorInfo: prep_sub(at/on)
Local Er-rors
Local Er-rors
• Local errors are errors that affect single elements in a sentence. [1]
[1] Ellis., R. (2008). The Study of Second Language Acquisition. 2nd ed. Oxford: OUP
Local Error Detecter Archi-tecture
Local Error Detecter Archi-tecture
Text
Erroneous Text
Grammatical ErrorSimulation
ASR ASR’
N-gram LM
Merged Hy-potheses
Error-typeClassifier
GrammaticalityChecker
N-gram LM
Feed-back
Error PatternsError Frequency
Lee, S., Noh, H., Lee, K., & Lee, G. G., (2011) Grammatical Error Detection for Corrective Feedback Provision in Oral Conversations, Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, San Francisco.
Two-Step ApproachTwo-Step Approach
• Data Imbalance Problem– Simply produce majority class– Or, High false positive rate
• Large number of error types – Makes model learning and selection procedure vastly compli-
cated• Grammaticality checking itself can be useful for some Ap-
plications– Categorizing learners’ proficiency level – Generating implicit corrective feedback such as repetition,
elicitation, and recast feedback
I am here at business
0 0 0 1 0
None None None PRP_LXC None
Grammaticality CheckingError Type Classification
Grammatical Error Detection
1)
2)
Grammaticality Checker
- Feature Extraction
Grammaticality Checker
- Feature Extraction
Grammaticality Checker
- Model Learning
Grammaticality Checker
- Model Learning• Binary Classification
– Support Vector Machine• Model Selection
– Radial Basis Kernel– Search for C, γ which optimize:
• Maximize F-scoreSubject to Precision > 0.90, False positive rate < 0.01
– 5-fold cross-validation
Error Type Classi-fication
Error Type Classi-fication
• Error type information is useful for– Meta-linguistic feedback– Sophisticated learner model
• Simplest way– Choose the error type associated with the top ranked er-
ror pattern– Two flaws:
• does not have a principled way to break tied error patterns• does not consider the error frequency
• Weighting according to error frequency– Score(e) = TS(e) + α * EF(e)
GES: Grammar Error Sim-ulator
GES: Grammar Error Sim-ulator
Automatic Speech Recog-
nizer
Grammatical Er-ror Simulator
Incorrect Sen-tences
Correct Sen-
tences
Error Types
<LM Adaptation & Grammatical Error Detection>
GES Applica-tion
GES Applica-tion
<Grammar Quiz Generation>
Markov Logic Net-work
Markov Logic Net-work
• subject-verb agreement errors• omission errors of prepositions• omission errors of articles
He want go to movie theater
Sungjin Lee, Gary Geunbae Lee. Realistic grammar error simulation using markov logic. Proceedings of the ACL 2009, Singapore, August 2009.Sungjin Lee, Jonghoon Lee, Hyungjong Noh, Kyusong Lee, Gary Geunbae Lee. (2011) Grammatical Error Simu-lation for Computer-Assisted Language Learning, Knowledge-Based Systems
Grammar Error SimulationGrammar Error Simulation
• Realistic errors– Encoding characteristics of learners’ errors using the Markov
logic
• Over-generalization of some rules of the L2
• Lack of knowledge of some rules of the L2
• Applying rules and forms of the first language into the L2
Overall ProcessOverall Process
NICT JLE CorpusNICT JLE Corpus
• Number of interviews– 167
• Number of sentences of intervie-wees– 8,316
• Average length of sentences– 15.59
• Nubmer of total errors– 15,954
<n_num crr=“x”>...</n_num>
POS(i.e. n=noun)
Grammatical system(i.e. num=number)
Corrected form
Erroneous part
Example) I belong to two baseball <n_num crr=“teams”>team</n_num>
PESAA: POSTECH ENGLISH SPEAKING ASSESSMENT & ASSISTANT
PESAA: POSTECH ENGLISH SPEAKING ASSESSMENT & ASSISTANT
CHAPTER 4
English oral proficiency assessment:International test
Reading aloud
Describing a picture
Answering to questions
Proposing a solution or opinion
Interview
Talking on a topic
Discussion
Giving an opinion
Talking on a subject
Answering to ques-tions
English oral proficiency assessment:Korean national test
• National English Ability Test (NEAT)
• Tasks– Answering short questions (communication)– Describing pictures (story telling)– Presentation
• Describing figures, tables, and graphs• Introducing products or events
– Giving an opinion (discussion)
English oral proficiency assessment:General common tasks
• Giving an opinion / discussion
• Rubrics– Delivery
• Pronunciation• Fluency (Prosody)
– Language use• Grammar• Word choice
– Topic development• Organization• Discourse• Contents
Requirements:Real environment
Reading aloud
Describing a picture
Answering to questions
Proposing a so-lution or opin-
ion
Interview
Talking on a topic
Discussion
Giving an opinion
Talking on a subject
Answering to ques-
tions
Existing systems for read speech
Spontaneous speechText-independent input
NEAT
51
Training data collection
• SNU pronunciation/prosody
Speech waveform
Spectrogram/ pitch contour
Word
PLU
Sentence stress
52
For Public Use
• Boston University radio news corpus– Speech from FM radio news announcers– 424 paragraphs (30,821 words)– ToBI labels (pitch accent stress)– 0.48 marked stress per word– PLU set: TIMIT phonetic labeling system
53
• Aix-Marsec database
Speech waveform
Spectrogram/ pitch contour
Multi-level annota-tion
Collecting Grammar Error Data:Picture description task
• From English learners of Korean• Story Telling based on pictures• 80 Students (5 tasks for each student)
Collecting Grammar Error Data: Error tagsets
• JLE Tagset– Consisting of 46 tags– Systematic tag structure– Some ambiguity caused by POS specific error tag structure
• CLC Tagset– World-widely used tagset including 76 tags– Systematic & Taxonomic tag structure– JLE issue is figured out by taxonomic tag structure
• NUCLE Tagset– 27 error tags– Quiet arbitrary tag structure
• UIUC Tagset– Only for articles and prepositions
PESAA: Pronuciation Feedback
EPD
Error information
UserUser
Forced Alignment
Comparison
Feedback Generation
Actual pronu
nciation
Speech input
MaterialMaterial
Error Detec-tion
Error candidates
Pronouncing Simulation
ASR
Word-level transcription
Orthographic pronunciation
simulation part
recognition part
error detection & feedback part
Pronunciation Error simulation:Pronunciation Variants
Canonical pronuncia-tionNative speaker’s pronuncia-tionNon-native speaker’s pronuncia-tion[straik]
[sɨtɨraikɨ]
Strike
Pronunciation Error simulation:Learning context rules using Generalized TBL
nth initial ma-chine annota-
tion
Collect transfor-mations
Best transforma-tion
List of trans-formations
Machine anno-tated data
Training in-put
Left-right ngram context
Iterative initialization
n := n + 1
Merge transforma-tions
Trainingreference
Majority choice/ Context
n := 0
nth order initial-ization rules
Apply
n
Pronunciation Error simulation:Multi-tag Result
• Example Input– Input
• Let’s go shopping• # L EH T S # G OW # SH AH P EH NG #
• Example Output– #/# L/L EH/EH T/T S/S #/# G/G OW/OW|AO #/# SH/SH AA/AH|AA P/P IH/IH NG/NG
#/#• #/# L/L EH/EH T/T S/S #/# G/G OW/AO #/# SH/SH AA/AA P/P IH/EH NG/NG #/#• #/# L/L EH/EH T/T S/S #/# G/G OW/OW #/# SH/SH AA/AA P/P IH/EH NG/NG #/#• #/# L/L EH/EH T/T S/S #/# G/G OW/AO #/# SH/SH AA/AH P/P IH/EH NG/NG #/#• #/# L/L EH/EH T/T S/S #/# G/G OW/OW #/# SH/SH AA/AH P/P IH/EH NG/NG #/#
Pronunciation Error detection/feedback
Error candi-date infor-
mation
Feedback pref-erence
Error confi-dence
Word ASR con-fidence
Phoneme ASR confidence
Feedback deci-sion
Feedback
Feedback DB
Pronunciation Error detection/Feedback:Components
Feedback preference
Error confi-dence
Phoneme ASR confidence
Word ASR con-fidence
)|Pr( xr
),,|Pr( 11 rhef ),,|Pr( 1 rhxe
)|Pr( xh
62
PESAA: Prosody Feedback
• Stress & Prosodic phrasing & boundary tone
Stress
Prosodic phrasing
Boundary tone
* Existence of word/sentence stress for each syllable/word
* Location of phrase breaks
* Type of boundary tone for each phrasal boundary
63
Sentence Stress Feedback:Architecture
Alignment
TextText
Analysis
Speech Analysis
Sentence Stress
Prediction
Model
Rule ApplicationRules
PredictedSentence
Stress
ModelTraining
Model
Sentence Stress
Detection
DetectedSentence
Stress
FeedbackDiff.
TextAnalysisText
Speech Signal
ModelTraining
64
Sentence Stress Prediction
• Feature used– Position info: the number
of phonemes in word, the number of syllables in word, …
– Stress info: word stress, sentence stress (rule-based prediction), …
– Lexical info: identity of word, identity of vowel
– Part-of-speech info
Name Description
S-basic Content words
U-basic Functional words
U-adhoc Unclassified FW EX LS POS
U-aux MD special cases
U-adv RP special cases
S-frgn FW foreign words
S-vb Last VB in multi-ple verbs
65
Sentence Stress Detection
• Feature used– Duration info: duration of vowel, duration of
syllable, normalized duration of word accord-ing to the number of syllables, …
– Intensity info: energy of vowel (+delta)– F0 info: f0 of vowel (+delta)– MFCC info: mfcc of vowel (+delta, +delta-
delta)– Lexical info: identity of vowel
66
Sentence Stress Feedback
• Adopting output probability– Feedback candidates: syllables in “predicted
stress” with low or high output probability
Pre-dicted stress
It may
be
the
most
im por tan
t ap point
ment
De-tected stress
It may
be
the
most
im por tan
t ap point
ment
Not stressed
Stressed
67
Sentence Stress Feedback:Snapshot
PESAA: Grammar Feedback
Spoken English
Written English
User Input
GE Pat-terns
Spoken GE Simu-
lator
GE tagged Texts/
SpeechTraining
Soft Constraint
Correct Sentences
Spoken GE Detec-
tor
SVMTraining
ASR/CNSPEEC
H
Written GE Detec-
tor
GE tagged Texts
Written GE
SimulatorTraining
Soft Constraint
Correct Sentences
GE Pat-terns
SVMTraining
TEXTGE Feed-
back
Grammar Error detection:Snapshot – written input
Grammar Error detection:Snapshot – spoken input
FIELD STUDYFIELD STUDY
CHAPTER 5
Field Study: Robot-Assisted Language Learning
Experimental Design1
2 Cognitive Effects
Affective Effects3
Sungjin Lee, Hyungjong Noh, Jonghoon Lee, Kyusong Lee, Gary Geunbae Lee, Seongdae Sagong, Moon-sang Kim. (2011) On the Effectiveness of Robot-Assisted Language Learning, ReCALL Journal, Vol.23(1), SSCI.Sungjin Lee, Changgu Kim, Jonghoon Lee, Hyungjong Noh, Kyusong Lee, Gary Geunbae Lee.Affective Ef -fects of Speech-enabled Robots for Language Learning. Proceedings of the 2010 IEEE Workshop on Spoken Language Technology (SLT 2010), Berkeley, December 2010Sungjin Lee, Hyungjong Noh, Jonghoon Lee, Kyusong Lee, Gary Geunbae Lee. Cognitive Effects of Robot-Assisted Language Learning on Oral Skills. Proceedings of Interspeech Second Language Studies Workshop, Tokyo, Sep 2010.
HRI Technol-ogy
HRI Technol-ogy
HRI Experimental Design
HRI Experimental Design
• Setting and participants– 24 elementary students– Ranging in age over 9-13– Divided into two groups (beginner, intermedi-
ate)
• Material and treatment– 68 lessons
• 17 lessons for each level and theme– Simple to complex task– 2 hours a week extended over 8 weeks
HRI Experimental Design
HRI Experimental Design
1) PC room
2) Pronunciation training room
3) Fruit and Vegetablestore
4) Stationerystore
Evaluation of Cognitive Effects
Evaluation of Cognitive Effects
• Data collection and analysis
– Evaluation method• Pre-test/Post-test
– For the listening skills• 15 items for multiple choice question• Cronbach’s alpha
– pre-test: 0.87, post-test: 0.66
– For the speaking skills• 10 items for 1-on-1 interview• Cronbach’s alpha
– pre-test: 0.93, post-test: 0.99
<Cognitive effects on oral skills for overall students>
Experiment Result
Experiment Result
*p < .05
Evaluation of Affective Factors
Evaluation of Affective Factors
• Data collection• Questionnaire (4 point scale without a neutral option)
• Data analysis– For satisfaction in using robots
• Descriptive statistics– For interest in learning English, Confidence with English,
Motivation for learning English• Pre-/Post-test
Affective Factor N Ɨ R ƗƗ
Satisfaction in using robots 10 0.73
Interest in learning English 16 0.93(0.96)
Confidence with English 12 0.91(0.90)
Motivation for learning English 14 0.91(0.83)
N Ɨ = Number of questions, R ƗƗ = Cronbach’s alpha in the form of pre-test(post-test)
Effects on Affective Factors
Effects on Affective Factors
Satis
fact
ion
in u
sing
robo
ts
Inte
rest
in le
arni
ng E
nglis
h
Confid
ence
with
Eng
lish
Mot
ivat
ion
for l
earn
ing
Engl
ish
0
1
2
3
4
Pre-testPost-test
Thank you
Thank you
Top Related