Kate’s Ongoing Work on Uncertainty Adaptation in ITSPOKE.
-
Upload
gordon-mccormick -
Category
Documents
-
view
221 -
download
0
Transcript of Kate’s Ongoing Work on Uncertainty Adaptation in ITSPOKE.
Kate’s Ongoing Work on
Uncertainty Adaptation in ITSPOKE
Outline
PSLC 2006 - 2007 Project
ALT 2007 - 2009 Project
Differences from PSLC Project
NAACL-HLT 2007 Study
PSLC Project: Background Does Treating Student Uncertainty as a Learning Impasse
Improve Learning in Spoken Dialogue Tutoring?
Learning Impasse (VanLehn, Siler & Murray, 2003):
“occurs when a student realizes that he or she lacks a complete understanding of a specific piece of knowledge [...] when a student gets stuck, detects an error, or does an action correctly but expresses uncertainty about it”
uncertain answers and wrong answers
PSLC Project: Background Does Treating Student Uncertainty as a Learning Impasse
Improve Learning in Spoken Dialogue Tutoring?
Uncertainty relates to learning (Craig et al. 2004):
positively correlates with learning in AutoTutor
Hypothesis: uncertainty can be accompanied by deliberation/inquiry aimed at increasing understanding
PSLC Project: Hypothesis Does Treating Student Uncertainty as a Learning Impasse
Improve Learning in Spoken Dialogue Tutoring?
uncertain: opportunity to learn something incorrect: opportunity to learn something
Most tutoring systems respond only to correctness
Our hypothesis: treating uncertainty as a learning impasse will increase learning
PSLC Project: Experimental Condition Keep ITSPOKE response to incorrect answers:
Simple questions: give answer and move on
Complex questions: remediation dialogue with more questions to increase understanding, then move on
ADD ITSPOKE response to uncertain answers:
Respond as if answer were incorrect
Keep ITSPOKE response to other correct answers:
move on to next question
PSLC Project: Control Condition #1
Keep ITSPOKE response to incorrect answers
Keep ITSPOKE response to correct answers
NO ITSPOKE response to uncertainty
PSLC Project: Control Condition #2 Keep ITSPOKE response to incorrect answers
Keep ITSPOKE response to 85% correct answers
CHANGE ITSPOKE response to a random 15% of correct answers
Respond as if answer were incorrect
15% of student answers are uncertain (HC-wade)
control for extra tutoring in experimental condition; should be less effective than targeting uncertainty
PSLC Project: Experimental Procedure Read background material (reduced) Take pretest (reduced) Work 1st problem with ITSPOKE (#58) Take posttest (reduced)
Tests modified to be “fill in blank” (no chance correct) May not see significant learning gains after 1 problem
Work isomorphic problem with ITSPOKE (#59) Experimental condition may have more correct answers
20 subjects/condition (low pretest only); 2 hours
PSLC Project: Isomorphic Problems Problem #58: Suppose a man is in a free-falling elevator and
is holding his keys motionless right in front of his face. He then lets go. What will be the position of the keys relative to the man's face as time passes? Explain.
Problem #59: A professor is sitting in his armchair holding his spectacles motionless in front of his face, in order to inspect how well he just cleaned them. All of a sudden, an earthquake causes his office floor to collapse. The professor is startled when he finds that both he and his armchair are in free-fall, and he drops his spectacles. What will be the position of the spectacles relative to the professor's face as time passes? Explain.
PSLC Project: Implementation WOZ version of ITSPOKE (“ideal” system conditions)
Speech input/output No essay (everyone gets walk-through)
Human Wizard Perform speech recognition and NLU (select anticipated
response components) Label uncertainty for each condition (checkbox; data)
Remaining Issues: students hit stop/start buttons when speak? random 15% of correct turns (non-uncertain?) need piloters so we can check logs etc ;)
PSLC Project
DEMO
ALT Project: Differences from PSLC
Adapting to Student Uncertainty over and above Correctness in a Spoken Tutoring Dialogue System
Experiment 1: 4 conditions, WOZ, all 5 problems Adaptation 1: treat uncertainty as learning impasse
(baseline/obvious/least-change strategy) Adaptation 2: new system responses to uncertain +
correct and uncertain + incorrect student answers, based (in part) on analysis of human tutor responses
Hypothesis: significantly improved learning in adaptive conditions as compared to controls (same 2 as PSLC)
ALT Project: Differences from PSLC
Adapting to Student Uncertainty over and above Correctness in a Spoken Tutoring Dialogue System
Experiment 2: 3 conditions, ITSPOKE, all 5 problems Adaptation: “best-performing” adaptation from Exp1
via learning, dialogue efficiency, survey, etc automatic uncertainty detection automatic speech recognition/NLU
Hypothesis: significantly improved learning in adaptive condition as compared to controls (same 2), but less improvement as compared to WOZ
ALT Project: Adaptation 2 Adaptation 2: new system responses to uncertain + correct and
uncertain + incorrect based (in part) on human tutor responses Pon-Barry et al., 2006: implemented human tutor responses,
but didn't find increased learning in experimental condition uncertain + correct: paraphrase answer
corpus#1: 4/5 and 0/34 other corrects corpus#2: 10% and 1% other corrects
uncertain + incorrect: refer to past dialogue corpus#1: 3/10 (point out misconception, follow
incorrect reasoning, state answer) and 0/7 other corrects
corpus#2: 30% and 5% other incorrects Human tutors use multiple strategies depending on context
NAACL07: Towards Adaptation 2 Exploring Context-Dependent Affect Adaptations in Spoken
Dialogue Systems (with Mihai, Diane, Joel) Proposal: develop multiple adaptations for affective state,
whose use depends on prior context in which state occurs Motivations:
Human tutors seem to do this (Pon-Barry et al., 2006) Prior work largely context-independent and no improved
learning (e.g., Aist et al., 2003, Pon-Barry et al., 2006) Call Center: apologize after user frustration, except if
multiple frustrated turns in row give to human operator
NAACL07: Towards Adaptation 2 Exploring Context-Dependent Affect Adaptations in Spoken
Dialogue Systems (with Mihai, Diane, Joel) Questions: in what contexts to adapt, and what strategy to
use in each context? (not clear for uncertainty in tutoring) Our Empirical Approach:
Apply Chi2 test is system corpora to find contexts significantly associated with increased/decreased affect
Formulate hypotheses explaining association Base context-dependent adaptations on hypotheses Refine adaptations via analysis of human tutor responses
NAACL07: Towards Adaptation 2 Apply approach in ITSPOKE Targeted affect: uncertain+correct, uncertain+incorrect Contexts are question properties: Question Act, Discourse
Structure Depth/Transition, Goal Repetitions “Tutor Question – Student Answer”, but 275 questions
Results: multiple contexts significantly associated with increased
or decreased uncertainty states hypotheses explain association and suggest adaptations
FW: Refine via analysis of human tutor response in contexts
NAACL07: Annotated ExampleSYS4: Good. Does the packet have an acceleration? If yes please specify its
direction. [SAQ, Advance, D1, GR0]
STUDENT4: horizontally forward yes [other+incorrect]
SYS5: Well... To find the packet's acceleration we will use Newton's
second law. What is the definition of this law? [LAQ, Push, D2, GR0]
STUDENT5: an object in motion tends to stay in motion until it's act
by an outside force [uncertain+incorrect]
SYS6: Well... [...] What can we determine about the acceleration of the
packet given that we know the net force acting on it is not zero and
points straight down? [DAQ, Advance, D2, GR0]
STUDENT6: I don't know [uncertain+incorrect]
SYS7: Well... [...] Does the packet have an acceleration? If yes please specify its direction. [SAQ, PopUp, D1, GR1]
Question Act Dependencies (p<.05)
LAQ/DAQ are hard questions: definition/deep reasoning Adaptation: reformulate as ``fill in blank” SAQ to work
through uncertainty by (also) answering simpler version correctly
Dependency Obs. Exp. Chi^2QACT ~ SSTATE 203.38LAQ ~ uncC + 72 22 133.98RPT ~ uncC - 7 14 4.15SAQ ~ uncC - 285 328 41.95DAQ ~ uncI + 37 27 4.57LAQ ~ uncI + 43 27 11.17SAQ ~ uncI - 377 408 17.1
Question Act Dependencies (p<.05)
SAQ are easier “fill in blank” questions: uncertain students may be having difficulty with path to solution
Adaptation: remind how SAQ relates to overall solution (and/or monitor frequency of uncertainty after SAQs)
Dependency Obs. Exp. Chi^2QACT ~ SSTATE 203.38LAQ ~ uncC + 72 22 133.98RPT ~ uncC - 7 14 4.15SAQ ~ uncC - 285 328 41.95DAQ ~ uncI + 37 27 4.57LAQ ~ uncI + 43 27 11.17SAQ ~ uncI - 377 408 17.1
Depth Dependencies (p<.05)
Deeper depths correspond to deeper knowledge gaps about topics in solution (+otherI, -otherC): uncertainty may concern perceived lack of cohesion between sub-topic and larger topic/solution
Adaptation: show how sub-topic fits in to overall solution
Dependency Obs. Exp. Chi^2D# ~ SSTATE 53.85D1 ~ uncC + 250 228 5.46D2 ~ uncC - 78 101 7.8D1 ~ uncI - 230 283 27.55D2 ~ uncI + 156 125 11.26D3+ ~ uncI + 93 70 9.76
Depth Dependencies (p<.05)
D1 on direct path to solution (+otherC, -otherI): uncertainty may relate to difficulty understanding necessary steps on path
Adaptation: reminder of steps on path, and/or monitor frequency of uncertainty at D1: occasional may be constructive (VanLehn et al., 2003) but frequent may be destructive and require intervention
Dependency Obs. Exp. Chi^2D# ~ SSTATE 53.85D1 ~ uncC + 250 228 5.46D2 ~ uncC - 78 101 7.8D1 ~ uncI - 230 283 27.55D2 ~ uncI + 156 125 11.26D3+ ~ uncI + 93 70 9.76
Transition Dependencies (p<.05)
Pushes correspond to deeper knowledge gaps (+otherI): uncertainty may concern perceived lack of cohesion between sub-topic and larger topic/solution (like D<1)
Adaptation: show how sub-topic fits in to overall solution
Dependency Obs. Exp. Chi^2TRAN ~ SSTATE 190.97PopUp ~ uncC - 23 36 5.89Advance ~ uncI - 223 254 9.06NewTopLvl ~ uncI - 49 67 6.47PopUp ~ uncI - 32 45 4.68PopUpAdv ~ uncI + 32 17 16.22Push ~ uncI + 100 70 16.37SameGoal ~ uncI + 43 25 14.24
Transition Dependencies (p<.05)
After PopUpAdv, uncertainty may relate to losing track of original question (not +otherI); after PopUp, to goal of sub-topic question(s)
Adaptation: remind how PopUpAdv relates to original question; clarify PopUp's subtopic goal
Dependency Obs. Exp. Chi^2TRAN ~ SSTATE 190.97PopUp ~ uncC - 23 36 5.89Advance ~ uncI - 223 254 9.06NewTopLvl ~ uncI - 49 67 6.47PopUp ~ uncI - 32 45 4.68PopUpAdv ~ uncI + 32 17 16.22Push ~ uncI + 100 70 16.37SameGoal ~ uncI + 43 25 14.24
Transition Dependencies (p<.05)
Uncertainty may relate to current topic/depth(-otherI) Adaptation: can be based on other contexts simultaneously
monitored (e.g., Depth, QACT)
Dependency Obs. Exp. Chi^2TRAN ~ SSTATE 190.97PopUp ~ uncC - 23 36 5.89Advance ~ uncI - 223 254 9.06NewTopLvl ~ uncI - 49 67 6.47PopUp ~ uncI - 32 45 4.68PopUpAdv ~ uncI + 32 17 16.22Push ~ uncI + 100 70 16.37SameGoal ~ uncI + 43 25 14.24
Transition Dependencies (p<.05)
SameGoal associated with increased incorrect overall; RPT with decreased correct: if student repeats answer, uncertainty may relate to original answer
(Rotaru and Litman, 2006): RPTs should be removed
Dependency Obs. Exp. Chi^2TRAN ~ SSTATE 190.97PopUp ~ uncC - 23 36 5.89Advance ~ uncI - 223 254 9.06NewTopLvl ~ uncI - 49 67 6.47PopUp ~ uncI - 32 45 4.68PopUpAdv ~ uncI + 32 17 16.22Push ~ uncI + 100 70 16.37SameGoal ~ uncI + 43 25 14.24
Goal Repetition Dependencies (p<.05)
Uncertainty may be increased simply because students haven't seen question before (not +otherC); some may be constructive
Adaptation: can be based on continued uncertainty at GR0 (monitor frequency like after SAQs, D0 )
Dependency Obs. Exp. Chi^2GR# ~ SSTATE 20.59GR0 ~ uncC + 301 281 5.46GR2 ~ uncC - 7 14 3.93
Goal Repetition Dependencies (p<.05)
Most students may remember answer from last time it was asked (no uncertainty) (not +otherC)
Adaptation: remind students about last time question asked (akin to Pon-Barry et al., 2006)
Dependency Obs. Exp. Chi^2GR# ~ SSTATE 20.59GR0 ~ uncC + 301 281 5.46GR2 ~ uncC - 7 14 3.93
Conclusions & Future Work Approach shows dependencies between uncertainty states
and contexts, suggests hypotheses leading to adaptations Will refine adaptations by analysing human tutor
responses to uncertainty states in these contexts Will study other automatically-monitorable contexts:
Topic Repetition: track similar + identical questions about topics, within and across 5 SYS problems (e.g. gravity, freefall); topic-dependent adaptations can be derived if uncertainty relates to specific topics
Will develop Combined/Precedence rules when multiple contexts trigger context-dependent adaptations