Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott...
-
Upload
darren-harrington -
Category
Documents
-
view
221 -
download
0
Transcript of Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott...
![Page 1: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/1.jpg)
Research Methods Tutoring in the classroom
Peter Wiemer-Hastings
David Allbritton
Elizabeth Arnott
Oussama BenKhadra
Jesse Efron
![Page 2: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/2.jpg)
Overview• Background of AI in Education and
Intelligent Tutoring Systems• On human learning• AutoTutor• RMT
• Architecture• Techniques• Deployment• Evaluation
![Page 3: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/3.jpg)
Overview of AIEd
• The goal: Interact with student to increase learning
• The ideal system:• Models domain knowledge• Models student knowledge• Adapts to student• Adapts its approaches to optimize
pedagogy
![Page 4: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/4.jpg)
Some influential systems
• Algebra tutor
• Project Listen
• Scenario-based interaction (e.g. Wolf)
• Inquiry-based learning
![Page 5: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/5.jpg)
What affects learning?
• Motivation
• Being “constructive”
• Doing example problems
• On the other hand: the Hawthorne effect
![Page 6: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/6.jpg)
Bloom's hierarchy (1956)
• Knowledge: recall facts
• Comprehension: understanding info
• Application: ability to use info
• Analysis: see patterns, organization
• Synthesis: use old ideas to create new
• Evaluation: compare, discriminate, assess
![Page 7: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/7.jpg)
Human tutors• Much better than classroom practice: up
to 2 standard deviations• Rely on doing problems with their
students• Don’t actually have deep understanding of
their students• Use basic dialog moves: question, pump,
prompt, hint, summarize• Have a script of topics to cover
![Page 8: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/8.jpg)
AutoTutor
• Started in 1997
• Models (non-expert) human tutors
• Talking head “incarnates” the tutor
• Uses dialog moves above
• Evaluates student answers by comparing to expected answers
![Page 9: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/9.jpg)
Latent Semantic Analysis
• Take a large corpus of texts
• Make a matrix of words in documents (paragraphs)
• Weight by inverse frequency
• Do singular value decomposition to “re-orient” the dimensions, reduce the noise
• Use vectors to represent “meaning” of words and texts.
• Use cosine to compare meanings.
![Page 10: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/10.jpg)
RMT
• Follows the AutoTutor approach
• Re-implemented in Lisp
• Interacts via the internet
• Includes spell-checking and synonym substitution
![Page 11: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/11.jpg)
Architecture
• Dialog manager handles logins, providing tutor utterances, collecting student responses.
• Responses evaluated with NLU components (LSA, aspell)
• Dialog Transition Network determines tutor’s next response
![Page 12: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/12.jpg)
![Page 13: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/13.jpg)
![Page 14: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/14.jpg)
DATN
• Graphical representation of tutor’s behavior
• Directly controls the tutor’s choice of response.
![Page 15: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/15.jpg)
Curriculum Script
• 5 topics: Ethics, types of studies, variables, reliability, validity
• Over 4000 lines• Each item contains:
• Question• (Picture)• Target good answers• Associated prompts, hints, questions• Bad answers• Summary
![Page 16: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/16.jpg)
RM Curriculum• Ethics of psychological experiments
• privacy, informed consent, ethics code, costs versus benefits
• Types of studies• experiments and correlational (observational, archival, case
studies, survey research)
• Variables• independent and dependent, operationalization, levels,
causality, confounding, categorical and quantitative variables, scales
• Reliability• systematic and random error, test-retest, internal
consistency, factors that affect reliability
• Validity• types of validity (internal, construct, external, statistical),
threats to validity.
![Page 17: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/17.jpg)
Types of student tasks
• Conceptual: facts, definitions, issues
• Analytic: Given a scenario, evaluate weaknesses, or fill in missing slots
• Synthetic: Given a task, create an approach (frame and slots)
![Page 18: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/18.jpg)
Example conceptual dialog
• Tutor: What does the principle of justice imply about how research should be done?
• Student: No group should have to take more risk than another group.
• Tutor: Good. What else?
• Student: The benefits of the research should be available to everyone
![Page 19: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/19.jpg)
Analytical example• Tutor: I'm going to tell you about a research project
someone conducted that has some ethical problems, and then you will tell me what the researcher should have done differently. OK? Here is how the project was done. Joe, a graduate student, did a study of high school students' attitudes about racism. He created a survey about racism and gave it out to 100 high school students. Before he gave out the survey, he got permission from his dissertation advisor and from the principal of the high school to give out an anonymous survey to the students. He also made sure each student read and signed a consent form that was attached to the survey. What should Joe have done differently to make the project comply with the requirements for ethical practices in research?
• Student: Huh?
![Page 20: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/20.jpg)
Synthetic example• Now let's practice designing a study. I'll give you
a hypothesis to test, and tell you what resources you have available. Then you design a study to test the hypothesis. Let's say that you want to find out whether reading a humorous greeting card causes people to be more likely to buy humorous magazines - that is your hypothesis. You have an agreement with a local magazine shop that allows you to use some of their customer purchase records for your research. …How will you set up the study?
![Page 21: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/21.jpg)
Demo
![Page 22: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/22.jpg)
Cool stuff on the way
• Shallow thinking traps
• Dialog Chains
• Research supervisor scam
![Page 23: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/23.jpg)
Leading students astray
• Failure to “get it right” strengthens learning
• Shallow thinking traps (STTs) in content material provide opportunity for failure
• Dialog chains (DCs) should get students back on track• A DC has a trigger and a network of
student responses and tutor questions
![Page 24: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/24.jpg)
Different tutoring style
• Traditional:• Tutor (usually) asks the questions and the
student (hopefully) answers.
• Pseudo-real world:• The tutor is disguised as a research
supervisor• Asks questions to assess student
knowledge (conceptual, analytic)• Assigns research tasks (analytic, synthetic)• Evaluates student’s work performance
![Page 25: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/25.jpg)
To the classroom
• Use in association with two core courses in Psychology next term
• Students use the system after they’ve covered the topic in the course
![Page 26: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/26.jpg)
In-course evaluation
• Pilot testing on various aspects
• 2x2 design: • Tutor or static information• Traditional tutor or research supervisor
• Pretests and posttests
![Page 27: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/27.jpg)
Pilot testing
• Questions:• 1. How easy/difficult was it for you to use the tutor?
• VERY EASY 1——2——3——4——5——6 VERY DIFFICULT
• 2. Was the tutor’s speech easy/difficult to understand?
• 1. How easy/difficult was the material presented by the tutor?
• N. Did you feel like you may learn/may not learn something during a tutoring session?
![Page 28: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/28.jpg)
Early results
• In general, subjects disliked the computerized speech -
• -Every subject in the "Miyako" mentioned her voice as something they did not like, and something that could be improved.
• "It was a female tutor with a male voice. Made it annoying."
• "Try to use a less computerized voice."• "The animation was fine, but the woman's
voice was annoying.“
![Page 29: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/29.jpg)
More results
• In general, subjects liked the animation - the only problem seemed to be that Miyako appears on top of the response box. Ss have to keep moving her back to the center of the screen. Merlin seemed to be the favorite.
• "I liked his facial expressions."• "He was funny looking. Relieved stress in a
way."• "Liked his outfit."
![Page 30: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/30.jpg)
Even more
• In general, subjects liked the feedback given and the hints and prompts and felt that the level of the material was appropriate
• "It was easy because he asked a question and in case you got it wrong, he will help you to get to the right answer.
• "The feedback was helpful."• "It didn't just give you the right answer if you
got it wrong."
![Page 31: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/31.jpg)
Some suggestions
• -Keep questions on the screen in the dialog box in case the student misses what the tutor asks.
• -Improve voices on the animated characters.• -Move Miyako so that she appears in the center of
the screen.• -Provide instructions at the beginning; a longer intro.• -More explanation of right answer.• -Sometimes the tutor asks the question before giving
the details of the experiment.
![Page 32: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/32.jpg)
Things they liked:
• -The feedback
• -The hints and prompts
• -Multiple chances to get the right answer
• -Various ways of asking questions
• -Lead me to the right answer
• -Merlin animation
![Page 33: Research Methods Tutoring in the classroom Peter Wiemer-Hastings David Allbritton Elizabeth Arnott Oussama BenKhadra Jesse Efron.](https://reader036.fdocuments.net/reader036/viewer/2022070408/56649e625503460f94b5e507/html5/thumbnails/33.jpg)
Issues
• For the research supervisor, does text-only make more sense?
• If not, what head do we give it?
• Is the supervisor scenario realistic enough to motivate students?
• Will it work?