Predicting Student Performance in Solving Parameterized Exercises
-
Upload
sherry-sahebi -
Category
Data & Analytics
-
view
82 -
download
0
Transcript of Predicting Student Performance in Solving Parameterized Exercises
Predicting Student Performance in Solving Parameterized Exercises
Shaghayegh Sahebi (Sherry)1, Yun Huang1, and Peter Brusilovsky1,2
1 Intelligent Systems Program, University of Pittsburgh2 School of Information Sciences, University of Pittsburgh
Predicting Student Performance in Solving Parameterized Exercises 2Shaghayegh Sahebi (Sherry)
Parameterized Questions
• One question template with multiple parameter sets– One template generates many questions– Each question be repeated multiple times by the
same student– Makes cheating difficult– The student can learn by practicing over time
Predicting Student Performance in Solving Parameterized Exercises 3Shaghayegh Sahebi (Sherry)
A parameterized question from QuizJET
Predicting Student Performance in Solving Parameterized Exercises 4Shaghayegh Sahebi (Sherry)
The Challenge
• Unproductive repetitions – Students who are not good in managing their
learning [Hsiao et. al, 2009]
• How to avoid this?– Personalized e-learning system– Predict the success of students’ future attempts
the same way as recommender systems– Predicting students’ performance (PSP)
Predicting Student Performance in Solving Parameterized Exercises 5Shaghayegh Sahebi (Sherry)
PSP for parameterized questions: how is it different from static questions?
• In static questions, the student solves a problem once– No attempt sequence on each question– Time-ignorant methods work well• Collaborative filtering approaches
• Assumption in parameterized questions: the student can learn by practicing over time– Attempt sequence for each student on each
question
Predicting Student Performance in Solving Parameterized Exercises 6Shaghayegh Sahebi (Sherry)
Our Goal
• To study the – recommender systems approaches – effect of attempt sequence
in PSP for parameterized questions
• Approaches: – Bayesian Knowledge Tracing (BKT)– Performance Factor Analysis (PFA)– Bayesian Probabilistic Matrix Factorization (BPMF)– Bayesian Probabilistic Tensor Factorization (BPTF)– Max baseline
Predicting Student Performance in Solving Parameterized Exercises 7Shaghayegh Sahebi (Sherry)
Bayesian Knowledge Tracing (BKT)
• Markov Model with two states• Models attempt sequence explicitly
K K K
Q Q Q
Initial knowledge
LearningP(T)
P(G),P(S)
Predicting Student Performance in Solving Parameterized Exercises 8Shaghayegh Sahebi (Sherry)
Performance Factor Analysis (PFA)
• Regression model
• No attempt sequencing but implicitly models attempt history
Predicting Student Performance in Solving Parameterized Exercises 9Shaghayegh Sahebi (Sherry)
Matrix Factorization (BPMF)
• From collaborative filtering • No attempt sequence modeling• We use Bayesian Probabilistic Matrix
Factorization (BPMF) [Xiong et al., 2010]
• Other models were used for static questions [Thai-Nghe et al., 2011]
1 0 0 01 1 0 10 0 1 10 0 0 1St
uden
ts
Questions/ topics
0.9
0
1.5
0.4
0 1.4
0 0.9
Stud
ents
KCs
0.8
0.5
0 0.3
0 0 0.5
0.8
KCs
Questions/ topics
Predicting Student Performance in Solving Parameterized Exercises 10Shaghayegh Sahebi (Sherry)
3D-Tensor Factorization (BPTF)
• Adds attempt sequence modeling to BPMF• We use Bayesian Probabilistic Tensor
Factorization (BPTF)• Other models used for static questions
Stud
ents
Time
Questions/ topics
…
Predicting Student Performance in Solving Parameterized Exercises 11Shaghayegh Sahebi (Sherry)
Max Baseline
• Predicting success (majority class) for every attempt
Predicting Student Performance in Solving Parameterized Exercises 12Shaghayegh Sahebi (Sherry)
Data
• From QuizJET system• Java Programming Questions• Six semesters• 166 Students• 103 questions• 69.04% success records (majority class)
Predicting Student Performance in Solving Parameterized Exercises 13Shaghayegh Sahebi (Sherry)
Study Setup
• Time-aware methods:– BKT: explicitly– PFA: counting previous success/failure– BPTF: student’s performance changes smoothly over time
• Time-ignorant methods:– Matrix factorization (BPMF)– Max baseline
• Collaborative filtering approaches:– Tensor factorization (BPTF)– Matrix factorization (BPMF)
• Knowledge component: question• 5-Fold user-stratified cross validation
– 80% of users in train data, rest in test data
Predicting Student Performance in Solving Parameterized Exercises 14Shaghayegh Sahebi (Sherry)
Results
Predicting Student Performance in Solving Parameterized Exercises 15Shaghayegh Sahebi (Sherry)
Time-aware methods perform better that matrix factorization
BKT PFA BPTF BPMF Max-Baseline66
67
68
69
70
71
72
73
74
75
76
Accuracy
Predicting Student Performance in Solving Parameterized Exercises 16Shaghayegh Sahebi (Sherry)
BKT overestimate student’s performance
BKT PFA BPTF900
950
1000
1050
1100
1150
1200
False Positive
Predicting Student Performance in Solving Parameterized Exercises 17Shaghayegh Sahebi (Sherry)
PFA and BPMF underestimate student’s performance
BKT PFA BPTF0
50
100
150
200
250
300
350
400
450
False Negative
Predicting Student Performance in Solving Parameterized Exercises 18Shaghayegh Sahebi (Sherry)
PFA predicts success better
BKT PFA BPTF BPMF0
20
40
60
80
100
120
140
Minority RecallMajority Precision
Predicting Student Performance in Solving Parameterized Exercises 19Shaghayegh Sahebi (Sherry)
BKT predicts failure better
BKT PFA BPTF BPMF0
20
40
60
80
100
120
140
160
180
Majority RecallMinority Precision
Predicting Student Performance in Solving Parameterized Exercises 20Shaghayegh Sahebi (Sherry)
Conclusion
• Attempt sequence is important in PSP for parameterized questions
• Recommender systems approaches are as good as the pioneers PSP methods – if they consider attempt sequence– Do not need to know the exact Knowledge
Components– Encourages more research on applying more
recommendation techniques in PSP
Predicting Student Performance in Solving Parameterized Exercises 21Shaghayegh Sahebi (Sherry)
Future work
• Other collaborative filtering approaches
• Ensemble of approaches
• Effect of knowledge structure (our AIEDCS paper)
• Personalize students’ experience according to our results
Predicting Student Performance in Solving Parameterized Exercises 22Shaghayegh Sahebi (Sherry)
Thank You!
Predicting Student Performance in Solving Parameterized Exercises 23Shaghayegh Sahebi (Sherry)
Implementation
• EM algorithm for BKT and set the initial parameters as follows: p(L0) = 0:5 , p(G) = 0:2 , p(S) = 0:1 , p(T) = 0:3 . For running PFA, we use
• the implementation of logistic regression in WEKA [3].
• For BPTF and BPMF: Matlab code prepared by Xiong et. al. We experimented with different latent space dimensions for BPTF and BPMF (5, 10, 20 and 30) and chose the best one, which has the latent space dimension of 10
Predicting Student Performance in Solving Parameterized Exercises 24Shaghayegh Sahebi (Sherry)
Predicting Students’ Performance
• Predicting the student’s capability to solve a problem or perform an educational task, mostly based on her performance in the past
• Predicting success/failure in solving a question
• Questions can be related to topics (Here, each topic can have multiple questions and each question is related to one topic)
Predicting Student Performance in Solving Parameterized Exercises 25Shaghayegh Sahebi (Sherry)
Results
No significant accuracy difference between all methods except BPMF and Max Baseline (P<0.05)
Predicting Student Performance in Solving Parameterized Exercises 26Shaghayegh Sahebi (Sherry)
Results
Predicting Student Performance in Solving Parameterized Exercises 27Shaghayegh Sahebi (Sherry)
Results
PFA tends to predict more failures for the students.
Predicting Student Performance in Solving Parameterized Exercises 28Shaghayegh Sahebi (Sherry)
Results
If BKT predicts a failure for a student, this prediction is more likely to be true compared to the other methods
Predicting Student Performance in Solving Parameterized Exercises 29Shaghayegh Sahebi (Sherry)
Results
if PFA predicts a success for a student, this prediction is more likely to be true compared to the other methods