Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment...
Transcript of Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment...
![Page 1: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/1.jpg)
Using Assessment System Data to GenerateIndividualized Learning Materials for
StudentsWhat We Learned from Developing the iDAP
Alex Brodersen & Ying ChengLAMBS lab, Department of Psychology
University of Notre Dame
A presentation at the 19th MARC conference, Nov. 7, 2019
![Page 2: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/2.jpg)
2
Overview
1 BackgroundAP-CATiDAP
2 Lessons learnedTeacher Access & Teacher EngagementPlan for HeterogeneityToolchainQuality ControlProcess Data
3 FutureLearning Modules
![Page 3: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/3.jpg)
Background 3
Background
![Page 4: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/4.jpg)
Background 4
AP-CAT
Full Name: “Cognitive Diagnostic Computerized AdaptiveTesting (CD-CAT) for AP Statistics”
A five-year CAREER project funded by the NationalScience Foundation
Two major componentsResearchEducational outreach
![Page 5: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/5.jpg)
Background 5
2018
Development
Testing Observation
RCTRefinement
Item BankWeb Platform
BugsUX/UIItem Keys
Feedback TypesLearning Outcomes
Engagement SurveyItem CalibrationAdaptive Testing
MobileDiagnostic Score ReportPilot Data
20152015 2016 2017 2018 2019
![Page 6: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/6.jpg)
Background 6
Item Bank
850 items
4 sections16 main topics158 attributes
![Page 7: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/7.jpg)
Background 7
![Page 8: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/8.jpg)
Background 8
![Page 9: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/9.jpg)
Background 9
![Page 10: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/10.jpg)
Background 10
Scaffolding
Step-by-step Solutions
Visible after assignment completion
![Page 11: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/11.jpg)
Background 11
![Page 12: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/12.jpg)
Background 12
![Page 13: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/13.jpg)
Background 13
Feedback
Granular feedback
Areas of opportunity
![Page 14: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/14.jpg)
Background 14
![Page 15: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/15.jpg)
Background 15
iDAP
Intelligent Diagnostic Assessment Platform(i-DAP) for High School Statistics Education
GOAL: Develop a holistic, personalized,learning system integrated into theclassroom
![Page 16: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/16.jpg)
Background 16
iDAP
Leverage attribute system developed in theAP-CAT
Map to 36 common core state standardson “statistics and probability”
![Page 17: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/17.jpg)
Background 17
![Page 18: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/18.jpg)
Lessons learned 18
Lessons learned
![Page 19: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/19.jpg)
Lessons learned 19
Lesson 1
Teacher Access & Teacher Engagement
![Page 20: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/20.jpg)
Lessons learned 20
Teacher Access & Teacher Engagement
Teacher approval
delivery enginereportingfeature requestsrollout of assignments
Example: In-Class AssignmentLength
![Page 21: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/21.jpg)
Lessons learned 21
Lesson 2
Plan for Heterogeneity
in learning environmentin students
![Page 22: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/22.jpg)
Lessons learned 22
Sample Heterogeneity
Not surprising
learning outcomes depend on schoolSurprising
Student self-predictions of their finalexam scores depend on school (Ober etal., In Prep)Patterns of incorrect responses
![Page 23: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/23.jpg)
Lessons learned 23
Lesson 3
Choose your toolchain wisely
![Page 24: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/24.jpg)
Lessons learned 24
![Page 25: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/25.jpg)
Lessons learned 25
1 con <� DBI : : dbConnect(RMySQL: :MySQL( ) , ## Connect to DB2 host = "domain.name.edu" ,3 port = 3306,4 user = "db_user" ,5 password = "db_pw" ,6 dbname = "db_name" )7
8 tb l (con , " act iv i ty " ) %>%9 f i l t e r ( role == "student" ) %>% ## Only student data
10 select(�firstname , �lastname , ## de�identi fy11 �role , �username,12 �id , �meta)13
![Page 26: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/26.jpg)
Lessons learned 26
Lesson 4
Have Quality Control and Contingency Plans
![Page 27: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/27.jpg)
Lessons learned 27
Lesson 4
Not all data are worth being collected
Not all data that have been collected areworth being analyzed
![Page 28: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/28.jpg)
Lessons learned 28
# Source: lazy query [?? x 4]# Database: mysql 5.7.27-0ubuntu0.18.04.1# [[email protected]:/dname]
url tabname t info<chr> <chr> <chr> <chr>
1 /results/461 results(student) 2019-09-25 11:30:28 <NA>2 /results/461 results(student) 2019-09-25 11:30:30 <NA>3 /results/461 student_info_btn 2019-09-25 11:30:34 "btn":"info","qid":"773","is_shown":true4 /results/461 student_info_btn 2019-09-25 11:30:36 "btn":"info","qid":"436","is_shown":false5 /results/461 student_info_btn 2019-09-25 11:31:30 "btn":"info","qid":"223","is_shown":true6 /results/461 student_info_btn 2019-09-25 11:31:38 "btn":"info","qid":"773","is_shown":false7 /results/461 Attributes(student) 2019-09-25 11:33:20 <NA>8 /results/461 results(student) 2019-09-25 11:33:29 <NA>9 /results/787 results(student) 2019-09-25 11:43:31 <NA>10 /results/787 results(student) 2019-09-25 11:46:55 <NA># ... with many, many, many more rows
![Page 29: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/29.jpg)
Lessons learned 29
Lesson 4
Quality Control
Dummy itemsResponse time data (Qualtrics hack)Open text responses to surveys
Contingency Plan
End users are used to having "delete"mean "put in trash"Choosing not to take the exam
![Page 30: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/30.jpg)
Lessons learned 30
Lesson 5
System paradata (Process data) can provide awealth of information
![Page 31: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/31.jpg)
Lessons learned 31
Example: Procrastination
Active Procrastination (Chu & Choi, 2005)Criticisms (e.g. Delay?) (Krause & Freund, 2014)
Attempting to validate a survey scale via process data
![Page 32: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/32.jpg)
Lessons learned 32
Data Source
Datalag = Assignment Deadline - Assignment Submission Timeacross 5 AssignmentsActive Procrastination Scale
Modeling lag as a behavioral indicator of procrastination
![Page 33: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/33.jpg)
Lessons learned 33
0
5
10
15
20
25
0 100 200
Mean Lag
freq
uenc
y
Histogram of Time to Due Date
![Page 34: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/34.jpg)
Lessons learned 34
●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
0
25
50
75
100
0 100 200
Mean Lag
SD L
ag
Mean vs Standard Deviation
![Page 35: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/35.jpg)
Lessons learned 35
Behavioral LV
Outcome Statisfaction
Preference for Pressure
Intentional Decision
Ability to Meet
Deadlines
assgn1 assgn2
os1
os2
os3
os4
pp1 pp2 pp3 pp4 id1 id2 id3 id4
md1
md2
md3
md4
assgn3 assgn4 assgn5
0.070.23*
-0.27*-0.09
0.23*
0.76*
0.48*0.14
-0.06 -0.47*
0.83
![Page 36: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/36.jpg)
Lessons learned 36
Model Fit
Model Fit
Model cfi tli rmsea(5%)
rmsea(95%)
srmr
ActiveProcrastination 0.964 0.957 0.074 0.082 0.071
![Page 37: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/37.jpg)
Lessons learned 37
Summary
Summary
Limitations
Future StudiesEstimate Latent Procrastination to provide remindersOther applications
![Page 38: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/38.jpg)
Future 38
Future
![Page 39: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/39.jpg)
Future 39
![Page 40: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/40.jpg)
Future 40
A holistic student view
Multiple Data SourcesAssessment DataBig 5Statistics AnxietyFree ResponseHelp Forum
![Page 41: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/41.jpg)
Thank You 41
Thank You
![Page 42: Using Assessment System Data to Generate Individualized ......lag = Assignment Deadline - Assignment Submission Time across 5 Assignments Active Procrastination Scale Modeling lag](https://reader034.fdocuments.net/reader034/viewer/2022042912/5f47186cb6492e7e226bbbff/html5/thumbnails/42.jpg)
Thank You 42
Acknowledgments
https://lambslab.nd.edu/