The Student Activity Meter for Awareness and Self-reflection
-
Upload
sten-govaerts -
Category
Education
-
view
955 -
download
1
description
Transcript of The Student Activity Meter for Awareness and Self-reflection
![Page 1: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/1.jpg)
THE STUDENT ACTIVITY METER FOR AWARENESS AND
SELF-REFLECTIONSten Govaerts, Katrien Verbert, Erik Duval, Abelardo Pardo.
1
SigCHI 2012, Long Case Study.
![Page 2: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/2.jpg)
our team:Human-Computer Interaction
•technology enhanced learning•music•research 2.0
2
![Page 5: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/5.jpg)
OBJECTIVES
• self-monitoring for learners
4
http://free-extras.com/images/obama_hope_progress-2848.htm
![Page 6: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/6.jpg)
OBJECTIVES
• self-monitoring for learners
• awareness for teachers
4
http://free-extras.com/images/obama_hope_progress-2848.htm
![Page 7: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/7.jpg)
OBJECTIVES
• self-monitoring for learners
• awareness for teachers
• time tracking
4
http://free-extras.com/images/obama_hope_progress-2848.htm
![Page 8: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/8.jpg)
OBJECTIVES
• self-monitoring for learners
• awareness for teachers
• time tracking
• learning resource use and recommendations
4
http://free-extras.com/images/obama_hope_progress-2848.htm
![Page 9: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/9.jpg)
OBJECTIVES
• self-monitoring for learners
• awareness for teachers
• time tracking
• learning resource use and recommendations
• part of Learning Analytics research [ACM LAK conf., Siemens 2011, Duval 2011]
4
http://free-extras.com/images/obama_hope_progress-2848.htm
![Page 10: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/10.jpg)
STUDENT ACTIVITY METER (SAM): DEMO.
5
http://dilbert.com/strips/comic/2000-02-24/
![Page 11: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/11.jpg)
http://research.uow.edu.au/learningnetworks/seeing/snapp/index.html SNAPP
![Page 12: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/12.jpg)
GISMO http://www.noe-kaleidoscope.org/pub/researcher/resources/video/aied/mazza.html
![Page 13: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/13.jpg)
http://www.cs.uu.nl/docs/vakken/uem/Janssen.pdfThe Participation Tool
![Page 14: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/14.jpg)
METHODOLOGY.
• design-based research [Obrenović 2011]: design a tool, give students & teachers and evaluate
• iterative process of design & evaluation
• paper mockups
• usability studies
• surveys on usefulness
9http://funnytogo.com/pictures/evolution/timeline.htm
![Page 15: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/15.jpg)
10
iter. evaluation goal methodology demo-graphics data
I.usability, satisfaction,
preliminary usefulness
task-based interview with think-aloud, SUS
& MSDT
12 CS students at KULeuven
timetracking via Twitter
II.assessing teacher
needs, use & usefulness
online survey with Likert items, yes/no and open questions
19 teachers & TA’s with
CGIAR coursesMoodle logs
III.
assessing teacher needs, expert
feedback, use & usefulness
online survey with Likert items, yes/no and open questions
12 participants from LAK MOOC
Moodle logs
IV. use, usefulness & satisfaction
structured face-to-face interview with tasks &
open questions
11 teachers & TA’s at Carlos III
Uni. Madrid
VM with extensive
tool tracking
![Page 16: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/16.jpg)
11
![Page 17: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/17.jpg)
11
![Page 18: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/18.jpg)
demographics
evaluation goal
design changes negative positive
I. 12 CS students
usability, satisfaction, preliminary usefulness
1st iteration small usability issues
•↑learnability•↓errors•good satisfaction•usefulness positive
![Page 19: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/19.jpg)
ITERATION TWO & THREE
• CGIAR: 19 teachers - LAK: 12
• An online survey about usefulness, teacher issues and how the tool can resolve these.
13
![Page 20: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/20.jpg)
ITERATION TWO & THREE
• CGIAR: 19 teachers - LAK: 12
• An online survey about usefulness, teacher issues and how the tool can resolve these.
13
![Page 21: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/21.jpg)
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher
✔
✗
?✔
✔
✔?
✔
??
✔
✔✔??✔?
✔
14
add-ressed
✔✔
✔
✔?
?
✔
✗
?
issue for teacher
add-ressed
✗
?
✔?
✔?
?
✗
?
CGIAR LAK1 2 3 4 5
Issues
Provide feedback to the studentsBeing aware of what students are doingKnowing about collaboration and communication *Knowing which documents are used and how muchKnowing how and when online tools have been usedFinding the students who are not doing well
Finding the best students
Knowing how much time students spentKnowing if external learning resources are used
Addressed by SAM
1 2 3 4 5
![Page 22: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/22.jpg)
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher
✔
✗
?✔
✔
✔?
✔
??
✔
✔✔??✔?
✔
14
add-ressed
✔✔
✔
✔?
?
✔
✗
?
issue for teacher
add-ressed
✗
?
✔?
✔?
?
✗
?
CGIAR LAK
![Page 23: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/23.jpg)
Provide feedback to the students
Being aware of what students are doing
Knowing about collaboration and communication
Knowing which documents are used and how much
Knowing how and when online tools have been used
Finding the students who are not doing well
Finding the best students
Knowing how much time students spent
Knowing if external learning resources are used
issue for teacher
✔
✗
?✔
✔
✔?
✔
??
✔
✔✔??✔?
✔
14
add-ressed
✔✔
✔
✔?
?
✔
✗
?
issue for teacher
add-ressed
✗
?
✔?
✔?
?
✗
?
CGIAR LAK
![Page 24: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/24.jpg)
CGIAR CASE STUDY
15
![Page 25: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/25.jpg)
CGIAR CASE STUDY
good indicator for effortunderstand the workload
use for course design optimization
obtain course overview
increase awareness
detect outliers
progress evolutioncompare students
want better 1 to 1 comparison tool
more details on student
want to search for students
more metricswanted
15
![Page 26: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/26.jpg)
CGIAR CASE STUDY
good indicator for effortunderstand the workload
use for course design optimization
obtain course overview
increase awareness
detect outliers
progress evolutioncompare students
want better 1 to 1 comparison tool
more details on student
want to search for students
more metricswanted
15
![Page 27: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/27.jpg)
LAK CASE STUDY
16http://maranash.blogspot.com/2011/01/conference-shout-out.html
![Page 28: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/28.jpg)
LAK CASE STUDY
chronological course dwell time
verify the status of the classroom activity
self-reflection to measure progress and increase
motivation
find students experiencing problems and low
engagement
more metrics
more data
16http://maranash.blogspot.com/2011/01/conference-shout-out.html
![Page 29: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/29.jpg)
LAK CASE STUDY
chronological course dwell time
verify the status of the classroom activity
self-reflection to measure progress and increase
motivation
find students experiencing problems and low
engagement
more metrics
more data
16http://maranash.blogspot.com/2011/01/conference-shout-out.html
![Page 30: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/30.jpg)
demographics
evaluation goal
design changes
negative positive
I.12 CS
students
usability, satisfaction, preliminary usefulness
1st iterationsmall usability
issues
•↑learnability•↓errors•good satisfaction•usefulness positive
II.19
teachers & TA’s
assessing teacher
needs, use & usefulness
help functionresource
recomm. not useful
•provides awareness•all vis. useful•many uses•90% want it
III.12
participants
assessing teacher
needs, expert feedback, use & usefulness
re-orderable parallel
coordinates with
histograms
most addressed needs are indecisive
•provides awareness and feedback•many uses•66% want it•recomm. can be useful
![Page 31: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/31.jpg)
ITERATION FOUR
18
• a CS course on C++ programming
• 11 people: 7 teachers, 2 TA’s & 1 course planner
• richer data set: tracking from programming environment
• qualitative study using a structured face-2-face interview with 25 open questions.
![Page 32: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/32.jpg)
ITERATION FOUR
18
• a CS course on C++ programming
• 11 people: 7 teachers, 2 TA’s & 1 course planner
• richer data set: tracking from programming environment
• qualitative study using a structured face-2-face interview with 25 open questions.
![Page 33: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/33.jpg)
ACHIEVABLE TASKS
19
identify students doing well and at risk
understand time spent better
better course overview
detect students for assistance
course planning
monitor tool and resource use
get a time summary
conflicting visions on students doing well
and at risk
http://www.tdpri.com/forum/telecaster-discussion-forum/48431-ic503z-cap-52-ri-blackguard-book.html
![Page 34: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/34.jpg)
ACHIEVABLE TASKS
19
identify students doing well and at risk
understand time spent better
better course overview
detect students for assistance
course planning
monitor tool and resource use
get a time summary
conflicting visions on students doing well
and at risk
http://www.tdpri.com/forum/telecaster-discussion-forum/48431-ic503z-cap-52-ri-blackguard-book.html
![Page 35: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/35.jpg)
USABILITY & SATISFACTION
20
• average SUS score: 69,69%
[Bangor et al. 2008]
![Page 36: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/36.jpg)
USABILITY & SATISFACTION
20
• average SUS score: 69,69%
all: want to continue using it9/11: give it to students
[Bangor et al. 2008]
![Page 37: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/37.jpg)
demo-graphics
evaluation goal
design changes negative positive
I.12 CS
students
usability, satisfaction, preliminary usefulness
1st iterationsmall usability
issues
•↑learnability•↓errors•good satisfaction•usefulness positive
II.19
teachers & TA’s
assessing teacher needs, use & usefulness
help functionresource
recomm. not useful
•provides awareness•all vis. useful•many uses•90% want it
III.12
participants
assessing teacher needs, expert
feedback, use & usefulness
re-orderable PC with
histograms
most addressed needs are indecisive
•provides awareness and feedback•many uses•66% want it•recomm. can be useful
IV.11
teachers & TA’s
use, usefulness & satisfaction
filter & search, icons,
zooming in line chart, editing PC
axes
conflicting visions of students
doing well or at risk
•provides time overview•provides course overview•PC assist with detecting problems•many uses & insights•100% want it
![Page 38: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/38.jpg)
CONCLUSION
• SAM is perceived useful and contributes to better awareness.
22
![Page 39: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/39.jpg)
CONCLUSION
• SAM is perceived useful and contributes to better awareness.
• SAM enables:
22
![Page 40: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/40.jpg)
CONCLUSION
• SAM is perceived useful and contributes to better awareness.
• SAM enables:
• to detect potential students doing well and at risk
22
![Page 41: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/41.jpg)
CONCLUSION
• SAM is perceived useful and contributes to better awareness.
• SAM enables:
• to detect potential students doing well and at risk• a better course overview
22
![Page 42: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/42.jpg)
CONCLUSION
• SAM is perceived useful and contributes to better awareness.
• SAM enables:
• to detect potential students doing well and at risk• a better course overview• understanding student time spending
22
![Page 43: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/43.jpg)
CONCLUSION
• SAM is perceived useful and contributes to better awareness.
• SAM enables:
• to detect potential students doing well and at risk• a better course overview• understanding student time spending• to find a wide variety of new insights
22
![Page 44: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/44.jpg)
CONCLUSION
• SAM is perceived useful and contributes to better awareness.
• SAM enables:
• to detect potential students doing well and at risk• a better course overview• understanding student time spending• to find a wide variety of new insights
• almost all want to continue using SAM22
![Page 45: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/45.jpg)
LESSONS LEARNED
• worked well:
23http://slurmed.com/fanart/dexus5/005_professor.png
![Page 46: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/46.jpg)
LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early
23http://slurmed.com/fanart/dexus5/005_professor.png
![Page 47: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/47.jpg)
LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early• Use real-world settings
23http://slurmed.com/fanart/dexus5/005_professor.png
![Page 48: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/48.jpg)
LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early• Use real-world settings• Visualisations are powerful tool for creating
awareness
23http://slurmed.com/fanart/dexus5/005_professor.png
![Page 49: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/49.jpg)
LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early• Use real-world settings• Visualisations are powerful tool for creating
awareness
• worked less well:
23http://slurmed.com/fanart/dexus5/005_professor.png
![Page 50: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/50.jpg)
LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early• Use real-world settings• Visualisations are powerful tool for creating
awareness
• worked less well:
• finding real-world settings
23http://slurmed.com/fanart/dexus5/005_professor.png
![Page 51: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/51.jpg)
LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early• Use real-world settings• Visualisations are powerful tool for creating
awareness
• worked less well:
• finding real-world settings• getting tracking data
23http://slurmed.com/fanart/dexus5/005_professor.png
![Page 52: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/52.jpg)
LESSONS LEARNED
• worked well:
• Evaluate visualisations iteratively and early• Use real-world settings• Visualisations are powerful tool for creating
awareness
• worked less well:
• finding real-world settings• getting tracking data
• longitudinal study would be useful23
http://slurmed.com/fanart/dexus5/005_professor.png
![Page 53: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/53.jpg)
24
http://www.fitbit.com/
http://media.cnbc.com/i/CNBC/Sections/News_And_Analysis/_News/_SPECIAL_REPORTS/
BetterYourBusiness_2012/BYB-Digital%20-%20April/Slideshows/Digital_Appliances_ab/10-
Smart-Dig-Appliances-Wifi-scale-.jpg
http://quantifiedself.com/
http://nikerunning.nike.com/nikeos/p/nikeplus/en_EMEA/sportband
![Page 54: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/54.jpg)
LAK13: Learning Analytics and Knowledge Conf.8-12 April 2013 Leuven, Belgium
http://lakconference.org/
![Page 55: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/55.jpg)
2
slides will appear on http://www.slideshare.net/stengovaerts
slides will appear on http://www.slideshare.net/stengovaerts 26
contact: [email protected]@erikduval
![Page 56: The Student Activity Meter for Awareness and Self-reflection](https://reader033.fdocuments.net/reader033/viewer/2022060107/5549b682b4c90564768b4ba0/html5/thumbnails/56.jpg)
2
slides will appear on http://www.slideshare.net/stengovaerts
slides will appear on http://www.slideshare.net/stengovaerts 26
If you are interested in visualisations and tabletops, we have a PhD position available.
contact: [email protected]@erikduval