Confidence in Learning Analytics
60
Open Universiteit www.ou.nl Confidence in Learning Analytics Citation for published version (APA): Drachsler, H., & Greller, W. (2012). Confidence in Learning Analytics. Document status and date: Published: 20/12/2012 Document Version: Peer reviewed version Document license: CC BY-NC-ND Please check the document version of this publication: • A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website. • The final author version and the galley proof are versions of the publication after peer review. • The final published version features the final layout of the paper including the volume, issue and page numbers. Link to publication General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal. If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please follow below link for the End User Agreement: https://www.ou.nl/taverne-agreement Take down policy If you believe that this document breaches copyright please contact us at: [email protected] providing details and we will investigate your claim. Downloaded from https://research.ou.nl/ on date: 14 Aug. 2020
Transcript of Confidence in Learning Analytics
LAK12_presentationDrachsler, H., & Greller, W. (2012).
Confidence in Learning Analytics.
Document status and date: Published: 20/12/2012
Document Version: Peer reviewed version
Document license: CC BY-NC-ND
Please check the document version of this publication:
• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website. • The final author version and the galley proof are versions of the publication after peer review. • The final published version features the final layout of the paper including the volume, issue and page numbers.
Link to publication
General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.
• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal.
If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please follow below link for the End User Agreement:
https://www.ou.nl/taverne-agreement
Take down policy If you believe that this document breaches copyright please contact us at:
[email protected]
Hendrik Drachsler & Wolfgang Greller LAK12, Vancouver, Canada, 30th April 2012
Confidence in Learning Analytics Understandings and Expectations from the Stakeholders
Goals of the Presentation
The Learning Analytics Framework
Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society.
4
Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society.
4
Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society.
4
Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society.
4
Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society.
4
Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society.
4
Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society.
Stakeholders vs. LA Framework
Opinions from the stakeholders toward the dimensions of the LA framework
5
6
Extrapolate opinions from different target groups:
(a) what is the current understandings and the expectations on LA
(b) is there a common understanding of LA
7
Data and survey available at: http://bit.ly/la_survey
• 4 weeks available • 156 people after clean up • 121 people full records
8
Participants
9
26%
16%
10
Responses from 31 countries [UK (38), US (30), NL (22)] Participants - Reach
11
Stakeholders
12
Stakeholders (a) who was expected to benefit the most from learning analytics
Teachers Parents Institutions Learners
12
Stakeholders (a) who was expected to benefit the most from learning analytics
Teachers Parents Institutions Learners
13
Stakeholders
Stakeholders
Teachers Parents Institutions Learners
Outcomes: 1. Teacher-student 84% 2. Student-teacher 63% 3. Student-student 46% 4. Teacher-teacher 41%
14
Objectives
Reflection
(a) reflection (b) prediction (c) unveil hidden information
16
n = 119 11% no changes at all 43% small changes 45% extensive changes
In which way learning analytics will change educational practice in particular areas?
Objectives
16
n = 119 11% no changes at all 43% small changes 45% extensive changes
In which way learning analytics will change educational practice in particular areas?
Item 2: Timely information about learning Item 8: Better insights by institutions in their courses
Item 5: Easier grading
Item 6: Objective assessment
Drachsler, H., et al. (2010). Issues and Considerations regarding Sharable Data Sets for Recommender Systems in Technology Enhanced Learning. 1st Workshop Recommnder Systems in Technology Enhanced Learning (RecSysTEL@EC-TEL 2010) September, 28, 2010, Barcelona, Spain.
17
Drachsler, H., et al. (2010). Issues and Considerations regarding Sharable Data Sets for Recommender Systems in Technology Enhanced Learning. 1st Workshop Recommnder Systems in Technology Enhanced Learning (RecSysTEL@EC-TEL 2010) September, 28, 2010, Barcelona, Spain.
Verbert, K., Manouselis, N., Drachsler, H., and Duval, E. (accepted). Dataset-driven Research to Support Learning and Knowledge Analytics. Journal of Educational Technology & Society.
Researcher: 1. Added context information (n=43, means 3.42)
Teacher: 1. Added context information (n=52, means 3.42)
Manager: 1. Sharing within the institution (n=16, means 3.63) 2. Anonymisation (n=19, means 3.53)
18
19
Technologies
Manouselis, N., Drachsler, H., Verbert, K., and Duval, E. (to appear). Recommender Systems for Learning. Berlin:Springer
19
Technologies Prediction
Manouselis, N., Drachsler, H., Verbert, K., and Duval, E. (to appear). Recommender Systems for Learning. Berlin:Springer
19
Technologies
Peter Kraker, Claudia Wagner, Fleur Jeanquartier, Stefanie N. Lindstaedt (2011): On the Way to a Science Intelligence: Visualizing TEL Tweets for Trend Detection Sixth European Conference on Technology Enhanced Learning (EC-TEL 2011)
Reflection
Manouselis, N., Drachsler, H., Verbert, K., and Duval, E. (to appear). Recommender Systems for Learning. Berlin:Springer
20
Technologies
3. Assessment 4. View on engagement
5. Compare learners 6. Prediction of peers
7. Prediction of learner performance
Trust in accurate and appropriate LA results ...
21
Constraints
Constraints
Effect size of LA on ...
23
Competences
23
Competences
24
Competences
24
Competences
Item 1: Numerical skills
Item 5: Ethical thinking
Item 1: Numerical skills
Item 5: Ethical thinking
competent enough to independently learn from
learning analytics
Limitations
26
3. Low awareness of LA; Only surveyed innovators and early adopters
4. Mainly opinions from western cultures
1. Dominance of responses from the HE sector
2. Absence of students
Main findings of the survey according to the LA framework
27
28
Stakeholders
1.Main beneficiaries of LA are learners and teachers followed by organisations
2.Biggest benefits would be gained in the teacher-to-student relationship
3.Learners require teacher support to learn from LA
+
28
Stakeholders
1.Main beneficiaries of LA are learners and teachers followed by organisations
2.Biggest benefits would be gained in the teacher-to-student relationship
3.Learners require teacher support to learn from LA
LA as
1. Reflection support is main objective from the stakeholders view ...
2. ...by revealing hidden information about learners
Objectives+
29
1. Reflection support is main objective from the stakeholders view ...
2. ...by revealing hidden information about learners
Reflection support only for teacher student relationship?
Objectives+
30
3. Willingness to share if data is anonymised
Educational Data+
3. Willingness to share if data is anonymised
Can we achieve a collection of reference datasets?
Educational Data+
1. Trust in LA algorithms is not well developed
2. High confidence on gaining a comprehensive view of the learning progress
Technologies+
31
1. Trust in LA algorithms is not well developed
2. High confidence on gaining a comprehensive view of the learning progress
How accurate can we measure a learning progress?
Technologies+
32
1. Data ownership is the most important topic
2. LA lead to breaches of privacy but privacy and ethical aspects are of lesser importance
3. Many organisations have ethical boards and guidelines in place
Constraints+
32
1. Data ownership is the most important topic
2. LA lead to breaches of privacy but privacy and ethical aspects are of lesser importance
3. Many organisations have ethical boards and guidelines in place
Do we need new policies for data ownership and privacy?
Constraints+
33
1. Skepticism that LA will lead to more independence of learners to control their learning process
2. Training need to guide students to more self-directedness and critical reflection
Competences+
33
1. Skepticism that LA will lead to more independence of learners to control their learning process
2. Training need to guide students to more self-directedness and critical reflection
Do we need mandatory courses on statistics for the edu. sector?
Competences+
35
1. Encourage the LA community to gain additional insights from our dataset:
2. Look for partners to survey public bodies like school foundations and governmental institutions
3. Compare LA in different educational cultures and extend survey to eastern countries
http://bit.ly/la_survey
37
Email: [email protected]
Supporting projects:
Document status and date: Published: 20/12/2012
Document Version: Peer reviewed version
Document license: CC BY-NC-ND
Please check the document version of this publication:
• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website. • The final author version and the galley proof are versions of the publication after peer review. • The final published version features the final layout of the paper including the volume, issue and page numbers.
Link to publication
General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.
• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal.
If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please follow below link for the End User Agreement:
https://www.ou.nl/taverne-agreement
Take down policy If you believe that this document breaches copyright please contact us at:
[email protected]
Hendrik Drachsler & Wolfgang Greller LAK12, Vancouver, Canada, 30th April 2012
Confidence in Learning Analytics Understandings and Expectations from the Stakeholders
Goals of the Presentation
The Learning Analytics Framework
Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society.
4
Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society.
4
Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society.
4
Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society.
4
Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society.
4
Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society.
4
Greller, W., & Drachsler, H., (submitted). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society.
Stakeholders vs. LA Framework
Opinions from the stakeholders toward the dimensions of the LA framework
5
6
Extrapolate opinions from different target groups:
(a) what is the current understandings and the expectations on LA
(b) is there a common understanding of LA
7
Data and survey available at: http://bit.ly/la_survey
• 4 weeks available • 156 people after clean up • 121 people full records
8
Participants
9
26%
16%
10
Responses from 31 countries [UK (38), US (30), NL (22)] Participants - Reach
11
Stakeholders
12
Stakeholders (a) who was expected to benefit the most from learning analytics
Teachers Parents Institutions Learners
12
Stakeholders (a) who was expected to benefit the most from learning analytics
Teachers Parents Institutions Learners
13
Stakeholders
Stakeholders
Teachers Parents Institutions Learners
Outcomes: 1. Teacher-student 84% 2. Student-teacher 63% 3. Student-student 46% 4. Teacher-teacher 41%
14
Objectives
Reflection
(a) reflection (b) prediction (c) unveil hidden information
16
n = 119 11% no changes at all 43% small changes 45% extensive changes
In which way learning analytics will change educational practice in particular areas?
Objectives
16
n = 119 11% no changes at all 43% small changes 45% extensive changes
In which way learning analytics will change educational practice in particular areas?
Item 2: Timely information about learning Item 8: Better insights by institutions in their courses
Item 5: Easier grading
Item 6: Objective assessment
Drachsler, H., et al. (2010). Issues and Considerations regarding Sharable Data Sets for Recommender Systems in Technology Enhanced Learning. 1st Workshop Recommnder Systems in Technology Enhanced Learning (RecSysTEL@EC-TEL 2010) September, 28, 2010, Barcelona, Spain.
17
Drachsler, H., et al. (2010). Issues and Considerations regarding Sharable Data Sets for Recommender Systems in Technology Enhanced Learning. 1st Workshop Recommnder Systems in Technology Enhanced Learning (RecSysTEL@EC-TEL 2010) September, 28, 2010, Barcelona, Spain.
Verbert, K., Manouselis, N., Drachsler, H., and Duval, E. (accepted). Dataset-driven Research to Support Learning and Knowledge Analytics. Journal of Educational Technology & Society.
Researcher: 1. Added context information (n=43, means 3.42)
Teacher: 1. Added context information (n=52, means 3.42)
Manager: 1. Sharing within the institution (n=16, means 3.63) 2. Anonymisation (n=19, means 3.53)
18
19
Technologies
Manouselis, N., Drachsler, H., Verbert, K., and Duval, E. (to appear). Recommender Systems for Learning. Berlin:Springer
19
Technologies Prediction
Manouselis, N., Drachsler, H., Verbert, K., and Duval, E. (to appear). Recommender Systems for Learning. Berlin:Springer
19
Technologies
Peter Kraker, Claudia Wagner, Fleur Jeanquartier, Stefanie N. Lindstaedt (2011): On the Way to a Science Intelligence: Visualizing TEL Tweets for Trend Detection Sixth European Conference on Technology Enhanced Learning (EC-TEL 2011)
Reflection
Manouselis, N., Drachsler, H., Verbert, K., and Duval, E. (to appear). Recommender Systems for Learning. Berlin:Springer
20
Technologies
3. Assessment 4. View on engagement
5. Compare learners 6. Prediction of peers
7. Prediction of learner performance
Trust in accurate and appropriate LA results ...
21
Constraints
Constraints
Effect size of LA on ...
23
Competences
23
Competences
24
Competences
24
Competences
Item 1: Numerical skills
Item 5: Ethical thinking
Item 1: Numerical skills
Item 5: Ethical thinking
competent enough to independently learn from
learning analytics
Limitations
26
3. Low awareness of LA; Only surveyed innovators and early adopters
4. Mainly opinions from western cultures
1. Dominance of responses from the HE sector
2. Absence of students
Main findings of the survey according to the LA framework
27
28
Stakeholders
1.Main beneficiaries of LA are learners and teachers followed by organisations
2.Biggest benefits would be gained in the teacher-to-student relationship
3.Learners require teacher support to learn from LA
+
28
Stakeholders
1.Main beneficiaries of LA are learners and teachers followed by organisations
2.Biggest benefits would be gained in the teacher-to-student relationship
3.Learners require teacher support to learn from LA
LA as
1. Reflection support is main objective from the stakeholders view ...
2. ...by revealing hidden information about learners
Objectives+
29
1. Reflection support is main objective from the stakeholders view ...
2. ...by revealing hidden information about learners
Reflection support only for teacher student relationship?
Objectives+
30
3. Willingness to share if data is anonymised
Educational Data+
3. Willingness to share if data is anonymised
Can we achieve a collection of reference datasets?
Educational Data+
1. Trust in LA algorithms is not well developed
2. High confidence on gaining a comprehensive view of the learning progress
Technologies+
31
1. Trust in LA algorithms is not well developed
2. High confidence on gaining a comprehensive view of the learning progress
How accurate can we measure a learning progress?
Technologies+
32
1. Data ownership is the most important topic
2. LA lead to breaches of privacy but privacy and ethical aspects are of lesser importance
3. Many organisations have ethical boards and guidelines in place
Constraints+
32
1. Data ownership is the most important topic
2. LA lead to breaches of privacy but privacy and ethical aspects are of lesser importance
3. Many organisations have ethical boards and guidelines in place
Do we need new policies for data ownership and privacy?
Constraints+
33
1. Skepticism that LA will lead to more independence of learners to control their learning process
2. Training need to guide students to more self-directedness and critical reflection
Competences+
33
1. Skepticism that LA will lead to more independence of learners to control their learning process
2. Training need to guide students to more self-directedness and critical reflection
Do we need mandatory courses on statistics for the edu. sector?
Competences+
35
1. Encourage the LA community to gain additional insights from our dataset:
2. Look for partners to survey public bodies like school foundations and governmental institutions
3. Compare LA in different educational cultures and extend survey to eastern countries
http://bit.ly/la_survey
37
Email: [email protected]
Supporting projects:
