A Study of Affective and Metacognitive Factors for Learning Statistics ...
Case Study: A Longitudinal Mixed-Method Study of Learning Gain - Applying...
-
Upload
bart-rienties -
Category
Education
-
view
354 -
download
0
Transcript of Case Study: A Longitudinal Mixed-Method Study of Learning Gain - Applying...
Case Study: A Longitudinal Mixed-Method Study of Learning Gain - Applying
Affective-Behaviour-Cognition Framework at Three Institutions
11-05-2016@drBartRienties & Dr Jekaterina Rogaten@learninggainshttps://twitter.com/LearningGains
ABC learning gains team
Dr Bart Rienties Dr Jekaterina Rogaten Prof Denise Whitelock
Dr Simon Cross Prof Allison Littlejohn
Prof Rhona Sharpe (OB) Dr Ian Scott (OB)
Prof Ian Kinchin (US) Prof Steven Warburton (US) Dr Simon Lygo-Baker (US)
Quantitative
Qualitative
Academic performance
VLE
Satisfaction
Interviews
Diary
Module design
Socio-Demographic
Structure of the ABC project
What does the literature say about Learning gain?
• The concept of learning gain is primarily used to examine the effect of any particular educational ‘intervention’
– Web of Science core collection and ERIC– The time frame of the search 2000 -
2016. – In total 231 studies were identified of
which 73 studies were selected. – All learning gains can be classified into
ABC– Meta-analytic literature review to be
submitted to the Educational Research Review in June 2016.
Example Arts AA100• The arts past and present
– 30 week, 60 credit, Level 1 module – Introduces to university-level study in the arts across a range of subject areas - art history, classical studies, English, history, philosophy,
music and religious studies.• Assignment 01 (10%) • Assignment 02 (10%); • Assignment 03 (15%); • Assignment 04 (20%); • Assignment 05 (15%); • Assignment 06 (20%);• Assignment 07 (10%)
– 25% - BA (Honours) History, 19% - BA (Honours) Humanities, 16.2% - BA (Honours) English Language and Literature, 12.5% - BA (Honours) English Literature
– Design: 75% Assimilative, 20% Assessment, 4% Productive
Participants
3262 students from years 2013/14 and 2014/15
2147 Females (age M=33.2, SD=12.8) & 1115 Males (age M=35.5, SD=14.2)
Motivational orientation:
36.8% - employment and personal development, 13.6% - employment, 23.9% - personal development
Occupational status:
43.9% in full-time employment, 24.2% part-time employment, 13.4% not in paid work, 10.9% unemployed
93.2% White; 19% reported study disability
50% have A level or equivalent, 30.9% have below A level, 14.3% have HE qualification
81.4% were only taking AA100 module
Multilevel Modelling
Level 1
Level 2
Level 4
Level 3
ABC1
Student1
ABC3 ABC1 ABC2 ABC3 ABC1 ABC2 ABC3ABC2
Student2 Student3
Course1 Course2
University1 University2
ABC1 ABC2 ABC3
Student4
ABC1 ABC2 ABC3
Student5
Course3
Learning Design
Assimilative Finding and handling information
Communication
Productive Experiential Interactive/
Adaptive
Assessment
Type of activity
Attending to information
Searching for and processing information
Discussing module related content with at least one other person (student or tutor)
Actively constructing an artefact
Applying learning in a real-world setting
Applying learning in a simulated setting
All forms of assessment, whether continuous, end of module, or formative (assessment for learning)
Examples of activity
Read, Watch, Listen, Think about, Access, Observe, Review, Study
List, Analyse, Collate, Plot, Find, Discover, Access, Use, Gather, Order, Classify, Select, Assess, Manipulate
Communicate, Debate, Discuss, Argue, Share, Report, Collaborate, Present, Describe, Question
Create, Build, Make, Design, Construct, Contribute, Complete, Produce, Write, Draw, Refine, Compose, Synthesise, Remix
Practice, Apply, Mimic, Experience, Explore, Investigate, Perform, Engage
Explore, Experiment, Trial, Improve, Model, Simulate
Write, Present, Report, Demonstrate, Critique
Method – data sets• Combination of four different data sets:
• learning design data (189 modules mapped, 276 module implementations included)
• student feedback data (140)• VLE data (141 modules)• Academic Performance (151)
• Data sets merged and cleaned• 111,256 students undertook these modules
Constructivist Learning Design
Assessment Learning Design
Productive Learning Design
Socio-construct. Learning Design
VLE Engagement
Student Satisfaction
Student retention
Learning Design151 modules
Week 1 Week 2 Week30+
Disciplines LevelsSize module
Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341
Week Assim Find Com. Prod Exp Inter Asses Total
-2 -.03 .02 -.02 -.09 .20* -.03 .01 .35** -1 -.17* .14 .14 -.01 .30** -.02 -.05 .38**
0 -.21* .14 .37** -.07 .13 .08 .02 .48**
1 -.26** .25** .47** -.02 .28** .01 -.1 .48**
2 -.33** .41** .59** -.02 .25** .05 -.13 .42** 3 -.30** .33** .53** -.02 .34** .02 -.15 .51**
4 -.27** .41** .49** -.01 .23** -.02 -.15 .35**
5 -.31** .46** .52** .05 .16 -.05 -.13 .28**
6 -.27** .44** .47** -.04 .18* -.09 -.08 .28**
7 -.30** .41** .49** -.02 .22** -.05 -.08 .33**
8 -.25** .33** .42** -.06 .29** -.02 -.1 .32**
9 -.28** .34** .44** -.01 .32** .01 -.14 .36**
10 -.34** .35** .53** .06 .27** .00 -.13 .35**
Model 1 Model 2 Model 3
Level0 -.279** -.291** -.116
Level1 -.341* -.352* -.067
Level2 .221* .229* .275**
Level3 .128 .130 .139
Year of implementation .048 .049 .090
Faculty 1 -.205* -.211* -.196*
Faculty 2 -.022 -.020 -.228**
Faculty 3 -.206* -.210* -.308**
Faculty other .216 .214 .024
Size of module .210* .209* .242**
Learner satisfaction (SEAM) -.040 .103
Finding information .147
Communication .393**
Productive .135
Experiential .353**
Interactive -.081
Assessment .076
R-sq adj 18% 18% 40%
n = 140, * p < .05, ** p < .01 Table 3 Regression model of LMS engagement predicted by institutional, satisfaction and learning design analytics
• Level of study predict VLE engagement
• Faculties have different VLE engagement
• Learning design (communication & experiential) predict VLE engagement (with 22% unique variance explained)
Model 1 Model 2 Model 3
Level0 .284** .304** .351** Level1 .259 .243 .265 Level2 -.211 -.197 -.212 Level3 -.035 -.029 -.018 Year of implementation .028 -.071 -.059 Faculty 1 .149 .188 .213* Faculty 2 -.039 .029 .045 Faculty 3 .090 .188 .236* Faculty other .046 .077 .051
Size of module .016 -.049 -.071 Finding information -.270** -.294**
Communication .005 .050 Productive -.243** -.274** Experiential -.111 -.105 Interactive .173* .221* Assessment -.208* -.221*
LMS engagement .117
R-sq adj 20% 30% 31% n = 150 (Model 1-2), 140 (Model 3), * p < .05, ** p < .01 Table 4 Regression model of learner satisfaction predicted by institutional and learning design analytics
• Level of study predict satisfaction
• Learning design (finding info, productive, assessment) negatively predict satisfaction
• Interactive learning design positively predicts satisfaction
• VLE engagement and satisfaction unrelated
Model 1 Model 2 Model 3
Level0 -.142 -.147 .005
Level1 -.227 -.236 .017
Level2 -.134 -.170 -.004
Level3 .059 -.059 .215
Year of implementation -.191** -.152* -.151*
Faculty 1 .355** .374** .360**
Faculty 2 -.033 -.032 -.189*
Faculty 3 .095 .113 .069
Faculty other .129 .156 .034
Size of module -.298** -.285** -.239**
Learner satisfaction (SEAM) -.082 -.058
LMS Engagement -.070 -.190*
Finding information -.154
Communication .500**
Productive .133
Experiential .008
Interactive -.049
Assessment .063
R-sq adj 30% 30% 36%
n = 150 (Model 1-2), 140 (Model 3), * p < .05, ** p < .01
Table 5 Regression model of learning performance predicted by institutional, satisfaction and learning design analytics
• Size of module and discipline predict completion
• Satisfaction unrelated to completion
• Learning design (communication) predicts completion
Constructivist Learning Design
Assessment Learning Design
Productive Learning Design
Socio-construct. Learning Design
VLE Engagement
Student Satisfaction
Student retention
150+ modules
Week 1 Week 2 Week30+
Rienties, B., Toetenel, L., (2016). The impact of learning design on student behaviour, satisfaction and performance: a cross-institutional comparison across 151 modules. Computers in Human Behavior, 60 (2016), 333-341
Communication
Next steps• Ethics• Data collection
Planned:• TEA conference (October)
• Modelling cognitive learning gains using learning analytics
• SRHE conference (December)• Presentation of the 4-level model for
OU+OB+US students samples
• OEB conference (December) • Modelling learning gains using learning analytics
Confirmed:• Teaching Excellence Framework: Raising Quality Standards (14th July)
• Presentation on Learning Analytics for measuring learning gains• EARLI conference (17-19th August)
• Presentation of the 3-level model for OU students sample• HEIR 2016 Conference (7-8 September)
• Presentation on ABC multi-level model OU student sample
• Data storage• Data analysis
Dissemination
Case Study: A Longitudinal Mixed-Method Study of Learning Gain - Applying
Affective-Behaviour-Cognition Framework at Three Institutions
11-05-2016@drBartRienties & Dr Jekaterina Rogaten@learninggainshttps://twitter.com/LearningGains