Lies, damned lies and statistics: an evaluation of learning styles in AEH

Elizabeth FitzGerald (née Brown)e.j.fitzgerald@open.ac.ukInstitute of Educational Technology

Introduction

• Adaptive educational hypermedia (AEH)• Personalisation mechanisms• Learning styles for user profiling• Experimental methodologies• Case studies: WHURLE and DEUS• Conclusions and discussion

Adaptive hypermedia/AEH

• Main ways of adapting hypermedia:– at the link level; – at content level; – via structural adaptation – or presentational adaptation

• User modelling is common in educational applications and tends to use either prior knowledge or preferred learning style

Why use learning styles?

• Apparently:–Students learn better when using preferences in

which they're successful–Students will be better learners when they can

expand these preferences–When the learning experience accommodates

various preferences, more students will be successful

Examples of learning styles

Reflector

Theorist

Activist

Pragmatist

Learning style models• Dunn + Dunn• MBTI (Myers-Briggs Type Inventory)• Kolb’s experiential model / Honey + Mumford• Riding's CSA (Cognitive Styles Analysis)• Herrmann's brain dominance theory• Multiple intelligences (7 or 9?)• Biggs SPQ (Surface Processing Questionnaire)• Field dependence vs field independence• VAKT (Visual/Auditory/Kinaesthetic/Tactile)• Wholist/holist vs analytic

• Which is best and how should it be used?

Experimental methodologies

• How can we evaluate the learning experience?• Which methods are most suitable?

– Quantitative– Qualitative

• Random controlled trials• Statistical tools

… but we are dealing with people, who are inherently complicated!

Case studies

• Two systems investigated:– WHURLE: a revision guide, used with undergraduates– DEUS: a web-based e-learning system, used with primary school

children

• Two different learning styles looked at:– WHURLE: visual/verbal– DEUS: global/sequential

• I wanted to find out if these user models were beneficial, from a quantitative perspective

WHURLE experimental study

• User trials carried out with an online revision guide for a taught module

• Over 200 university students involved• Used a visual-verbal approach, investigating 2 variables:

– Visual and verbal environments– Visual-verbal learning style of students

• Feedback/evaluation via assessment data, questionnaires, interviews and log files

WHURLE revision guide: system architecture

Lesson Plan

Adaptation Filter

Display Engine

Virtual Document

Chunks

Links User Model

Skin

+ +

The TitleSome text some text some text some more text some more text. Text text text Some text some text some text some more text some more text. Text text text.

Some text some text some text some more text some more text. Text text text Some text some text some text some more text some more text. Text text text.

Learning styles in WHURLE

• Lesson plan produced for visual, verbal and no preference users

• Chunks created: mix of visual, verbal, no preference or universal

• Students filled in a learning styles questionnaire during first log-in

• Users then randomly assigned to matched group, mismatched group or neutral group

Student information

• Mostly 2nd/3rd year undergraduates• Average age was 21, gender ratio of 3.6 males:1 female• Out of 221 students who logged on at least once:

– 105 were visual– 105 were bimodal (no preference)– 11 were verbal

Screenshots

Verbal environmentVisual environment No preference environment

What were we investigating?

• To see if matching or mismatching would make a difference

• To see if there were any differences between students with different learning styles

• To see if there were any differences between students who used the different environments

Main findings of the study

• Matching or mismatching made no difference to student performance

• No difference between students with different learning styles

• No difference between students who used the different environments

Statistical resultsHypothesis: Statistical significance:

H1: matched students will do significantly better

F(4,210)=0.66, p=0.62, Wilks’ Lambda=0.98, partial eta squared=0.1

H2: mismatched students will do significantly worse

F(4,210)=0.66, p=0.62, Wilks’ Lambda=0.98, partial eta squared=0.1

H3: one type of learning style is more beneficial

F(2,106)=0.46, p=0.63, Wilks’ Lambda=0.99, partial eta squared=0.01

H4: one type of learning environment is more beneficial

F(4,210)=0.59, p=0.67, Wilks’ Lambda=0.98, partial eta squared=0.01

Secondary findings• No correlation between amount of use of the system and

student performance• Qualitative data suggests students found it an enjoyable and

useful resource• All students interviewed agreed that personalisation was

important

• Phase II, n=144. Testing mode choice (visual/verbal) and switching behaviours when users allowed free reign via analysis of web log data.

User choice and switching seemingly not driven by learning style but by other factors (time available; preparation for verbal task; boredom etc).

Conclusions from WHURLE trial

• Personalising for visual-verbal learning style does not seem to have much educational benefit

• However, many students studying for Computer Science degrees seem to be visual learners

• Students feel that personalisation in web-based learning is important

• BUT… there are problems with visual/verbal learning style, also with the type of users involved in the trial

Visual/verbal learning style

• Number of problems:– Majority of people are mostly visual*– Difficulty of determining granularity of a visual image– Visual representations susceptible to bias/subjectivity– Visual and verbal information equivalence– Dual coding theory

• Different learning style selected for next trial: global/sequential

* to a greater or lesser extent**

** depending on what’s being learned

The problem with participants

• Too many studies are tested on “pet students”• Users for such studies should be based in the

“real world”• Resultant data will be:

– More realistic – More widely applicable

• As a consequence, AEH will become more useful

AEH use in schools• School children can be termed “real” users• Only limited amount of research done with schools• Bajraktarevic, 2003: 2 studies involving GCSE pupils

– ILASH: summarising and questioning strategies supported by link and text adaptation in user interface

– Global and sequential learning styles as adaptation mechanism

• Neglect of subsequent school-based research

Overview of DEUS case study• Investigation into matching and mismatching pupils’ learning

styles with a web-based AEH system• Initial pilot study and software testing by target users

prior to study• Case study: Glapton Primary School, Nottingham:

– Mixed school, non-denominational– Around 300 pupils on roll, aged from 3-11 years old– Mostly white British background with English as first language– Number of pupils with learning difficulties/disabilities

close to national average– Attainment on entry to school is slightly below national expectations

Pilot study• Few learning style models in AEH used with children• Development of modified questionnaire to assess

visual/verbal and sequential/global learning styles• Collected data on 2 occasions – 10-day gap• Main findings:

– Visual/verbal style seemed fairly stable but skewed– Sequential/global style was more changeable but normally distributed

Distribution of visual/verbal learning style preferences in 10-11 year old children

0

2

4

6

8

10

12

14

11 9 7 5 3 1 -1 -3 -5 -7 -9 -11

Learning style visual verbal

Num

ber

of p

upil

s

Time 1

Time 2

Distribution of sequential/global learning style preferences in 10-11 year old children

0

2

4

6

8

10

12

14

16

18

11 9 7 5 3 1 -1 -3 -5 -7 -9 -11

Learning stylesequential global

Num

ber

of p

upil

s Time 1

Time 2

Experimental design: overview

Post-test (assess domain knowledge and learning

style)

Pupils assigned into experimental groups

2 hours of intervention (including structured learning

using workbook exercises)

Pre-test (assess domain knowledge and learning

style)

The creation of DEUS

• Previous work with WHURLE and visual/verbal learning style

• New system conceptually similar to WHURLE: user model, lesson plans and chunks

• Coded in PHP/XHTML and mySQL• Learning styles adaptation mechanism contained within

lesson plan and reflected in navigation• Range of multimedia used

DEUS login screen

Using DEUS• Personalised login• Topic choice in

negotiation with teacher/school

• Navigation tailored to global or sequential, matched or mismatched

• Workbooks used for tasks

Data analysis: overview• 82 participants, aged 9-11 years old, over 3 weeks• Assigned into 1 of 4 groups:

• Smallest group size = 15 pupils• Assessment via pre-test and post-test quizzes• Knowledge construction aided by specific tasks in

workbooks that matched the environment (sequential/global)

Sequential Global

Sequential ms mms

Global mmg mg

Style

Environment

m = matchedmm = mismatched

Main hypotheses

• H0 – there will be no statistically significant difference in knowledge gained between users from different experimental groups

• H1 – matched pupils will do significantly better than mismatched pupils

• H2 – mismatched pupils will do significantly worse than matched pupils

• H3 – one type of learning style is better than another• H4 – one type of learning environment is better than

another

Summary of main findings

• No difference between knowledge gains of matched/mismatched pupils

• No difference between pupils with different learning styles

• No difference between pupils using different environments

• Also no difference between SEN/non-SEN pupils• Hypotheses 1-4 rejected; revert to null hypothesis

Additional investigations: browsing time

• H0 – there will be no statistically significant difference in amount of browsing time between users from different experimental groups

• H1 – browsing times are different in matched versus mismatched users

• H2 – browsing times are different for different learning styles

• H3 – browsing times are different for different learning environments

• H4 – browsing times are different for SEN pupils compared to non-SEN pupils.

Findings: browsing time

• No difference in browsing time between matched or mismatched groups

• No difference also between pupils with different learning styles

• No difference between SEN and non-SEN pupils• Learning environment DID make a difference

– Global took less time to browse than sequential– Only by about 15 minutes though

(3¾ hours compared to 4 hours)– Supports H3 hypothesis

Correlations

• No correlation between knowledge gained and browsing time

• Significant negative correlation (r=-0.364, p<0.001) between post-test score and browsing time – attributed to lack of motivation

• Significant positive correlation (r=0.555, p<0.001) between number of pages visited and browsing time

Conclusions and discussion• No particular benefit to using sequential/learning styles

as personalisation mechanism in AEH• Outcomes of this study were based on “real users”• Flaws of the study:

– Reliability and validity of modified questionnaire– Dynamic nature of sequential/global learning style– The Hawthorne effect

• Two overarching problems:– Learning styles as a concept– The process of learning: a “wicked problem”

Future considerations

• A fundamental need for high-quality research:– Solid experimental design (see Robson, 2002)– Address the limitations/controversies of any aspect of

the study (theoretical/methodological etc)– Use “real” participants – not just those who are

convenient– Statistical analysis and effect size

In this class there is a serialist pragmatist kinaesthetic learner (who is also field-dependent, not to mention his MBTI) primarily a convergent thinker, high on logico-mathematical intelligence but low on linguistic intelligence, sitting next to a holist, reflector, primarily visual and field-independent... who is also chronically shy (no-one mentions that). Even assuming that such things can be assessed with some validity and reliability, which is itself far from clear — what are you going to do about it? There are after all thirty other students in the class, each of whom could be described in similar terms. And two-thirds of them are female, and one-third male (two of whom are gay). Five of the class are from ethnic minorities, two are dyslexic, one is visually impaired, and three are clinically depressed (although only one of them knows it). Six are "mature" students — at least, they are chronologically over 25.

Thanks for listening

Thanks and acknowledgements to: Tim Brailsford; Tony Fisher; Amir Pourabdollah;

Adam Moore; Cees van der Eijk, Sue Jones and Shaaron Ainsworth.

Also to all those students and pupils who participated in the user trials.

References• Brown, E. J.; Brailsford, T. J.; Fisher, T. and Moore, A. (2009). Evaluating learning style personalization in adaptive

systems: quantitative methods and approaches. IEEE Transactions on Learning Technologies (Special Issue on Personalization) 2(1) pp. 10–22. http://oro.open.ac.uk/30224

• Brown, E. J. (2008) PhD thesis: The Use of Learning Styles in Adaptive Hypermedia. http://etheses.nottingham.ac.uk/577/

• Brown, E., T. Fisher and T. Brailsford (2007) Real users, real results: examining the limitations of learning styles within AEH. Proceedings of the Eighteenth ACM Conference on Hypertext and Hypermedia (HT07), Manchester, UK, 10-12 Sept 2007, p57-66. [Awarded Theodore Holm Nelson Prize for Best Newcomer Paper] http://oro.open.ac.uk/29998

• Brown, E., T. Brailsford, T. Fisher and C. van der Eijk (2007) Revealing the hidden rationality of user browsing behaviour. Proceedings of the Eighteenth ACM Conference on Hypertext and Hypermedia (HT07), Manchester, UK, 10-12 Sept 2007, pp85-94. http://oro.open.ac.uk/29999

• Brown, E., T. Brailsford, T. Fisher, A. Moore and H. Ashman (2006) Reappraising cognitive styles in adaptive web applications. Proceedings of the 15th International World Wide Web Conference (WWW2006), Edinburgh, UK, 22-26 May 2006. http://oro.open.ac.uk/30002

• Coffield, F., D. Moseley, E. Hall and K. Ecclestone (2004) Learning styles and pedagogy in post-16 learning: A systematic and critical review. Learning & Skills Research Centre.

• Coffield, F., D. Moseley, E. Hall and K. Ecclestone (2004) Should we be using learning styles? What research has to say to practice. Learning & Skills Research Centre.

• Robson, C. (2002) Real World Research: A Resource for Social Scientists and Practitioner-researchers. Oxford, UK, Blackwell Publishing.

• Weibelzahl, S. (2005). Problems and pitfalls in the evaluation of adaptive systems. in Adaptable and Adaptive Hypermedia Systems. S. Y. Chen and G. D. Magoulas. Hershey, PA, IRM Press: 285-299.

• YouTube video: “Learning Styles Don't Exist” - Prof. Daniel Willingham: http://tinyurl.com/esteem-ls (7 mins long)