Learning from Usain Bolt: integrating ipsative assessments

into our learning

12th eAssessment Question

Ken CurrieOpus Learning (& CAPDM)

27th March 2014

1

This Talk

• Three parts:– Who are we, and why is eAssessment an issue for

Opus– What are we trying to do with eAssessment– Our approach to Delivery, Support and

eAssessment• Fourth theme?

– Implementation

2

Lots out there

• Online Learning and Assessments is a very active area– Lots of experts– Lots of non-experts– Huge amount of literature

• When I was asked to stand in for my colleague I thought– “what is there to add to an area pinned down by

such expertise?”

3

Useful Quote

• “Large scale online learning could provide continuity for learners throughout their lives between different education systems.” – Association of Learning Technology Draft Green

Paper 2013.• Then you realise that the game is changing

– Advancement is very fluid– As the owner of a problem we can make a

contribution

4

History of Opus Learning

• A spinout from CAPDM.com– Developing and delivering Masters programmes

for over 20 years• Opus delivers highly interactive courses

– Developing tools to manage and monitor learning– Marketing through partner colleges worldwide

• An SQA Approved Centre

5

Opus Learning

• Opus is a wholly online SQA Approved college– Offering HNC/Ds and other courses– With its own students and tutors

• Opus is also a B2B provider of– Ready packaged courses and programmes– A full own-brand ‘College in a Box’– Consultancy for aspiring SQA centres

• Opus /CAPDM is a custom developer & publisher of Online Distance Learning

6

Opus Business Model

• Focussed, highly scalable and repeatable• No significant technical limitations on its

geographic market• Minimal marginal costs of servicing additional

students– economies of scale for Opus and its partners

• Defence against price competition?– Quality of delivery systems, plus content

&structure of learning materials7

Partnering?

• Opus (and its sister company CAPDM) are:– Looking for technology partners, including Moodle

development– Looking for learning partners, including developing

more content and units• And assessment options!

8

Issues 1: the Design Challenge

• To build, deliver and manage– 99 HN courses for Opus– 18 masters level courses for Edinburgh Napier– … + others!

• To manage > 100 academics/teachers/authors• To manage very large domains of content• To deliver to many platforms

– Moodle, Blackboard, ePub, tablet/iPad, PDF, …

9

SQA: the Start Point

• SQA provide ‘arrangement’ documents for all– Frameworks (e.g. HND Business) and – Units (e.g. Business Accounting)

• Unit specifications of:– Outcomes– Knowledge and/or Skills– Evidence Requirements– + Assessment Guidelines

10

HND Business: Unit by Unit

• HND Business: each student has to– Sit 6 formal examinations– Deliver 6 documented portfolio exercises– Complete 22 other assessment reports

• total word count of 34,500

• An HE Diploma?– 12 modules each with final examinations or

assignment coursework

11

Assessments & the SQA

• Why are we trying to ‘change’ things?• Units are stand alone

– Can be delivered in a very flexible manner– But very clear pointers as to how assessment

should be carried out• No longer appropriate to base an assessment

of competence on a final three hour written examination?

12

Issues 2: Assessment

• Assessment can take many forms– Formative, e.g. through topic quizzes– Summative, e.g. exams and assignments– But not Ipsative, e.g. through comparison of self

with earlier marks?• Ipsative?

– I had to look this up:• “the practice of assessing present performance against

the prior performance” (Wikipedia)

• “Be the best that you can be” (www.ipsative.co.uk)13

Any link to Usain Bolt?

• No!• But we like his approach: Bolt, Sep 2013.

– "In The 200m, If I Can Master The Bend There Is Room For Improvement"

• Bolt plots his progress against every conceivable parameter on a daily basis– Continued, measured improvement– Should we not do the same?

14

Our Response: a Digital Work Book

• Longitudinal assessment– Augments examination and assignment

• Part of our ‘Social Learner’ aim– No faith in social learning (students creating the

learning content)– Great belief in breeding social learners– Many levels from ‘none’ to

• Interact with content for own ends (e.g. DWB)• Comment locally with others (e.g. Scotsman)• Interact and discuss with all peers (e.g. Forums)

15

Ipsative via the DWB

• Students build up a Reflective Log, over time– A longitudinal measure (Digital Work Book)– Continually improve it– Shared with the tutor (peers potentially)

16

A Component of Learning Analytics

• MOOCs are gathering global analytics– Measure everything but the individual

• Opus is looking to move towards developing the ‘Digital Self’– How you model yourself as an individual, and

access your data– How you are measured at all times– Our initial response – the DWB – begins to

recognise the potential of ‘always-on’ learning

17

Interact with Content Objects

Learner interacting with content, leaving traces

Student interacts with content

Makes notes, responds to quiz, send messages, etc

Capture notesStore in MLE

Print off record of learning The Digital Workbook

18

Social Networks

• Opus is looking to build 000s of small groups– MOOCs are looking for a group of 000s

• Our social networks are small scale but highly linked (see Anderson 2003)

• Our goal: 5,000 students, but– 100 centres with 50 students– Model the individual, or a small group

19

How do students interact when learning

Student

Environment

Content Tutor

Student/Student

Content/ContentTutor/Tutor

Environment/Environment

Student/Tutor

Tutor/Environment

Student/Environment

Student/Content

Content/Environment

Tutor/Content

Based on diagram inAnderson 2003 20

Big Data?

• The analytics of the individual are of more interest to us– Not specifically looking at group behaviour– Individuals, in depth and across courses

• Digital Self profiles may be limited– E.g. me to Facebook and Linked-In (no pedometer

around my waist!)– How do we enhance profiles?

21

Role of the Learning Materials

• Authors are encouraged to develop clear ‘concept maps’ for students to follow– Learning paths must go through key ‘concept

gateways’– Evidence of competent/critical thinking gathered

• The core is learning content that– Is interactive– Measures its own consumption– Builds ‘Digital Self’ data

• How to make it more intelligent? 22

Student Learning Aims

• Our aims for our learners– To see how they change over time– To teach these learners how to learn

• Their interface– Our content, systems and support– Must be smart

23

Analytics: Some Conclusions

• Analytics and tracking, via the DWB, etc., appears to– Provide the potential of personalised learning

within mechanical presentation of study– Permit formative assessment in the content,

driving tracking and learning progress– Provide feedback on Tutor effectiveness and the

effectiveness of the pedagogy underpinning the content

24

The Digital Workbook is

• A portfolio embedded in the content

• A dashboard for learning • Rich seam of data and

behaviours for mining• Ongoing formative

assessment, and potential summative assessment

• A permanent record (e.g. a PDF) for the student 25

Tracking through the learning content

• Opus content is massively connected to the students Work Book

• Every reflection, exercise, quiz, short response etc. is logged within the learning content and context

• Tutors can view, track and compare Work Books

26

Digital Work Book: its Role

• A key assessment tool – An integral part of the design and the content– Can be coupled with ‘concept gateways’

• Evidence gathering– For demonstration of competent/critical learning

• In the future it will become• our main assessment toolset• our main behaviour manager• But not the only one …

27

Learning Objective Profiling

• Build Learning Objectives into every course– Ensure that all content relates to at

least one LO– Weight all formative

assessmentquestions againstat least one LO

28

( c ) c o p y r ig h t C A P D M L td . 2 0 0 7

Lea rn in gOb j ec tiv es

C o u rse m a p

A

D

C

B

KLO1 KLO2

KLO3

KLO7

KLO10

KLO4 KLO5 KLO6

KLO8 KLO9

KLO11

Aggrega tion s

A’

D’

C’

B’

Lea rn in gOu tcom es

A ssessm en tA im s

Mapp ing?

Learning Objective Profiles

• Couple Learning Objectives to Personalised Feedback

• Developbalancedcoursesand assessment

29

Content-based Study Planner

30

• Encourages students to plan & plot their progress– Allows tutors a

‘helicopter view’ of student progress and an early warning of problems

– Use DWB to seethe detail

In Conclusion

• Do not just set exams and give badges– Track and Assess– Intervene and support– Audit and learn lessons

• Build a community for the whole qualification• Record learning and make available for future

generations• Continually improve learning content

31

The End

Ken CurrieOpus Learning

kwc@capdm.com

32

DWB Examples – Source Page

33

34

35

Assessor/Tutor Activity Model in Distance Learning

1. Monitor VLE activity, identify any candidate with lack of progress and contact course leader for action.

2. Monitor Digital Workbook (if used) and identify any outliers in progress, contact course leader with details.

3. Respond to direct tickets from candidate, making use of FAQs for responses, if new response is required, copy response to course leader.

4. Responding to assessments; remediation advice

36

Opus StrategySmall Private Open Courses

• Organise cohorts in groups of 10s, not groups of 10k’s or 100k’s

• Localise tutors and assessors• Build communities including assessors• Track activity and behaviour at keyboard level and at

cognitive level• Record activity – perhaps award a badge for partially

completed activity• Guide the learners towards completion, intervene as

necessary37

Creating the Social Learner

• Our DWB is part of a social learner strategy– Rich interaction spaces– Personal interaction with the content (DWB)– Interaction on specific issues (traversing the web)– Social learner (Forums, Facebook, etc.)

• Manage forum and virtual space activity

38