EVALUATING YOUR EVALUATING YOUR E-LEARNING COURSEE-LEARNING COURSE

LTU WorkshopLTU Workshop11 March 200811 March 2008

Common practiceCommon practice

End-of-course student questionnaire Standard version with last-minute adaptations:

changes to how the course elements are named including some additional items

Usually with similar problems… low return rates no follow up returns, and sometimes do not analyse

results poor quality responses limited enhancement function

Research shows…Research shows…

Evaluation of e-learning courses mainly affected by: distributed configuration of teams and

distance of students

Strategies for evaluating courses are problematic even at compliance levelSense of evolution – looking for the right strategyTeams worry about response rates but don’t use data availableUndefined responsibilities

Staff’s lack of skills, confidence & motivation

Common problemsCommon problems

Evaluation is not planned (in advance) as part of the overall course design

Feedback is gathered only from students Feedback is gathered only at the end of the

course/module Feedback is gathered only through traditional

methods Feedback requested do not consider all e-

learning aspects

Recommendations& Strategies

Collect feedback from tutors, Collect feedback from tutors, administrators and technical supportadministrators and technical support

Usual context: team meetings affected by distribution of staff (part time or at a distance).

Swinglehurst (2006) approach – tips for e-learning course teams:Formally organise frequent staff meetings, where tutors,

administrators and support staff meet to analyse course. These meetings could be face-to-face and/or online.

Define an agenda for each meeting, and rotate the member of staff responsible for this agenda, to cover key issues related to the different aspects of the course (e.g. student support, academic feedback, encouraging discussions)

Limit the recording of the discussion at these meetings to a minimum, briefly documenting the topics covered and the decisions made, and disseminate them.

Questionnaire to staff

Collect feedback during the run of the Collect feedback during the run of the coursecourse

Daly et al. (2006) approach - offers a simple and effective mechanism for the collection of student feedback during the run of a course:embedding evaluation activities, encouraging students to think about

how the course design/materials/activities have support their learning.Main benefits: to identify difficulties when they students are

experiencing them, and the opportunity to explore students’ experience.

Online learning diaries - running throughout the course, in which students are encouraged through brief questions to post their thoughts regarding their learning process and how the course has supported them.

Collect feedback at the end of the Collect feedback at the end of the course/modulecourse/module

QuestionnairesQuestionnaire/question features: language, questions

types Mode of application:

• Online (using an online survey tool)• By e-mail • Paper-based

Timing at which feedback is collected Responsibility for collection and analysis

Focus groups

Use alternative strategiesUse alternative strategies

Computer logs that the particular e-learning system in use provides (e.g. Course Statistics in the Blackboard VLE).Monitoring basic statistics such as last login date, number

of messages sent by users, areas of content and discussion boards/forums visited by users.

Statistics are not an indications of the quality of the student/tutor participation or of a satisfactory online experience – but useful tool for monitoring online ‘presence’, to obtain an overall picture of the ongoing activity, as well as to detect problems

Include aspects relevant to the use of Include aspects relevant to the use of technologytechnology

Wide range of aspects to include, and this list would vary depending on context and on the objectives and audience for the evaluation

Main issues: quality, usefulness and frequency of use of course components (online

activities, resources, face-to-face events, readings, online discussions/seminars, tutor support, technical support, etc.)

how well online activities run (timing, frequency, sequence, instructions, interactions, feedback, time on task, etc.)

e-learning experience (workload, involvement, online participation facilitators and restrictions, etc.)

role of tutors (engagement, feedback, support, etc.)

Warwick University and Bristol University’s guidelines IHEP’s Quality on the Line report (Phipps and Merisotis, 2000)UCL’s Quality Framework (Greenhalgh, 2001)