User Evaluation of OUM’s HTML Modules

towards proposing an Enhanced Model

Lim Tick MengWidad Binti Othman

Safiah Bt Md YusofWoo Tai Kwan

Datin Halimatolhanin binti Mohd Khalid

2

OUM’s Blended Mode of LearningOUM’s Blended Mode of Learning

The quality of print modules

determines OUM’s success in its

implementation of ODL

Our Research is guided by these two principles

• Open University Malaysia has been exploring the best possible means of instructional delivery that make ODL a preferred choice of learning

• It is necessary to leverage technology in the best possible way to enhance students’ learning

3

Background1. OUM is in the processing of converting the print

modules (more than 1500) to HTML modules

2. Since learning using modules constitutes 80% of learning time, it is vital to ensure that in any innovational change related to the use of modules, quality of the change in relation to effective learning should always be ascertained.

3. In evaluating the effectiveness of a learning system, learners should be involved as one of the key group of evaluators

4

1.1. What are OUM learners’ perceived quality of the web- What are OUM learners’ perceived quality of the web- based HTML modules from the dimensions of based HTML modules from the dimensions of technical, interface-design, learner-support technical, interface-design, learner-support andand instructional adequacy instructional adequacy respectively?respectively?

2.2. Is there any difference between male learners and Is there any difference between male learners and female learners in their overall perceived quality of female learners in their overall perceived quality of the web-based HTML modules?the web-based HTML modules?

3.3. How should the framework for the HTML modules be How should the framework for the HTML modules be designed to enhance its efficacy as a self-managed designed to enhance its efficacy as a self-managed learning tool and resource?learning tool and resource?

• Moving from behaviorist to constructivist learning

• Chickering and Gamson (1987) – The Seven Principles of Good Practice

• Mccombs and Vakili (2005)’s learner-centered framework

• Various guidelines and standards for evaluating online learning and online learning objects (example: Elissavet & Economides, 2003; Quiter & Weber, 2004; Nicole A. Buzzetto-More & Kaye Pinhey, 2006; Sternberger, C. (2006); Thair M. Hamtini, 2008)

6

The Seven Principles of Good Practice – Adoption till Today

• Encouraging contact

• Developing reciprocity and cooperation

• Encouraging active learning

• Providing prompt feedback

• Emphasizing time on task

• Communicating high expectations

• Respecting diverse talents and ways of learning

7

• Identify Scope of EvaluationIdentify Scope of Evaluation

• Prepare instrument for EvaluationPrepare instrument for Evaluation

• Administer user evaluation Administer user evaluation (survey) at various centres(survey) at various centres

Population of the study1. OUM learners who uses the HTML

modules for learning the compulsory subjects

– OUMH1103 – Learning Skills for Open and

Distance Learners

– OUMH1203 – English for Written Communication

– OUMH1303 – English for Oral

Communication

9

Sampling Method

Purposive and Convenient sampling– Survey forms distributed to all learners of

the 3 modules who were present at T3 (all learning centres)

–Analyses were carried out on the returned forms which were completed by the fifth tutorial

10

Aspects of Evaluation

Closed-Ended Items (40 items)– Technical Adequacy (7 items)– Interface-Design Adequacy (4 items)– Learner-Support Adequacy (7 items)– Instructional Adequacy (22 items)

Open-Ended Items (2 items)

11

The closed ended items

Adopted a four-point Likert Scale– 1 - Strongly Disagree

– 2 - Disagree

– 3 - Agree

– 4 - Strongly Agree

12

This is to avoid users to select the NOT SURE option

Cronbach’s Alpha For Items Measuring Each Aspect of Adequacy

Adequacy No of Items Cronbach’s Alpha

Technical 4 .916Interface-

Design7 .871

Learner-Support

4 .922

Instructional 22 .952

13

Profiles of the Evaluators (Learners)

14

Learner Evaluation of Adequacies

Learner Evaluation of HTML: Module Adequacies

16

t- test to compare the rating of Male and Female Learner Evaluators

• The analysis shows that there is significant difference between ratings of the male Learners (M=3.12, SD=.49) and the female Learners (M=3.01, SD=.57); t(815)= 2.689, p=0.007, d=0.20)

• The results suggest that the gender factor does have an effect on the perceived adequacy of the HTML modules. The Cohen’s d value of 0.02 however is indicative of a small effect size.

17

Problems Raised by Respondents

NO Problems using the HTML ModulesNumber of

Respondents

1 There is problem logging-in/launching the module 34

2 It is tiring watching the computer for too long 23

3 The system appears to be slow 214 There is no face-to-face discussion 85 We do not know when to use the learning

objects 8

6 There are not enough self-assessment activities 3 18

Suggestions by Respondents

NO Problems using the HTML ModulesNumber of

Respondents

1 Should have more interactive activities and exercises

11

2 Face-to-face instruction is still important 10

3 The print modules should also be provided 10

19

Implications and Suggestions (1)

1. Learning objects should be linked to the module at appropriate nodes and links, not just placed at repository and dissociated from the module content.

2. It needs the subject matter expert to work hand-in-hand with the instructional designers to accomplish the above task

20

Implications and Suggestions (2)

3. To cater for learners of varying learning styles, the level of interactivity of the learning objects should be enhanced in the following ways:– Contains microworld that allows exploratory

learning

– Provides different responses for different inputs

– Provides varying levels of entry for different ability groups

21

Implications and Suggestions (3)

4. Assessment activities should not be just providing feedback in the form of scores. It should be “intelligent” enough to provide different responses for different inputs. The responses should be meaningful in that it guides learning

5. The HTML modules should have the functionality to “inform” learners of their own performance as well as their performance as compared to the peers

22

Implications and Suggestions (4)

5. To provide an alternative to face-to-face instruction, mini-lectures in the form of Video (FLV) should be embedded

6. The source of mini lectures could be– Effort from the academics (i-lecture)– Identify quality YouTube lectures– Other Open Sources

23

24

Web-Video Lecture(YouTube Resource etc)

HyperlinkedLecture

EmbeddedLecture

WebResource

Hyperlink EmbeddedLO0

HyperlinkScore

Database

Module Page (1)

Module Page (2)

Module Page (3)

Online Forum and Synchronous

Communication Tool

WebResource

MyVLE Environment

Web Environment

OnlineAssessment

That ‘s all,Thank You