1

CHAPTER ONE

Introduction

1.1 Background

The Communicative Language Teaching (CLT) approach has been widely discussed

as well as applied in numerous ways in language classrooms around the world since the

1970s. The CLT approach is a commonly recognized approach in the field of English as a

second language/foreign language (ESL/EFL) (Brown, 2001). The impact of the CLT

principles is evident in many aspects— not only in teaching/learning activities, but also in

material development. Indeed, teaching and learning materials are continuously being

reexamined to determine whether they conform to the CLT characteristics.

Meanwhile, recent advances in computer technology have dramatically changed the

learning and teaching environment in English education over the past four decades. The

relationship between technology and language teaching/learning has interested a great

number of second and/or foreign language educators. With the rapid development of

multimedia and the Internet, terms such as web-based learning, educational multimedia,

hypermedia, and courseware have been coined to refer to learning activities that involve

specific computer tools and facilities. In addition, the emergence and development of

Computer-Assisted Language Learning (CALL) has drawn a significant amount of attention

in the field of second/foreign language learning for years. Numerous CALL-related studies

have been conducted, and web-based language learning materials are increasingly being

developed. Just like other types of language teaching and learning materials, CALL materials

are expected to be learner-centered, interactive, and authentic (Brown, 2001; Levin & Bruce,

2001; Levy, 1997). According to Bradin (1999), it is important for language teachers to look

at CALL software as critically as they look at textbooks.

2

Software companies continue to produce new packages, including CD-ROMs and

online courseware, while a large number of instructional software packages are released

every year. In this web-based learning era, language teachers frequently adopt and adapt

online courseware as one of the teaching tools, for use either in or outside the classroom; one

of the advantages of online courseware is that it is not constrained by space and time.

Learners can access online courseware almost anywhere at any time they choose— outside the

classroom, at home, or at a self- learning center on campus. Learners can thus develop their

autonomy and become independent learners, one of the ultimate goals in education (McDevitt,

1997); an effective user-centered CALL tool should be able to guide learners toward this

autonomy. However, the wide selection of online courseware available creates challenges for

language educators in choosing appropriate tools to suit different instructional contexts and

various kinds of learners. Evaluators’ evaluation criteria are crucial. Therefore, appropriate

evaluation criteria and evaluation tools have become essential.

When selecting and evaluating suitable online CALL courseware, two issues play an

important role: policy considerations and evaluation criteria. Policy and decision makers,

such as stakeholders, make selections based on a series of relevant concerns. In Taiwan,

English teachers, administrators, and principals have been involved in the decision-making

process regarding English teaching materials; those who are not language teachers tend to

focus on costs as well as the designer/publisher’s reputation. In addition, government policy

affects the decision-making; for example, the Ministry of Education (MOE) may deve lop and

publish its own textbooks for schools or develop curriculum guidelines that may affect

decisions about instructional materials. Consequently, teachers often complain that the

chosen textbooks’ and/or courseware’s content do not meet teachers’ requirements. Moreover,

learners’ views and needs are frequently ignored as they are not practically involved in the

process of material selection and evaluation.

3

Evaluation criteria also play a significant role; what kinds of criteria are to be

included are quite relevant. Since CALL adopts a multidisciplinary perspective (Levy, 1997),

incorporating applied linguistics, second language acquisition (SLA), education, information

technology (IT), and computer science, it requires both pedagogical and technical criteria,

such as whether the courseware offers a clear learning goal, is based on a certain learning

theory, or helps learners achieve learning objectives. Furthermore, the degree of technical

difficulty and user- friendliness must be considered.

1.2 Statement of the Problem

In recent years, the impact of the CLT approach has encouraged language teachers to

pay more attention to learner-centeredness in language teaching and learning. Meanwhile, the

user-centeredness of CALL tools has drawn increased attention. For instance, Squires and

Preece (1996) emphasize the importance of considering the way students learn and the

usability of the courseware design in creating natural and intuitive interaction between

learners and the courseware. Heller (2005) also claims that a CALL tool should “work

flawlessly with a user- friendly design and self-explanatory user manual for instructors and

users” (p. 120). Although the literature focuses on the need for user-centered tools in CALL

(Hémard, 2003), many CALL tools do not function as they should. According to Felix (2003),

many CALL tool designs do not follow established learning theories or principles; instead,

designers focus more on technical features rather than on the evaluation of these CALL tools

with the help of users.

Consequently, adequate evaluation of CALL materials is becoming increasingly

crucial. Only can user- friendly and learner-centered courseware attract learners to use it

outside the classroom, thereby accessing its advantages. Thus, language educators and/or

institution administrators should have suitable guidelines or criteria to follow when

evaluating and selecting web-based courseware; these guidelines or criteria should lead the

4

evaluation process in the direction of learner-centeredness/user-centeredness. Since the

learner is the center of the language learning process as well as the main user of CALL

materials, evaluating programs and materials from the learner’s standpoint is necessary. In

other words, evaluation criteria for CALL courseware should be aligned with the perspective

of learner-centeredness.

1.3 Purpose of the Study

The purpose of this study is two-fold. First, the study aims to provide language

teachers with evaluation criteria for online CALL courseware based on the concept of

learner-centeredness. Second, the study expects to increase the awareness of learner-

centeredness among English teachers by using a developed learner-centered evaluation

checklist to examine the commercial courseware My English Tutor (MyET) (L Labs, 2006).

1.4 Significance of the Study

Despite the fact that many CALL-related evaluations have been conducted, limited

research exists discussing issues relevant to online CALL courseware evaluation in Taiwan—

particularly in regards to learner-centeredness. Indeed, among the few studies conducted in

Taiwan, most have focused on evaluating CD-ROMs (e.g., Chang, 1992; Lin, 2002; Su, 2003;

Yeh, 2002) or websites (e.g., Tsai, 2002) and have focused on specific issues such as

pronunciation training (e.g., Hong, 2003).

Unlike previous studies, the current study focuses on online courseware and aims to

propose a checklist based on the premises of learner-centeredness for online courseware

evaluation. As such, it seeks to make several contributions to the field of English language

teaching, including: 1) connecting the learner-centered principles with CALL research, 2)

raising awareness among language instructors of learners’ needs and learner-centered design,

and 3) providing a set of specific learner-centered evaluation criteria for online CALL

courseware that are more specific than the current general evaluation criteria.

5

1.5 Definition of Terms

The study operationally defines the following terms:

CALL Tools

This term refers to any tools used on a computer basis and aimed at enhancing

language learning, such as CD-ROMs, software, online courseware, and websites.

Courseware

Longman Dictionary of Language Teaching and Applied Linguistics (Richards, Platt,

& Platt, 1998) defines courseware as “computer programs used in computer assisted

learning” (p. 113). According to the online glossary of the Learning Technology Jargon

Buster (2006), courseware refers to “computer software and associated materials designed for

educational or training purposes.” Similarly, in the E-learning Glossary (2006), courseware

means “educational software that delivers course material and instruction via computer.” In

short, courseware is a word coined from course and software, referring to software with

teaching and learning purposes and functions. In this research, MyET is regarded as CALL

courseware as it is designed for use as an English instructional program.

Online Courseware

Online courseware is courseware used in a web-based environment— i.e., on the

Internet.

Learner/user

In this study, learner represents those who learn English as a second/foreign language

while user refers to those who use CALL courseware to study English. In the CALL

environment, the user in a broad sense may include the learner, teacher, and courseware

administrator. However, the user in this study refers only to the learner. Thus, the learner and

the user are interchangeable in this study.

6

Teacher-evaluators

This term refers to the three teachers involved in this study who conducted the

evaluations (see section 3.2.2).

Student-evaluators

This term refers to the 22 college students involved in this study who conducted the

evaluations (see section 3.2.2).

1.6 Organization of the Thesis

This thesis consists of five chapters. Chapter One states the background and purpose

of the study as well as its significance and operational definitions of terms. Chapter Two

provides an overview of the literature on learner-centeredness and CALL, emphasizing the

salience of learner-centeredness in a CALL tool. It also discusses certain issues relevant to

English teaching/learning material evaluation, including the classification of existing

evaluation approaches and evaluation criteria for ESL/EFL materials, by reviewing previous

studies related to English textbook evaluation and CALL courseware evaluation. Finally, it

presents the research questions based on the review of the literature. Chapter Three describes

the research design, including the rationale, participants, data collection and analysis

procedures, and validation in three phases. Chapter Four presents the learner-centered

evaluation checklist developed in this study as well as the results of applying the checklist to

the evaluand, MyET. In addition, it examines the effectiveness of the checklist as well as

answers and discusses the research questions. Finally, Chapter Five draws conclusions and

offers implications to language teachers, language institution stakeholders, CALL tool

designers, and material developers; it further identifies the limitations of the study and

proposes suggestions for further research in this area.

7

CHAPTER TWO

Review of the Literature

This chapter reviews the literature on learner-centeredness in education and its

relationship with CALL along with evaluation issues impacting English teaching/learning

materials. Research questions are proposed at the end of the chapter based on the review of

the literature.

2.1 Learner-centeredness in Education

Learner-centered concepts are instructional design and teaching practices based on

cognition and learning (Mynard, 2003). The concept of learner-centeredness formally

emerged in 1990 from the American Psychological Association’s (APA) Presidential Task

Force on Psychology in Education (Lambert & McCombs, 1998). One of the task force’s

purposes was to develop general principles to establish a framework for school redesign and

reform. Twelve fundamental principles about learners and learning were proposed and later

revised in 1997. Now the document includes 14 principles, which are categorized into four

domains essential to learning─metacognitive and cognitive factors, affective and

motivational factors, developmental and social factors, and individual differences factors

(McCombs, 2000) as shown in Table 2.1. A learner-centered model embracing five premises

was shaped (McCombs & Whisler, 1997)─summarized in Table 2.2.

8

Table 2.1

The learner-centered psychological principles

Cognitive and Metacognitive Factors

Principle 1: Nature of the learning process.

The learning of complex subject matter is most effective when it is an intentional process of

constructing meaning from information and experience.

Principle 2: Goals of the learning process.

The successful learner, over time and with support and instructional guidance, can create meaningful,

coherent representations of knowledge.

Principle 3: Construction of knowledge.

The successful learner can link new information with existing knowledge in meaningful ways.

Principle 4: Strategic thinking.

The successful learner can create and use a repertoire of thinking and reasoning strategies to achieve

complex learning goals.

Principle 5: Thinking about thinking.

Higher order strategies for selecting and monitoring mental operations facilitate creative and critical

thinking.

Principle 6: Context of learning.

Learning is influenced by environmental factors, including culture, technology, and instructional

practices.

Motivational and Affective Factors

Principle 7: Motivational and emotional influences on learning.

What and how much is learned is influenced by the learner’s motivation. Motivation to learn, in turn,

is influenced by the individual’s emotional states, beliefs, interests and goals, and habits of thinking.

Principle 8: Intrinsic motivation to learn.

The learner’s creativity, higher order thinking, and natural curiosity all contribute to motivation to

learn. Intrinsic motivation is stimulated by tasks of optimal novelty and difficulty, relevant to personal

interests, and providing for personal choice and control.

Principle 9: Effects of motivation on effort.

Acquisition of complex knowledge and skills requires extended learner effort and guided practice.

Without learners’ motivation to learn, the willingness to exert this effort is unlikely without coercion.

9

Table 2.1 (continued)

Developmental and Social Factors

Principle 10: Developmental influence on learning.

As individuals develop, they encounter different opportunities and experience different constraints for

learning. Learning is most effective when differential development within and across physical,

intellectual, emotional, and social domains is taken into account.

Principle 11: Social influences on learning.

Learning is influenced by social interactions, interpersonal relations, and communication with others.

Individual Differences Factors

Principle 12: Individual differences in learning.

Learners have different strategies, approaches, and capabilities for learning that are a function of prior

experience and heredity.

Principle 13: Learning and diversity.

Learning is most effective when differences in learners’ linguistic, cultural, and social backgrounds

are taken into account.

Principle 14: Standards and assessment.

Setting appropriately high and challenging standards and assessing the learner and learning progress

─including diagnostic, process, and outcome assessment─are integral parts of the learning process.

Source: McCombs, B. L. (2000). Assessing the role of educational technology in the teaching and

learning process: A learner-centered perspective. Paper presented on Secretary’s Conference on

Educational Technology 2000.

http://www.ed.gov/print/rschstat/eval/tech/techcon00/mccombs_paper.html

10

Table 2.2

Premises of the APA’s learner-centered model

Premise One

Learners are distinct and unique, and this distinctiveness and uniqueness must be taken into account.

Premise Two

Learners’ uniqueness includes their emotional states, learning rates, learning styles, abilities, talents, and

needs.

Premise Three

Learning is a constructive process, so, what is being learned should be relevant and meaningful to the

learner and be based on the learner’s prior knowledge and experience.

Premise Four

Learning occurs best in a positive environment in which the learner has positive interpersonal interactions

and feels appreciated and validated.

Premise Five

Learning is a natural process; learners are naturally curious and interested in learning about their world.

Source: Lambert, N. & McCombs, B. L. (Eds.). (1998). How students learn: Reforming schools

through learner-centered education. Washington, DC: American Psychological Association.

Educators define learner-centeredness as “the perspective that couples a focus on

individual learners [… ] with a focus on learning. This dual focus then informs and drives

educational decision making” (McCombs & Whisler, 1997, p. 9). McCombs and Whisler’s

definition is based on the understanding of the Learner-centered Psychological Principles as a

demonstration of the knowledge base on the learner and learning” (McCombs, 2000).

Bransford et al. (1999) propose that learner-centeredness refers to the environments focus ing

on knowledge, skills, attitudes, and beliefs that learners bring to an educational setting.

Besides, learner-centered environments include teachers who are aware of the knowledge,

beliefs, understandings, and cultural practices that learners construct and bring to the learning

environment. Sparks and Hirsh (1997) used the term learner-centeredness to refer to learning

new beliefs and new visions of practice corresponding to the needs of students and teachers

as learners. In short, learners must be involved in the educational decision-making process;

11

individuals’ differences should be respected and accounted for in the learning process.

In the language teaching field, Nunan (1997) explains that, in learner-centered

environments learners are actively involved in the learning process while the focus is on

the learner. In other words, language teaching is now seen as language learning, placing

the learner at the center of attention in language education.

In learner-centered instruction, the spotlight has shifted from the teacher to the

learner, concentrating on what learners need and how they learn rather than on what the

teacher wants to teach. This transformation does not devalue the teacher’s professional

role, but instead attempts to encourage learners to move toward a primary goal of

education: becoming an autonomous learner. Learner-centered teachers recognize the

importance of building on the knowledge that students have already accumulated during

the learning process. In this regard, “teachers no longer function as exclusive content

expert or authoritarian classroom managers” (Weimer, 2002, p. 14).

The Learner-centered Model (see Table 2.1 and Table 2.2) is claimed to be

research-validated as it integrates research and theory from psychology and education

(McCombs, 2000). Research (e.g., Bain, 2004; McCombs & Whisler, 1997; McCombs,

2000; Palmer, 1998) has identified the positive effects of learner-centeredness on

education. Bain’s (2004) research, for instance, focuses on college teachers who adopted

learner-centered instruc tion and concludes that these faculty have a strong faith in

student ability, creating a safe learning environment allowing students to try. In addition

to identifying the benefits of learner-centered practice, McCombs and Whisler (1997)

point out that the abundant evidence has proved that “motivation, learning, and

achievement are enhanced where learner-centered principles and practice are in place”

(p. 62). For example, in McCombs and Stiller’s study (in McCombs & Whisler, 1997),

teachers report that with a learner-centered approach there is less student disruption and

12

fewer discipline problems. McCombs (2000) further emphasizes the positive feedback

of learner-centeredness between the learner and the teacher as well as the importance of

an encouraging climate of learning both in and outside the classroom. A report provided

by the Committee on Academic Programs and Teaching (CAPT) Learner-centered Task

Force of University of Southern California (2006) also indicates the positive results of

the practice of learner-centered teaching embedded in units across the school.

2.2 Learner-centeredness and CALL

The emergence and transformation of CALL is intertwined with the trends in

educational theory and language acquisition theory at different times. Warschauer and

Kern (2000) use three metaphors to describe the three prominent stages of development

in the CALL field─a tutorial metaphor (computer-as-tutor), a construction metaphor

(computer-as-pupil), and a toolbox metaphor (computer-as-tool) :

From the 1950s to 1960s, behaviorism and the structural perspective dominated

education and language teaching, greatly impacting the development of CALL. Many

software developers built numerous drills and practice exercises consisting of grammar

and vocabulary tutorials into their programs to emphasize repetition in learning (Levy,

1997; Warschauer & Kern, 2000). The earliest CALL programs “strictly followed the

computer-as-tutor model” (Warschauer & Kern, 2000, p. 8).

During the 1970s, influenced by cognitive/constructivist views of learning, the

agency in CALL programs was shifted from the program to the learner. Learning

software designers realized that the computer was something to be controlled by the

learner instead of being a controller over the learner. More resources and functions

along with interactive tasks (interacting with computers) were incorporated into CALL

programs. This period defined the computer-as-pupil model generation.

13

Currently, the sociocognitive approach is influencing the language teaching

profession, and communicative competence has become a buzzword in this era

(Warschauer & Kern, 2000). Language educators have started to focus more on

providing authentic contexts and materials to enhance learners’ communicative ability.

Learners no longer merely interact with computers; more importantly, they interact with

other humans via computers. Under this model of computer-as-tool, computers offer

learners access to information and to other people. For example, many learning websites

and web-based learning environments provide a computer-mediated communication

(CMC) function, using computers to assist authentic communication between two or

more people. Technological advances like this have changed language learners’

activities and made autonomy and independence in learning become something

achievable (Benson & Voller, 1997).

A great number of studies demonstrate the positive and effective impact that

technology brings to learning (McCombs, 2000). Indeed, previous research has

indicated that computer-related technology can facilitate language learning by

supporting learner-centered practice in two primary aspects: 1) researching and creating

an authentic product for assessment and 2) providing an unrestricted number of sources

on most topics (Encyclopedia of Educational Technology, 2005). Recently, researchers

(e.g., Felix, 1999; Norlan, 2000) have focused on the learner/user rather than the

technology. According to Felix (1999), the focus has shifted to what the learner can do

with the technology and not what the technology can do. Such fundamental shifts have

led to increased attention to learner-centered design, user-centeredness, and/or user-

friendliness of CALL tools.

Since computer-related technology is meant to serve as a learning tool to

promote effective learning, CALL tool developers should involve the personal domain

14

of a learner and address learners’ needs in the design process. As McCombs (2000)

states, the learner-centered framework serves as a reminder that the human issues can

never be left out of the technology-supported networked learning environments. Easy-

to-use, relevance to the user’s life, meaningful learning activities, clear instructions

combined with vivid and interactive interfaces are necessary features of high-quality

courseware, as such courseware— with its learner-centered design— focuses on the tasks

and goals of learners, motivates them, and encourages them to take more responsibility

for their own learning when using the CALL tool. Consequently, when technology is

regarded as a learning tool instead of a teaching approach, it has the capacity to greatly

facilitate learning by providing individualized opportunities for students (Nolan, 2000).

In this regard, CALL tools can become more usable for learners. Usability has

been the focus of CALL courseware developers, researchers, and language educators. A

number of researchers (e.g., Felix, 2003; Heller, 2005; Hémard, 2003; Nielsen, 2000;

Rubin, 1988) have pointed out the importance of high-quality usability of CALL tools.

The usability of a CALL tool involves the features related to the user from the user’s

perspective; as such, users play a crucial role in the effectiveness of CALL courseware.

Only CALL courseware that places the learner at the center of the learning process can

effectively enhance learning.

Despite the increased attention on usability, user-centeredness, and user-

friendliness in the CALL area, little existing literature discusses the integration of

learner-centeredness into CALL tool evaluation. Moreover, few studies examine CALL

courseware from the perspective of learner-centeredness. The following section reviews

literature relevant to English material evaluation in general as well as that specific to

CALL material evaluation.

15

2.3 ESL/EFL Material Evaluation

Tomlinson (1998) defines material as “anything which is used by teachers and

learners to facilitate the learning of a language” and “anything which is deliberately

used to increase the learners’ knowledge and or experience of the language” (p. 2). This

section provides an overview of the literature related to ESL/EFL material evaluation. It

begins by discussing evaluation approaches─and thus providing some important

evaluation terms and concepts─before investigating evaluation criteria in general and

those for online courseware

in particular.

2.3.1 Evaluation Approaches

Evaluation approaches can be classified according to evaluation instruments,

objectives, timing, and the evaluator’s perspective. First, general evaluation instruments

include checklists, interviews, and evaluation frameworks and models based on certain

specific guidelines or principles. A checklist is one of the most common types of

evaluation; it provides a list of items that include important points to be examined.

Checklists vary in content, length, style, purpose, and target, but all have been

developed in an attempt to help language teachers, students, and/or even administrators

to choose appropriate materials for instructional use. Traditional checklists have

received criticisms (Oliver, 2000); for example, they assume all criteria are of equal

importance, allowing no gradation of the degree to which a specific evaluation object

meets the different criteria, and are not able to consider the contextual conditions of

learning (McDougall & Squires, 1995; Tergan & Schenkel, 2003). However, they serve

as one of the most frequently used methods for evaluations because of their ease of use.

Interviews cover basic information about the material and the reviewer’s

subjective description by offering a direct manner to obtain evaluators’ responses

16

(Reeder, Heift, Roche, Tabyanian, Schlickau, & Golz, 2004). Some researchers who

have carried out evaluations through interviews have found that many insights can be

gained in utilizing this approach (Le & Le, 1997). However, although interviews enrich

research data, one of the drawbacks is that their dynamics and flexibility may result in

inconsistency and cause the difficulty in analyzing data (Patton, 1980).

In the recent CALL field, several researchers (e.g., Chapelle, 2001; Hubbard,

1988; Squires & Preece, 1996) have proposed evaluation frameworks and/or models for

CALL tools instead of using certain evaluation criteria.

Second, the objective of evaluations can be addressed via formative and

summative evaluations. According to Trochim (2006), formative evaluations aim to

strengthen or improve the object being evaluated, while summative evaluations focus on

the effects or outcomes of the object. Lau (2000) contends that formative evaluations

investigate the feedback on any factors influencing teaching and learning processes and

then design and develop an instructional technology efficiently, whereas summative

evaluation aims to justify the implementation of the instructional technology.

Another way to distinguish evaluation approaches concerns timing— namely,

through predictive and retrospective evaluations. The former is conducted before

materials are adopted, whereas the latter functions as a reflective way to examine

learning materials, which is more empirical (Ellis, 1998). Instructors frequently use

predictive evaluation before selecting courseware. Squires and McDougall (1996) define

it as “the assessment of the quality and potential of a software application before it is

used with students” (p. 147).

Finally, evaluation approaches can be categorized in terms of the evaluator’s

perspective, classified as either judgmental or empirical (Chapelle, 2001). In a

judgmental analysis, the analysis is usually done by material developers and teachers;

17

CALL software is often the object of such evaluation. Chapelle (2001) used an example

question to explain this type of subjective analysis: “Does the software provide learners

the opportunity for interactional modifications to negotiate meaning?” (p. 53).

Responses to this evaluation question vary depending on the evaluator. On the other

hand, an empirical analysis is normally conducted by collecting data from learners, such

as by observing learners’ performance when they are using the courseware. Empirical

evaluation methods are extremely diverse, including case studies, pilot projects, project

monitoring, field studies, replicated experiments, synthetic experiments, product

simulations, and process simulations. This type of evaluation is usually time consuming.

It is important to note that judgmental and empirical evaluations are complementary

rather than mutually exclusive (Chapelle, 2001).

2.3.2 Evaluation Criteria

With a great number of language-learning materials on the market, the

responsibility of identifying which materials to use and how to use them falls upon

language teachers and school stakeholders (e.g., administrators). Numerous evaluation

methods and suggestions as well as evaluation criteria have been proposed for

evaluating learning materials; some focus on general material evaluation while others

are used for specific materials such as textbooks, websites, or online courseware. This

section reviews evaluation criteria for ESL/EFL textbooks and web-based CALL

courseware, respectively.

It is necessary to include literature on textbook evaluation in this section for

three reasons. First, although the features of textbooks differ from those of online

courseware, the learning theories and teaching methodology are/should be similar. In

addition, research on ESL/EFL textbook evaluation emerged much earlier than that of

CALL courseware evaluation; as such, many discussions have examined the relation

18

between the learner and the textbook (e.g., Cunningsworth, 1984; Ellis, 1998; Littlejohn

& Windeatt, 1989; Sheldon, 1988; Tomlinson, 1998), and these may provide insights

into CALL evaluation. Finally, teachers should evaluate web-based learning materials as

seriously as they do any print materials (Bradin, 1999; Susser & Robb, 2004).

Criteria for Textbook Evaluation

Abundant literature exists on ESL/EFL textbook evaluation. Researchers have

suggested various aspects for evaluating and selecting English materials; to a great

degree, these researchers’ evaluation criteria are compatible. For instance,

Cunningsworth (1984) provided a checklist, which includes 70 questions, encompasses

7 areas: language content, selection and grading, presentation and practice, the

development of language skills and communicative abilities, supplementary materials,

learners’ motivation, and overall evaluation (pp. 75-79). Meanwhile, Littlejohn and

Windeatt (1989) identified six perspectives as fundamental in the evaluative approach:

general or subject knowledge in materials, views on the nature and acquisition of

knowledge, views on the nature of language learning, role relations implicit in materials,

opportunities for the development of cognitive abilities, and values and attitudes in

materials.

Subsequent researchers have followed quite similar points of view as these

previous studies. For example, Ansary and Babaii (2002) collected ten ESL/EFL

material evaluation checklists, proposed by Williams (1983), Skierso (1991), Ur (1996)

and others from 1970 to 2000; these checklists served as the basis of their study. Table

2.3 summarizes the studies on ESL/EFL textbook evaluation criteria reviewed in this

section; a summary of the global categories for ESL/EFL textbook evaluation,

developed by the researcher, is provided in Table 2.4.

19

Table 2.3

Summary of studies on ESL/EFL textbook evaluation reviewed in this study

Name (Year) Theme of the Study Results

Ansary & Babaii

(2002)

Systematic Textbook Evaluation A set of universal features of

EFL/ESL textbooks proposed

Cunningsworth

(1984)

Evaluating and Selecting EFL Teaching

Materials

Seven areas and 70 questions

focusing on learners’ needs and

processes proposed

Ellis

(1988)

Empirical Evaluation of Language

Teaching Materials

Guidelines and process for evaluation

proposed

Littlejohn & Windeatt

(1989)

Material Selection Six perspectives along with 6

questions suggested

Sheldon

(1988)

Evaluating ELT Textbooks and Materials A checklist of 17 categories and 53

criteria suggested

Skierso

(1991)

Textbook Selection and Evaluation A checklist of 11 categories and 104

items developed

Tomlinson

(1998)

Materials Evaluation Principles and process of developing

criteria proposed

Ur

(1996)

Assessing a Coursebook Guidelines and a checklist of 19

criteria provided

Williams

(1983)

Developing Criteria for Textbook

Evaluation

A checklist with 28 criteria proposed

20

Table 2.4

Global categories for ESL/EFL textbook evaluation

1. Theoretical background/Approach of the textbook

2. Goals and objectives of the textbook

3. Content of the textbook

4. Activities and tasks of the textbook

5. Author(s)/Developer(s) of the textbook

6. Supplementary materials and aids of the textbook

7. Physical Make-up/Presentation of the textbook

8. Administrative Concerns

9. Overall Value

In addition to the global textbook evaluation categories in Table 2.4, several

researchers and educators have pointed out the necessity of evaluating the teaching/learning

materials from learners’ standpoints and the importance of embracing the learner in the

evaluation process. In other words, learners’ needs, learning purposes, language levels, age,

cognitive state, and learning processes should also be taken into consideration. For example,

Cunningsworth (1984) suggests that learners’ needs and processes relate to teaching materials.

Meanwhile, Littlejohn and Windeatt (1989) argue that an evaluation approach should “look

beyond goals of language learning itself” and include issues such as “learners’ perceptions of

knowledge, language learning and roles” (p. 174). After reviewing the literature on the

evaluation of learning materials in general, the following section discusses evaluation criteria

specifically related to online CALL courseware.

Criteria for Online CALL Courseware Evaluation

CALL combines technology and language education, creating the distinctive features

of new-generation learning tools such as e- learning tools. Reeder and his colleagues (2004)

21

have distinguished pedagogical traits from technical ones in e- learning media. The former

include the proximity that web-based courseware offers to “authentic or simulated linguistic

and cultural settings” (p. 256); thus, CALL programs can promote a high degree of

interaction between the program and the learner. The latter include the “extensive use of the

multimedia capacities of computers, including complex graphic elements [… and… ] fairly

sophisticated sound elements” (p. 257).

On one hand, due to the unique characteristics of CALL tools, the approaches to

evaluating CALL courseware should differ from the way in which traditional English

materials are investigated; on the other hand, the principles and criteria for textbook

evaluation and CALL courseware evaluation in fact overlap to a certain degree, revealing

their essential roles in ESL/EFL material evaluation. This can be seen in, for example, Susser

and Robb (2004), who advise a practical and comprehensive framework with five modules

and items to checkmark for evaluating websites for language skills. The first three modules

overlap with each other as well as with evaluation criteria of textbooks; the fourth and the

fifth modules concern courseware and online courseware specifically.

Current literature highlights at least two deficiencies in the current approaches to

CALL courseware evaluation: the lack of a link between courseware design and instructional

methodology and the lack of new approaches to evaluating new genres of CALL courseware.

The first problem emerges from the fact that most evaluative criteria fail to link the design of

courseware to its pedagogical methodology; few CALL courseware evaluations provide a

methodological framework (Reeder et al., 2004). A few researchers have examined this issue,

attempting to link pedagogy to CALL tool evaluation. For instance, Hubbard (1988)

developed a framework for CALL courseware development and claims that courseware

evaluation needs to be related to pedagogical principles and their relevant constituents. Using

SLA theories, Chapelle (1998) proposed seven hypotheses and corresponding suggestions for

22

the development of multimedia CALL; issues of learning and learners play an important part,

which— as she expects— have provided “a starting point for development of a complementary

relationship between SLA research and CALL practice” (p. 33). Chapelle (2001) later sheds

light on the need for improvements in CALL tool evaluation and suggests five principles of

evaluation for evaluating CALL tools.

The second problem relates to the lack of a new and adequate approach to evaluating

CALL courseware (Reeder et al., 2004). Due to the distinctiveness of CALL tools, evaluation

criteria cover a wide range of issues. A few researchers have endeavored to propose

evaluation frameworks or models, instead of checklists. For example, Reeder and his

colleagues (2004), focusing on empirical evaluations, suggest a research agenda consisting of

four steps: 1) constructing an instrument to describe CALL materials, 2) building a

theoretical framework and taxonomy, 3) developing a suite of new instruments and guidelines

for evaluation derived from the framework, and 4) testing the suite of instruments and

guidelines on CALL materials in a setting with a representative range of users (p. 265).

However, instead of specifying evaluation criteria, this model simply provides guidelines on

how to conduct an empirical evaluation of CALL courseware.

In addition to instructional concerns, technical issues play an important role in CALL

courseware design and evaluation as well. Numerous researchers (e.g., Bradin 1999; Hémard,

2003; Nielsen & Mack, 1994; Squires & Preece 1999) have incorporated technological

criteria in CALL evaluation. These technological facets take into account 1) usability

(Nielsen, 2000)— the measure of the quality of the user experience when interacting with a

website or software, 2) accessibility (Susser & Robb, 2004)— aspects that facilitate or prevent

people with disabilities from accessing and using the online courseware, and 3) functionality

(Bradin, 1999)— easy navigation, rapid loading, and the quality of multimedia design.

23

A review of the current literature on evaluation frameworks and checklists for online

CALL courseware/websites─summarized in Table 2.5─ reveals both diversity and similarity.

However, overall it can be said that the evaluation criteria proposed in these studies fall into

two main categories— pedagogical and technical— followed by several subcategories. These

are illustrated in Table 2.6.

24

Table 2.5

Summary of studies on online CALL courseware evaluation reviewed in this study

Name (Year) Theme of the Study Results

Bradin

(1999)

Instructional Aspects of Software

Evaluation

A two-step checklist with follow-up

criteria proposed

Broadbent

(2002)

Decision Making Step by Step A checklist of 40 criteria proposed

Chapelle

(2001)

Principles for CALL Evaluation Five principles of evaluation

developed

Elissavet & Economides

(2003)

An Evaluation Instrument for

Hypermedia Courseware

A checklist of 124 items presented

Hémard

(2003)

Language Learning Online:

Designing towards User

Acceptability

A checklist for web design evaluation

provided

Hubbard

(1988)

An Integrated Framework for

CALL Courseware Evaluation

An evaluation framework consisting 3

major sections offered

Jolliffe et al.

(2001)

Selecting and Reviewing Learning

Resources

A checklist of 45 criteria proposed

Khan & Vega

(1997)

Factors to Consider When

Evaluating a Web-based Instruction

Course

Thirty-six evaluation criteria with

follow-up questions indicated

Murray & Barnes

(1998)

Evaluating Multimedia Language

Learning Software from a

Pedagogical Viewpoint

A framework with 15 questions

presented

Nielsen & Mack

(1994)

Heuristic Evaluation A usability engineering method

proposed

25

Table 2.5 (continued)

Odell

(1986)

Evaluating CALL software A checklist with two 2 main

categories─technical design and

Pedagogic evaluation─offered

Reeder et al.

(2004)

Toward a Theory of E/Valuation for

Second Language Learning Media

A research agenda consisting of 4

steps suggested

Robb

(1996)

ESL Software Evaluation An evaluation form with 29 criteria

offered

Squires & Preece

(1999)

Evaluating the Potential of

Educational Software

The Jigsaw Model proposed

Susser & Robb

(2004)

Evaluations of ESL/EFL

Instructional Websites

A framework with 5 modules and

checking items advised

Table 2.6

Global categories for online CALL courseware evaluation

Pedagogical Concerns

1. Curricular Issues

2. Learning Theory

3. Teaching Methodology

4. Accuracy

5. Learners’ Choices

6. Authenticity

7. Interactivity

8. Feedback

9. Authority

10. Bias/Stereotypes

Technical Considerations

1. Usability

2. Presentation/Physical Makeup

3. Navigation

4. Speed

26

As Table 2.6 shows, global categories for online CALL evaluation fall into two areas:

pedagogical concerns and technical considerations. This section will provide further

descriptions of these categories and subcategories.

Pedagogical concerns. Pedagogical concerns involve those issues regarding language

teaching and learning in the courseware:

n Curricular issues (e.g., Broadbent, 2002; Elissavet & Economides, 2003) include

the goal, objectives, and purpose of the courseware; the target users/learners; and

the skills focused on (listening, speaking, reading, writing, or integration of certain

skills). Evaluation criteria should consider whether the scope and aims of the

courseware are explicit, whether the purpose of the courseware is stated clearly,

whether the courseware fulfils its purpose, who the intended users are, what kind of

skills are being developed, what the coverage (e.g., topics) of the courseware

includes, and whether the courseware can be integrated into a curriculum.

n Learning theory (e.g., Tomlinson, 1998) refers to the theoretical background of

learning behind the courseware, such as behaviorist, cognitivist, and sociocognitive

approaches; this must be taken into account in the courseware evaluation to ensure

that the materials provided by the courseware fall in line with the educational

system’s approach to learning.

n In addition, the teaching methodology (e.g., Susser & Robb, 2004) adopted in the

courseware is another essential criterion. Teaching methodology concerns the

approach/method the learning tool adopts and thus affects the learning outcome to a

great degree.

n Accuracy (e.g., Bradin, 1999; Robb, 1996) refers to issues such as whether the

content is presented accurately, whether the information is reliable, and whether the

information sources are salient and noteworthy.

27

n Learners’ choices (e.g., Glidenston, 1994; Robb, 1996; Shin & Wastell, 2001)

addresses the freedom given to users in the learning process, such as whether the

user is allowed to control the order of presentation and whether the user is allowed

to skip certain activities.

n The authenticity (e.g., Jolliffe et al., 2001; Murray & Barnes, 1998) of the learning

materials in the courseware— as well as whether the materials are provided in

appropriate contexts— is vital for providing meaningful input for learners.

n Taking advantage of technological advances, CALL courseware should be

interactive (e.g., Elissavet & Economides, 2003; Susser & Robb, 2004), providing

learners with opportunities to interact with the computer or with others via the

computer.

n Feedback (e.g., Broadbent, 2002; Gildenston, 1994) is a vital element in the

courseware as appropriate feedback enhances learners’ motivation. Issues like

whether error handling is meaningful and helpful and whether immediate feedback

is offered for an online language- learning tool are important.

n Authority (e.g., Reeder et al., 2004) is another essential component in courseware.

The absence of language professionals or instructors on a courseware design team

creates a gap between design and actual use. When evaluating CALL courseware,

one has to ponder whether the information on the author(s) is available and

sufficient, whether the author(s) has trustworthy education, expertise, or experience

in related fields, and whether the author(s) can be contacted for further information.

n Finally, bias or stereotypes (e.g., Gildenston, 1994; Jolliffe et al., 2001; Murray &

Barnes, 1998) in the form of the material developer’s background, knowledge, and

attitudes toward language learning and different cultures can significantly impact

the content presented in the courseware.

28

Technical issues. Most existing evaluation criteria concern four primary technical

issues: usability, presentation/physical makeup, navigation, and speed.

n Usability (e.g., Nielsen & Mack, 1994) is an umbrella term covering different but

comparable meanings in different fields and sub-fields. The terminology related

to usability has changed with various shifts in the field. In the 1980s and 1990s,

the buzzwords were functionality, usability, usability engineering, and user-

centered design; in the 2000s, the term user experience engineering started to

appear. Scholars encouraged practitioners to “look beyond traditional usability

concerns, such as ease of learning and effectiveness, to broader user issues, such

as aesthetics, collaboration, accessibility, credibility, persuasion, and pleasure”

(Wilson, 2005, p. 6). Nielsen’s (2000) well-known heuristic model provides

guidelines for software design with a high degree of usability. In short,

evaluating the usability of CALL courseware covers the following issues:

whether the user interface is easy to use, whether the information is easy to

access, whether the online courseware is fee-based, whether a user must register

a name and password before using it, whether the instructions or guide are clear

and easy-to-follow, whether the learner has choices in using the courseware (e.g.,

sound volume and text font), and whether the courseware offers different

pathways corresponding to different learning styles/needs/paces/levels.

n The presentation/physical makeup of the courseware can impact learners’ ability

to access content. Presentation/physical makeup (e.g., Robb, 1996) ranges widely

from the layout (word font, color, format, spacing, etc.) to multimedia (graphics,

sounds, audio/video files, animation). Important criteria focus on whether the

web pages or information presented is comprehensible, whether the layout of the

courseware is attractive to users and facilitative for learning, and whether the

29

format is consistent.

n Related to presentation/physical makeup is navigation (e.g., Chapelle, 2001;

Robb, 1996). The quality of navigation of online courseware plays a crucial role,

and criteria such as how easy it is to move from page to page and/or from link to

link, whether all internal and external links work properly, and whether any

incorrect links or dead ends exist need to be considered.

n The speed and ease (e.g., Broadbent, 2002) in downloading a web page is critical

for web-based courseware.

As stated earlier the criteria in the studies reviewed here can be divided into two main

categories─pedagogical and technical. These two facets are like the wings of a bird; only

when they are equal in strength can the bird succeed in flying. Thus, a balanced view of these

two evaluation schemes is necessary.

2.4 Research Questions

Recent studies and practices of CALL material evaluation have come into a new era;

consequently, researchers continue to produce new evaluation models for CALL courseware.

Yet most such models and frameworks emphasize or incorporate general aspects. Since

learners are the center of the language learning process as well as the primary users of

courseware, evaluating a program from the learners’ perspective is essential. Language

teachers and those involved in the decision-making process must keep the learner in mind

when evaluating and selecting CALL materials.

Based on the review of the literature, this study claims that the concept of learner-

centeredness in language teaching is no different from the idea of user-centeredness in CALL.

Nonetheless, few researchers have discussed this connection between learner-centeredness

and CALL evaluation. Therefore, this study aims to explore the following questions:

30

1. What are the criteria for evaluating online CALL courseware from the perspective

of learner-centeredness?

2. Can a learner-centered evaluation checklist based on such criteria assist in

evaluating the learner-centeredness in online courseware, using MyET as an example?

The methodology used to answer these questions is described in the following chapter,

and Chapter Four discusses the research results that answer these questions.

31

CHAPTER THREE

Methodology

3.1 Introduction

In order to answer the research questions identified in the previous chapter, this study

adopted primary qualitative research methods consisting of three stages; at each stage,

specific data were collected for analysis. This chapter outlines the three phases (see Table 3.1)

as well as the participants, data collection and analysis procedures, and validation of results at

each stage. The first stage involved developing a checklist aimed at collecting evaluation

criteria associated with the characteristics of learner-centeredness, followed by the

establishment of an evaluation checklist focused on learner-centeredness. The second stage

involved piloting the checklist developed in the first stage by applying the checklist in

evaluating the commercial CALL courseware MyET; the evaluation results of MyET in terms

of its learner-centeredness were analyzed during this stage. The third stage involved a meta-

evaluation─evaluating the effectiveness of the checklist itself; during this stage, evaluators’

feedback on the checklist was collected for analysis. The discussion of the validation of

results addresses how the validity of the analysis was double-checked or confirmed by

another English teacher or group of people.

32

Table 3.1

An overview of the three-stage study

Stage I. Development of the Learner-centered Evaluation Checklist

Stage I aimed to develop an evaluation checklist based on the premises of learner-

centeredness. More than 45 studies were reviewed and a checklist of 39 items was developed.

Data collection and analysis during this stage lasted from July 2006 to December

2006.

Stage II. Application of the Evaluation Checklist: Evaluation of MyET

Stage II aimed to pilot the constructed evaluation checklist developed in Stage I.

Three teachers and 22 non-English-major students from a university in northern Taiwan used

the checklist to conduct an evaluation of MyET. Some descriptive statistical analysis was

conducted on the quantitative data resulting from use of the checklist.

Data collection was done in January 2007 and data analysis was conducted from

February 2007 to April 2007.

Stage III. Meta-evaluation of the Evaluation Checklist

Stage III aimed to explore the effectiveness of the evaluation checklist. The teachers

were interviewed and the students were given three open-ended questionnaire questions in

order to elicit their feedback on the checklist.

Data collection was done in January 2007 and data analysis was conducted from

February to April 2007.

3.2 Stage I. Development of the Learner-centered Evaluation Checklist

The first stage aimed to develop an evaluation checklist based on the characteristics of

learner-centeredness. More than 45 sets of evaluation checklists, models, guidelines, and

studies were reviewed and 39 criteria generated.

3.2.1 Rationale

The rationale for creating a learner-centered checklist is based on two main concerns.

First, as mentioned in Chapter Two, although a great number of evaluation checklists and

models have been proposed in the ESL/EFL literature, most share numerous criteria in

common; moreover, their purpose is to help language educators evaluate whether materials

are adequate or not. The coverage of the evaluation criteria is generally broad and lacks the

33

basis of learning theories (Tergan & Schenkel, 2003). However, ESL/EFL materials are

expected to be learner-centered; therefore, it is necessary to incorporate this characteristic

into the evaluation criteria so that language teachers can keep learners in mind while

evaluating and selecting appropriate materials for their students. Ideally, students are

practically involved in the evaluating and selecting process; however, in reality this seems

rare in the EFL settings in Taiwan. EFL materials or textbooks are usually chosen by English

teachers without students’ involvement due to economic and efficiency issues.

In addition, the need exists for a new genre of evaluation tools for the new generation

of CALL materials. For example, experts have asserted that user- friendliness and usability are

crucial components of CALL tools. It has also been argued that most evaluation checklists

focus on technical rather than on learning and educational issues (Tergan & Schenkel, 2003).

Therefore, it is necessary to link instructional and methodological considerations to technical

designs in a CALL tool. This study attempted to develop an evaluation checklist, based on

existing software/website evaluation checklists and literature on learner-centeredness as well

as ESL/EFL material evaluation, for online CALL courseware. Language educators may thus

have another evaluation tool to assist them in evaluating and selecting suitable CALL tools

for learners.

This study adopted the form of a checklist as the evaluation tool. Although traditional

checklists have received several criticisms (see section 2.3.1), a checklist, compared to other

evaluation methods, is the most frequently used tool due to its convenience and ease (Tergan,

2003). This study implemented a checklist as the evaluation tool but at the same time it also

left room for evaluators to provide comments and feedback in order to attain more descriptive

data regarding each evaluation decision made on the checklist.

34

3.2.2 Data Collection

Since CALL is a multidisciplinary area (Levy, 1997), the range of data collection at

this stage covered a wide scope, including current literature and studies in the fields of

applied linguistics, second language acquisition (SLA), and information technology (IT). Two

primary areas of data were collected: evaluation criteria for ESL/EFL textbooks and for

multimedia learning materials— e.g., software and online courseware, from general

educational materials to English learning courseware. More than 45 sets of evaluation

checklists, models, and guidelines were collected and reviewed during this stage (see

Appendix I for the sources adopted and adapted).

3.2.3 Data Analysis

Each evaluation criterion, item, or guideline of the data collected was individually

reviewed to determine whether it conformed to the premises of learner-centeredness

discussed in section 2.1. After a review of the options, the basic format of the checklist was

established. Based on the literature review, evaluation criteria fell into two main categories:

pedagogical and technical aspects, each with specific subcategories (see Table 2.6). The

developed evaluation checklist followed the categorization.

Those criteria that seemed most able to help discriminate whether learning materials

or tools were learner-centered were selected as evaluation items for the checklist and placed

under different subcategories. For example, one criterion in a reviewed study said, “Is

feedback on learner responses encouraging?” (Son, 2003) and another said, “The hypermedia

courseware provides opportunities for interaction” (Elissavet & Economides, 2003); criteria

such as these correspond to the Premise Four of the APA’s Learner-centered Model:

“Learning occurs best in a positive environment in which the learner has positive

interpersonal interaction and feels appreciated and validated.” Thus, these criteria were

selected as items under the category feedback of the checklist. On the other hand, another

35

evaluation checklist asked, “Is the content accurate?” and “Is there information on the

author?” (Son, 2003). Although such criteria are not inadequate, they seem irrelevant to the

Premises of the APA’s learner-centered model. Thus, such questions were not included in the

evaluation checklist. The wording was modified when necessary.

3.2.4 Construction Process

The study adopted a 4-point Likert-scale ranging from 0 (very unlikely) to 3 (very

likely). Using an even number of options helps avoid evaluators’ tendency to choose the

neutral score available on an odd-numbered scale (Brown, 2000). To make the study more

meaningful, both teachers and students— the main users of the courseware— participated in

the study in order to increase the validity of the study by triangulating different data sources,

even though the checklist was aimed at language educators’ use.

Slight differences existed in the design between the checklists used by the teacher-

evaluators and those by the student-evaluators (see Appendices I and II for comparison). The

teacher-evaluators used a Chinese-English version, with headings specifying the main

categories and subcategories. The checklist for student-evaluators was modified somewhat to

be shorter and more concise; categories were not listed, and all descriptions were in Chinese.

This modification stemmed from the fact that a long checklist might exhaust student-

evaluators; moreover, descriptions in Chinese were deemed more comfortable for the

students.

In addition, time constraints made it impossible to interview all students about their

feedback on MyET. Therefore, an additional evaluation question was included in the student-

evaluator checklist— “In general, this courseware is helpful for learners; namely, it is able to

meet learners’ needs or enhance English learning” (see Appendix II)— to obtain their overall

opinion on the courseware being evaluated. At the end of the student-evaluator checklist were

three open-ended questions used for eliciting feedback on the checklist. The three open-ended

36

questions and the rationale for adding them are presented in section 3.4.3.

3.2.5 Validation

Because the data analysis and construction process were subjective, it was necessary

to obtain further validation of the analysis. After the evaluation tool was developed, a college-

level English teacher reviewed the checklist and recommended revisions to wording and

grammar; these recommendations were followed. For example, the teacher suggested

changing the word “easy-to-access” into “easy to access”. The checklist was subsequently

proposed and discussed at a meeting1 with several professors and graduate students of

Tunghai University, Taichung, Taiwan. Revisions to wording and the number of evaluation

items were again made according to the feedback and suggestions received from the meeting.

The original checklist included about 50 items; 39 items were left after these modifications.

The deleted items were considered irrelevant to learner-centeredness. In addition, a table

matching each evaluation criterion with one or more premises of the APA’s learner-centered

model developed to reveal the process of construction of the checklist (see Appendix III).

Afterwards, a colleague also helped match the criteria with the premises of learner-

centeredness in order to double check the researcher’s decisions. All the items matched were

the same except item I, 2.4; one thought it went with Premise Two, and the other thought it

corresponded to Premise Four. Otherwise, both shared agreement to a great degree. Another

university teacher looked over the table as a further check on the content of the validity of the

checklist.

3.3 Stage II. Application of the Evaluation Checklist: Evaluation of MyET

The second stage aimed to pilot the constructed evaluation checklist and to provide

the 25 participants an experience of using the checklist. The participants conducted an

1 The meeting was an informal review session held on November 3, 2006, with graduate students and professors in the Department of Foreign Languages and Literature of Tunghai University, Taichung, along with visiting faculty member Dr. Antony Kunnan.

37

evaluation of MyET using the checklist. Some descriptive statistical analysis was conducted

on the quantitative data resulting from use of the checklist.

3.3.1 Rationale

The checklist was tried out in order to attain actual experience of using the developed

checklist. The online CALL courseware MyET was chosen as the evaluand. MyET was

designed and developed by L Labs in Taiwan. Founded in 2002, the team members of L Labs

come from various countries such as Taiwan, Canada, the U.S., Japan and Korea and all hold

master or doctoral degrees. L Labs serves as a “technology provider” working with “content

providers” and English materials developers (L Labs, 2006). MyET functions as a platform

for material developers and publishers to offer target users accessible interfaces. Thus far,

several material providers— such as Studio Classroom, International Community Radio

Taipei (ICRT), and Longman (Pearson)— have adopted MyET to offer online learning courses,

although their target users and pedagogical purposes vary. For instance, General English

Proficiency Test (GEPT) Courses provided by Studio Classroom aim to familiarize learners

with the test style and model of GEPT, whereas Studio Classroom Standard Course gives

online learning materials for learners’ use in self-study to improve English proficiency.

Technologies used in this online courseware focus primarily on developing learners’

listening and speaking skills. One feature that L Labs claims differentiates the courseware

from other online courseware in Taiwan is its Automatic Speech Analysis System (ASAS),

which is designed to train learners’ pronunciation and speaking skills by analyzing and

recognizing the sounds of the user, indicating the problems of each sound and syllable as well

as providing suggestions for improvement. Dr. Paul Pimsleur ’s learning method is adopted as

the theoretical background of MyET, with the claim that the way of learning a language by

repeating sentences via the courseware is able to assist learners to build a way of learning

similar to learning a native language (L Labs, 2006). Except for a few free trial lessons, all

38

users have to pay to get an authorized number to access MyET courses (see Appendix VI for

a few sample pages of MyET).

This study adopted MyET as the evaluand for several reasons. First, it is a locally

developed online English self- instruction tool in Taiwan, although the development team

includes professionals from various countries. The target users of this CALL courseware are

EFL learners, mostly in Taiwan and other Asian countries. Second, since entering the market,

the courseware has reportedly registered over one hundred thousand users in Asia, including

more than ten local colleges and high schools and ten corporations (L Labs, 2006). It seems

to have gained popularity over the past few years, as more and more self- learning centers and

language labs in Taiwan have been offering MyET as part of their supplementary learning

materials for students. In addition, most of the online materials’ authors or providers (e.g.,

Studio Classroom and Longman) are recognized English publishers who have historically

used magazines and books as teaching and learning tools and have now expanded to include

MyET. It is assumed that these providers have evaluated the potential of MyET. Given the

diversity and scope of online courses, much remains to be seen as to whether MyET is able to

meet users’ expectations. As such, the current study adopted MyET as the evaluand and

expected that the results of evaluation would provide some practical implications for the

CALL field in Taiwan.

This study originally intended to evaluate three different courses under MyET.

However, the participants had used only two: Studio Classroom Standard Course and Let’s

Talk in English by Studio Classroom. Therefore, the study was adapted accordingly. The

difference between the two courses is merely the level of difficulty and the texts (readings

and dialogues). Since MyET serves as a platform, the two courses are presented with the

same interfaces and learning activities, such as repetitions and role-plays. The degree of

learner-centeredness— if any— of the two courses and of the overall interface design of

39

MyET was evaluated using the checklist.

3.3.2 Participants

Twenty-five participants— 3 teachers and 22 students— were involved in the second

stage of the study. The three English teachers (teacher-evaluators) work in National Taiwan

Normal University, Taipei, Taiwan; the school has adopted several MyET courses for its

English self- learning center. In addition to MyET, the self- learning center also has adopted

other courseware for students. Two of the teachers have about ten years of teaching

experience; one of them is the director of the English self- learning center. The third teacher

has taught in the school for one year. All of them have had experience using MyET as

supplementary materials in class; one teacher had his students use MyET for seven to eight

months, one had had his students use MyET for four months before the interview was

conducted, and the third one for two months. Since all three teachers had used MyET, it was

not necessary to train them on how to use the courseware. The three teachers evaluated the

Studio Classroom Standard Course, one of the courses under MyET.

The remaining 22 participants (student-evaluators) were non-English-major freshmen

at National Taiwan Normal University, from a Basic English class taught by one of the

teacher-evaluators. They were all students in the low-achievement class and had been using

Let’s Talk in English, another course under MyET, for three months. They were required to go

to the self- learning center to use Let’s Talk in English as outside-class materials. These

students might go to the self- learning center anyt ime they wanted; the times they went were

recorded. The teacher even offered incentives for those who used the courseware most. Since

different courseware offer different functions and focuses, the teacher of the class asked his

students to use MyET mainly for pronunciation and speaking practice. He expected his

students’ interest in English learning to increase via diverse learning tools with different

functions. This group of learners might represent the majority of MyET users in English self-

40

learning centers in colleges, Taiwan. Since one of the purposes of a self- learning center is to

offer various self- learning and supplementary materials for learners, low-achievement

learners, in particular, need such assistance, whereas high-achievement and highly motivated

learners tend to have developed their own ways of learning and are relatively active in

learning.

3.3.3 Data Collection

The instrument adopted at this stage was the checklist developed during Stage I. The

differences between the teacher-evaluator and student-evaluator versions were addressed

earlier in section 3.2.4. In addition to the checklist, a formal semi-structured interview with

each teacher-evaluator was conducted before the teacher-evaluators applied the checklist in

order to obtain general information about their background and concepts about learner-

centeredness as well as to ensure that they understood the objective of the evaluation and the

meaning of each criterion in the checklist. The interview consisted of three primary questions:

1) How long have you been teaching English? 2) How long have your students been using

MyET? and 3) What do you think about learner-centeredness? All interviews were audio

recorded with a tape recorder and conducted in Chinese (see Part I in Appendix IV for

transcripts, including an English translation). Notes were taken when necessary during the

interviews.

Meanwhile, data from the student-evaluators were collected from their evaluations of

MyET using the checklist. The 22 participants were asked to do the checklist during class

time. They were told that the researcher and their English teacher would like to know their

feedback on MyET. They were also reminded that three additional open-ended questions

enclosed in the checklist were concerning the evaluation checklist itself. As soon as they

completed filling out the checklist, they handed it to the researcher. Unlike with the teacher-

evaluators, no interviews were conducted with the student-evaluators.

41

3.3.4 Data Analysis

The results from the teacher-evaluators’ and student-evaluators’ ratings of MyET were

entered into and analyzed with Microsoft Excel. The evaluation results were analyzed and

categorized based on the scores received. The categories included: strengths of the

courseware, weaknesses of the courseware, and non-scored evaluation criteria. The findings

are presented and further discussed in Chapter Four. Because the courses evaluated by the

teachers and students differed, this study did not attempt to compare the evaluation results of

the two courses with each other. However, except for the difficulty of the content, all the

tasks and interfaces of the two courses were similar. Thus, it should be possible to discuss to

what degree the two groups of raters share their opinions of MyET as a whole.

3.3.5 Validation

Although the current study aimed to increase the awareness of learner-centeredness

among English educators as they evaluate courseware, learners’ opinions were collected in

order to triangulate them with teacher-evaluators’ evaluations. According to Patton (1980),

triangulation is “a powerful solution to the problem of relying too much on any single data

source or method” (p. 193). Involving the primary users’ opinions should provide further

insights into learners’ needs as well as expectations. In addition, to confirm the findings, a

colleague of the researcher helped double-check the results using Microsoft Excel to ensure

that the numbers entered and the quantitative results, including total scores and the mean of

each criterion, were accurate. This colleague also helped review the findings selected by the

researcher based on the rankings of the ratings (see section 4.2 for findings and discussion).

3.4 Stage III. Meta-evaluation of the Evaluation Checklist

The third stage aimed to investigate the effectiveness of the evaluation checklist.

Interviews and written open-ended questions were adopted respectively to elicit the

participants’ feedback on the checklist itself.

42

3.4.1 Rationale

This study’s main focus is to develop an evaluation checklist based on the premises of

learner-centeredness; as an evaluation tool, it was essential to investigate whether the

checklist was able to function as expected and planned. Thus, an evaluation of the

evaluation— a meta-evaluation (Stufflebeam, Madaus, & Kellaghan, 2000)— was necessary.

Meta-evaluation can be defined as a critical review of the evaluation design and

implementation. At the same time, since the primary purpose of the study is to reinforce the

awareness of learner-centeredness in CALL eva luation, Stage III investigated whether the

checklist was able to draw the evaluators’ attention to learner-centeredness.

3.4.2 Participants

The participants at Stage III were the same as those in Stage II (see section 3.3.2).

3.4.3 Data Collection

To obtain participants’ feedback on the effectiveness of the checklist, a semi-

structured interview with seven open-ended questions for each of the three teacher-evaluators

was conducted after they used the checklist. The initial interview questions are provided in

Table 3.2. Nonetheless, any interview— regardless of whether it is structured or semi-

structured— has its own dynamics during the process itself. Consequently, the actual

interview questions differed slightly from the proposed ones (see Part II in Appendix VI for

transcripts, including an English translation). Interviews were used to obtain data during this

stage because they are the most direct way to pursue in-depth information (McNamara, 1999).

They also allowed for flexibility in asking follow-up questions based on interviewees’

responses. All interviews were tape recorded and conducted in Chinese.

43

Table 3.2

Interview questions of the second interviews with teacher-evaluators

1. Are these evaluation questions easy to answer?

2. What can be done differently in a revised version?

3. Do you think the checklist can focus your attention on learner-centeredness?

Why or why not?

4. Have you used any other evaluation checklists for ESL/EFL materials?

If yes, what are the differences between the one or ones you used and this one?

5. What are the strengths and weaknesses of this evaluation checklist?

6. Do you think this evaluation checklist will be helpful for English educators

in evaluating and selecting a suitable online courseware? If so, how?

7. What do you think of the evaluation checklist in general?

To collect the data of the student-evaluators’ perspectives on the checklist, three

written open-ended questions were added to the checklist. The following offers the English

version of the three questions.

1. Generally speaking, do you find these evaluation items easy to answer? Why/Why

not?

2. Do you think this evaluation checklist is able to focus the attention of teachers and

students on learners’ needs and conditions?

3. In addition to these evaluation items, what concerns should be taken into account

when a teacher is evaluating and selecting suitable learning materials to meet

learners’ needs?

The rationale for adopting open-ended questions stemmed mainly from time

constraints related to interviewing all 22 student-participants. Also, conducting only selective

interviews would require determining who should be selected, which might cause another

issue affecting the study. Open-ended questions offered all participants room to provide

comments on the questions.

44

3.4.4 Data Analysis

After collecting the data through interviews with the teachers, the researcher first

uploaded the audio-recorded tape into the computer and saved it as a Microsoft.WAV file.

The interviews were then played via the Microsoft Media Player program and selectively

transcribed by the researcher herself following certain basic principles: first, only main

interview questions and follow-up questions as well as the interviewee’s responses to these

questions were transcribed. Questions and conversations unrelated to the research were

omitted— e.g., previous studies that the teacher-evaluator’s students did. Second, redundant

words/phrases, such as discourse markers— e.g., mmm (“嗯” in Chinese)— were omitted.

Third, any repetitions or answers requiring self-corrections by the speaker were omitted; only

the corrected words/phrases/sentences were kept. The transcripts were translated into English

and marked with line numbers for ease of reference (see Appendix VI). The responses from

the interviewees were reviewed and divided into several categories based on the purpose of

each interview question. The categories generated include: overall opinions, strengths of the

checklist, weaknesses of the checklist, and suggestions for further revisions.

On the other hand, to analyze data collected from the student-evaluators, certain

categories based on their responses on the three open-ended questions were set up. The

process for categorizing was as follows. First, all of the comments were listed and reviewed.

Responses embracing the same meaning, although maybe expressed differently, were

gathered together. For example, “okay” (“還好” in Chinese) and “all right” (“尚可” in Chinese)

were regarded as the same category. This process resulted in several categories (see section

4.4.2). Then, these comments were counted and converted into numbers (see Table 4.10).

3.4.5 Validation

A colleague of the researcher assisted in the review of the transcripts, comparing the

transcripts and the audio-tape to ensure that the transcripts covered all the main points of the

45

interviews. No revisions were done. Another college- level English teacher read through the

whole transcripts and marked which category each response belonged to— e.g., overall or

specific, and strengths or weaknesses— with the goal of double-checking the categorization

generated by the researcher. The teacher’s categorization and the researcher’s were exactly

the same.

3.5 Summary

This chapter has described the research design and process of the three-stage study.

The following chapter will discuss the findings, including the establishment of the learner-

centered evaluation checklist, the evaluation results of MyET and the effectiveness of the

developed checklist. In addition, the research questions are answered and discussed.

46

CHAPTER FOUR

Findings and Discussion

4.1 Introduction

This three-stage study aimed to answer the research questions proposed in Chapter

Two— namely, 1) what are the criteria for evaluating online CALL courseware from the

perspective of learner-centeredness? and 2) Can a learner-centered evaluation checklist based

on such criteria assist in evaluating the learner-centeredness in online courseware, using

MyET as an example? Chapter Three presented the research design used to answer the

questions. This chapter will state and discuss the results from each stage, including the

developed learner-centered checklist, the evaluation results on MyET, and the meta-

evaluation of the evaluation checklist. Section 4.2 will address the first research question,

section 4.3 the findings from the MyET evaluations, and section 4.4 the second research

question. Table 4.1 provides an overview of the findings and discussion.

47

Table 4.1

An overview of the findings and discussion

Stage I: Development of the Learner-centered Evaluation Checklist

1. Features of the Evaluation Checklist

2. Definitions of the Subcategories

3. Summary of Stage I

Stage II: Application of the Evaluation Checklist: Evaluation of MyET

1. Teacher-evaluators’ Evaluation Results

Strengths of the Courseware

Weaknesses of the Courseware

Non-scored Evaluation Criteria

Controversial Issues

2. Student-evaluators’ Evaluation Results

Strengths of the Courseware

Weakness of the Courseware

Non-scored Evaluation Criteria

3. Caveats about Stage II

4. Summary of Stage II

Stage III: Meta-evaluation of the Learner-centered Evaluation Checklist

1. Teacher-evaluators’ Feedback

Overall Opinions toward the Evaluation Checklist

Strengths of the Evaluation Checklist

Weaknesses of the Evaluation Checklist

Suggestions for Further Revisions

Teacher-evaluators’ Concepts on Learner-centeredness

2. Student-evaluators’ Feedback

Overall Opinions toward the Evaluation Checklist

Positivity about the Evaluation Checklist

Negativity about the Evaluation Checklist

Additional Comments

3. Summary of Stage III

48

4.2 Stage I: Development of the Learner-centered Evaluation Checklist

To answer the first research question, the learner-centered evaluation checklist, which

includes 39 evaluation criteria, was constructed. Table 4.2 highlights all the evaluation

categories and items of the checklist (see Appendix I for the checklist used by the teacher-

evaluators). To allow for a triangulation of data, another version of the checklist was

generated for student-evaluators’ use in order to attain students’ feedback (see section 3.3.5

and Appendix II for the checklist used by the student-evaluators). The versions include

certain differences: 1) neither the main categories/subcategories nor English translations were

included on student-evaluators’ checklist, for simplicity, and 2) whereas teacher-evaluators

had the chance to provide feedback on MyET during face-to-face interviews, student-

evaluators responded to an additional evaluation criterion— “In general, this courseware is

helpful for learners; namely, it is able to meet learners’ needs or enhance English learning” on

the checklist (see item No. 40 in Appendix II).

49

Table 4.2

Learner-centered evaluation checklist for online CALL courseware

I. Pedagogical Criteria

1.1 The information on the objectives is sufficient for learners. 0 1 2 3

1.2 The objectives of the course are associated with learners’

learning development.

0 1 2 3

1.3 The objectives are achievable for learners. 0 1 2 3

1

Learning

Objectives

1.4 The objectives reflect what learners need to know. 0 1 2 3

2.1 The learning activities are able to arouse learners’ interests in

participation.

0 1 2 3

2.2 The instructions of the learning activities are clear for learners. 0 1 2 3

2.3 The guidance and models of learning activities are clear for

learners.

0 1 2 3

2.4 There is a variety of learning activities for learners with varied

learning styles.

0 1 2 3

2

Learning

Activities

2.5 There is a variety of learning activities for learners with multiple

intelligences (e.g., interpersonal, mathematics… .).

0 1 2 3

3.1 The content matches learners’ English proficiency level. 0 1 2 3

3.2 The content is related to learners’ life experience. 0 1 2 3

3.3 The content covers a variety of topics that are able to attain

learners’ interests.

0 1 2 3

3.4 The content is culturally appropriate for learners. 0 1 2 3

3.5 The content accommodates learners with various learning styles. 0 1 2 3

3.6 The content helps learners build up strategic ability. 0 1 2 3

3.7 The content stimulates recall of prior learning. 0 1 2 3

3

Content

3.8 The content enhances learners’ cognitive development. 0 1 2 3

4

Skills

4.1 The course offers skill-training that is able to meet learners’

needs.

0 1 2 3

5.1 The course offers immediate and proper feedback on learners’

performance and/or responses.

0 1 2 3

5.2 The course offers feedback that is able to motivate learners. 0 1 2 3

5

Feedback

5.3 The course offers meaningful and helpful error handling which

is able to help learners improve.

0 1 2 3

50

5.4 The course offers feedback that can be saved and accessed easily

so that learners may realize what difficultie s they have in the

learning process.

0 1 2 3

5.5 The course offers an opportunity for learners to give personal

feedback on their learning experience.

0 1 2 3

6.1 Learners are allowed to choose the learning activities that they

want to use.

0 1 2 3

6.2 Learners have their control over their own pace. 0 1 2 3 6

Freedom 6.3 Learners are able to learn the course independently outside the

classroom.

0 1 2 3

7.1 Learners have opportunities to interact with the courseware. 0 1 2 3 7

Interactivity 7.2 Learners have opportunities to interact with other learners. 0 1 2 3

8.1 The course offers an entry requirement verification device to see

if the learner meets the entry conditions.

0 1 2 3

8.2 The course is able to help arouse learners’ awareness of their

own learning progress/status.

0 1 2 3

8.3 Learners are able to find where they have been (i.e. a learning

profile is offered) and where they are heading during the course.

0 1 2 3

8

Learners’

profile

8.4 The learning profile is easy to access. 0 1 2 3

II. Technical Criteria

1

Instructions

1.1 The instructions on how to use the courseware are easy for

learners to follow.

0 1 2 3

2.1 The interfaces are easy for learners to use. 0 1 2 3 2

Interfaces 2.2 The icons/symbols are intuitive and easy to understand. 0 1 2 3

3

Layout

3.1 The overall presentation (colors, font, spacing, etc.) is able to

interest learners.

0 1 2 3

4.1 It is easy to install the courseware. 0 1 2 3 4

Accessibility 4.2 It is easy to access the course to be learned. 0 1 2 3

5

Support

5.1 The technical support offered (e.g., FAQ, online support) is

sufficient.

0 1 2 3

51

4.2.1 Features of the Learner-centered Evaluation Checklist

Several features distinguish the current checklist from others. First, this checklist

focuses on learner-centeredness, and all evaluation criteria used are related to the principles,

premises, and characteristics of learner-centeredness discussed in section 2.1. For instance,

item I, 3.2 “The content is related to learners’ life experience”─adapted from Odell (1986),

Broadbent (2002), and Su (2003)─is connected to Premise Two of the APA’s Learner-

centered Model regarding learners’ exposure to interaction with others (see Table 2.2); item I,

2.1 “The learning activities are able to arouse learners’ interests in participation”─adapted

from Cunningham (1995) and Elissavet and Economides (2003)─conforms to Premise Four

of the Learner-centered Model concerning learners’ uniqueness. Table 4.3 lists examples of

the evaluation criteria echoing the premises of learner-centeredness (see Appendix III for all

the evaluation criteria matching with the Premises of the learner-centered Model). As a matter

of fact, a criterion might be corresponding to more than one premise of learner-centeredness.

52

Table 4.3

Examples of the evaluation criteria echoing the premises of the APA’s learner-centered model

Premises of Learner-centeredness Evaluation Criteria in the Checklist

1. Learners are distinct and unique, and this

distinctiveness and uniqueness must be taken into

account.

I, 2.5 There is a variety of learning activities for

learners with multiple intelligences (e.g.,

interpersonal, mathematics… .).

2. Learners’ uniqueness includes their emotional

states, learning rates, learning styles, abilities,

talents, and needs.

I, 3.5 The content accommodates learners with

various learning styles.

I, 4.1 The course offers skill-training that is able to

meet learners’ needs.

3. Learning is a constructive process, so, what is

being learned should be relevant and meaningful to

the learner and be based on the learner’s prior

knowledge and experience.

I, 8.2 The course is able to help arouse learners’

awareness of their own learning progress/status.

I, 8.3 Learners are able to find where they have been

(i.e. a learning profile is offered) and where they are

heading during the course.

4. Learning occurs best in a positive environment in

which the learner has positive interpersonal

interactions and feels appreciated and validated.

I, 7.2 Learners have opportunities to interact with

other learners.

5. Learning is a natural process; learners are

naturally curious and interested in learning about

their world. Learners must be involved in an

educational decision-making process, and

individuals’ differences should be respected and

accounted for in the learning process.

I, 3.8 The content enhances learners’ cognitive

development.

I, 6.1 Learners are allowed to choose the learning

activities that they want to use.

Note: References to the evaluation criteria are provided; for example, “I, 1.1” refers to item 1.1 in the

pedagogical category and “II, 1.1” refers to item 1.1 in the technical category.

In addition, the evaluation checklist is category-oriented. All of the criteria were

divided into two main categories— pedagogical or technical aspects— along with 13

subcategories. Such categorization offered evaluators an advanced organizer of the areas

being examined. Due to the fact that most current evaluation checklists for CALL tools pay

more attention to technical issues, it was vital to incorporate pedagogical facets into the

evaluation checklist. Within the primary pedagogical category were eight subcategories:

53

learning objectives, learning activities, content, skills, feedback, freedom, interactivity, and

learners’ profile. Furthermore, certain technical issues relevant to users’ needs were included

in the technical category, which contained five subcategories: instructions, interfaces, layout,

accessibility, and support. Each subcategory included from one to eight evaluation criteria

(see Table 4.2). Definitions of these subcategories are further provided in section 4.2.2.

Finally, a 4-point Likert scale was utilized ranging from 0 (very unlikely) to 3 (very

likely). In addition to indicating the degree to which the evaluators agreed with each criterion

description, evaluators could leave comments about their decisions in the comment column.

Open-ended comments offer insight into evaluators’ thinking in the process of evaluating and

provide further explanations of their judgments.

4.2.2 Definitions of the Subcategories

Thirteen subcategories derived from the review of studies were considered crucial in

evaluating the learner-centeredness of online CALL courseware. Table 4.4 lists the definitions

of each subcategory and its sample evaluation criteria.

54

Table 4.4

Definitions of subcategories and sample evaluation items from the evaluation checklist

I. Pedagogical Criteria Subcategory Definition Sample Evaluation Item

1. Learning objectives

Course objectives to be achieved by learners. 1.1 The information on the objectives is sufficient for learners.

2. Learning activities

Course activities provided to learners for use with the courseware.

2.4 There is a variety of learning activities for learners with various learning styles.

3. Content Content presented by the course, including reading texts and dialogues.

3.1 The content matches learners’ proficiency level.

4. Skills Skills the courseware trains learners to develop through learning activities and content.

4.1 The course offers skill-training that is able to meet learners’ needs.

5. Feedback The courseware’s feedback system/mechanism, such as correction or grading learners’ performance.

5.2 The course offers feedback that is able to motivate learners.

6. Freedom The choices and control level allowing learners to choose what to do/ not to do while using the courseware.

6.2 Learners have their control over their own pace.

7. Interactivity The interaction mechanism and activities enabling learners to interact with one another or with the courseware.

7.2 Learners have opportunities to interact with other learners.

8. Learners’ profile

The profile of a learner in using the courseware, including the learner’s background information, completed lessons, grades/performance of each lesson, etc.

8.3 Learners are able to find where they have been (i.e., a learning profile is offered) and where they are heading during the course.

II. Technical Criteria Subcategory Definition Sample Evaluation Item

1. Instructions The instructions presenting either in general or in specific lesson/activity to guide users.

1.1 The instructions on how to use the courseware are easy for learners to follow.

2. Interfaces The overall interfaces of the courseware, such as the icons and symbols.

2.2 The icons/symbols are intuitive and easy to understand.

3. Layout The overall physical presentation of the courseware, such as colors, word fonts, and spacing.

3.1 The overall presentation (colors, font, spacing, etc) is able to interest learners.

4. Accessibility The easiness and convenience of the course when used.

4.1 It is easy to install the courseware.

5. Support The technical support from the courseware company.

5.1 The technical support (e.g., FAQ, online support) is sufficient.

55

4.2.3 Summary of Stage I

As mentioned in Chapter Two, the need exists for a new means for evaluating CALL

tools─a new genre of learning materials. Recent literature has indicated the lack of the

inclusion of instructional concerns in most of the existing evaluation checklists. Thus, the

current study attempted to construct an evaluation checklist with a specific focus on learner-

centeredness. Of the 39 evaluation criteria, 32 were pedagogical criteria and 7 technical ones,

indicating two relevant points. First, the checklist emphasizes pedagogical aspects more than

technical issues. Second, the research results from Stage I revealed that most criteria— mainly

technical ones— that current evaluation checklists have adopted are not actually related to the

concept of learner-centeredness. For example, one criterion said, “The hypermedia

courseware includes information regarding how often is updated” and “The hypermedia

courseware includes information regarding its latest update” (Elissavet & Economides, 2003).

Such evaluation questions are not inadequate; however, they are irrelevant to learner-

centeredness.

Therefore, the 39 evaluation criteria in the evaluation checklist address the first

research question of this study. It is worth noting that the constructed evaluation checklist

differs from most existing evaluation checklists for CALL materials in that this checklist has

a clear purpose: attempting to raise English teachers’ awareness of learners/users and their

concerns.

4.3 Stage II: Application of the Evaluation Checklist: Evaluation of MyET

During Stage II, the learner-centered evaluation checklist was used to evaluate MyET.

Generally speaking, the evaluators held a positive attitude toward MyET. The student-

evaluators expressed higher appreciation of the courseware (mean score of 2.23 out of 3) than

the teacher-evaluators (mean score of 1.66 out of 3). This section discusses these findings

along with teacher-evaluators’ and student-evaluators’ opinions of MyET. Although initially

56

meant to be an example evaluation, this stage generated much data, and analyzing this

information provided two concrete benefits. First, the evaluation results showed the

usefulness of the checklist in some way; second, the results demonstrated MyET’s degree of

learner-centeredness, at least in an initial way.

4.3.1 Teacher-evaluators’ Evaluation Results

The total of the three evaluators’ responses for each evaluation criterion ranged from

8 (highest) to 1 (lowest), with a mean score of 1.66 out of 3. A total score above 7 is

considered high. A score of 6 might involve the scores 3, 2, and 1, in which 1 is regarded as a

low score; 7 might include 3, 3, and 1. However, a score of 8 represents 3, 3, and 2, in which

all three teacher-evaluators gave high scores. Based on the results, the following discussion

will explore the strengths and weaknesses of MyET as well as non-scored evaluation criteria.

(Note: References to the checklist evaluation criteria are provided; for example, “I, 1.1”

refers to item 1.1 in the pedagogical category and “II, 1.1” refers to item 1.1 in the technical

category.) Meanwhile, the findings obtained from the first interview during Stage II (see

section 3.3.3) regarding the teaching background of the teacher-evaluators and their concepts

about learner-centeredness will be addressed in section 4.4.

Strengths of the Courseware

Looking at the results by evaluation criterion, 6 of the 39 evaluation criteria received

the highest score (8) while another 6 attained the second highest score (7).

Table 4.5 displays the high-scoring evaluation criteria, highlighting the strengths of

the courseware examined. All of the six evaluation criteria were addressed by all three

teacher-evaluators.

57

Table 4.5

The high-scoring evaluation criteria from the teacher-evaluators’ evaluation results

Item No. Item Description Total*

I, 2.3 The guidance and models of learning activities are clear for learners. 8

I, 6.1 Learners are allowed to choose the learning activities that they want to use. 8

I, 6.2 Learners have their control over their own pace. 8

I, 6.3 Learners are able to learn the course independently outside the classroom. 8

I, 8.2 The course is able to help arouse learners’ awareness of their own learning

progress/status.

8

II, 2.2 The icons/symbols are intuitive and easy to understand. 8

Note: * Here the total means the total of the three evaluators’ responses.

According to the teachers’ evaluations, MyET’s Studio Classroom guidance and

learning activities models are clear (I, 2.3). In addition, all three criteria related to freedom (I,

6.1, 6.2, and 6.3) received the highest score, indicating consensus in regards to the

courseware’s sufficient user control of the learning process. The teacher-evaluators also

determined that MyET is able to help learners become aware of their learning status and

progress (I, 8.2). Finally, they found the icons/symbols to be intuitive and easy to understand

(II, 2.2).

Examining the findings category by category, freedom, interfaces, instructions, and

layout earned the highest averages. Interestingly, three of the five technical subcategories

received high marks, while only one of the eight pedagogical subcategories did so, which

seems to indicate that MyET’s technical features outperformed its pedagogical facets. This

finding echoes a phenomenon discussed in many current studies: most existing commercial

courseware emphasizes technical characteristics more than pedagogical considerations (Neri,

Cucchiarini, Strik, & Boves, 2002). Another possible reason for higher number of technical

criteria receiving high marks is that technical criteria may be easier to judge since they are

relatively visible, while pedagogical ones are more abstract.

58

Weaknesses of the Courseware

Looking at the results by each evaluation criterion, several evaluation criteria scored

relatively low, with only one or two points. However, a few of these low-scoring items were

not scored by all three evaluators, which in turn lowered the overall scores for the items. The

teacher-evaluators’ explanations for this (identified during interviews) will be addressed in

the next section. Two of the low-scoring items were answered by all three teacher-evaluators

and, as such, are indeed low-scoring evaluation items that reveal clear weaknesses of MyET.

Table 4.6 shows the low-scoring evaluation items answered by all teacher-evaluators. Table

4.7 lists the low-scoring evaluation items to which not all teacher-evaluators responded.

These imply possible weaknesses of MyET.

The results indicate that the evaluators did not agree that the courseware serves

certain learner-centered features— namely: 1) the content helps learners build up strategic

ability (I, 3.6) and 2) there is a variety of learning activities for learners with multiple

intelligences, e.g., interpersonal, mathematics… (I, 2.5). Other low-scoring items included, for

example, the content accommodates learners with various learning styles (I, 3.5), the course

offers feedback that can be saved and accessed easily so that learners may realize what

difficulties they have in the learning process (I, 5.4), the learning profile is easy to access (I,

8.4), and the course offers an entry requirement verification device to see if the learner meets

the entry conditions (I, 8.1).

59

Table 4.6

The low-scoring evaluation items from the teacher-evaluators’ evaluation results (responded to

by all evaluators)

Item No. Item Description Total*

I, 3.6 The content helps learners build up strategic ability. 2

I, 2.5 There is a variety of learning activities for learners with multiple intelligences

(e.g., interpersonal, mathematics… ).

3

Note: * Here the total means the total of the three evaluators’ responses

Table 4.7

The low-scoring evaluation items from the teacher-evaluators’ evaluation results (responded to

by only one or two evaluators)

Item No. Item Description Total*

I, 3.5 The content accommodates learners with various learning styles. 1

I, 5.4 The course offers feedback that can be saved and accessed easily so that

learners may realize what difficulties they have in the learning process.

1

I, 8.4 The learning profile is easy to access. 1

I, 8.1 The course offers an entry requirement verification device to see if the learner

meets the entry conditions.

2

II, 5.1 The technical support offered (e.g., FAQ, online support) is sufficient. 2

I, 4.1 The course offers skill-training that is able to meet learners’ needs. 3

I, 5.2 The course offers feedback that is able to motivate learners. 3

Note: * Here the total means the total of the three evaluators’ responses, reflecting one or two

evaluators’ non-responses.

60

When looking at the results by category, support received the lowest score (2) while

skills scored only 3. These figures may indicate that the courseware does not provide

sufficient technical support— e.g., FAQ and online support (II, 5.1)— and the skill-training

offered by the courseware may not be able to meet learners’ needs (I, 4.1). In fact, although

MyET increased its various learning courses, including reading and writing programs, it

continues to emphasize speaking skills, particular pronunciation; its learning activities are

based primarily on the Automatic Speech Analysis System (ASAS), which focuses on

improving learners’ pronunciation skills. MyET’s Chinese commercial refers to the

courseware as “My Oral-training English Tutor” (“我的口說英語家教” in Chinese). Thus, it is

not surprising that the teacher-evaluators did not think this courseware offered skill-training

that was able to meet learners’ various needs, such as reading and writing.

Non-scored Evaluation Criteria

Several evaluation items that received quite low scores do not necessarily point to

shortcomings of the courseware since they were not responded to by every teacher-evaluator.

Teacher-evaluator A did not score 14 evaluation items, and Teacher-evaluator C did not

respond to 7 items; Teacher-evaluator B answered all 39 evaluation questions. Teacher-

evaluator A wrote “unknown” or “not applicable” in several items, and Teacher-evaluator C

made question marks when she did not mark a score. Non-scored answers do not equate to

zero in this sense.

These evaluators were not able to score the items for several reasons. First, the

evaluation description was not applicable to the evaluand. For example, Teacher-evaluator A

did not think the feedback offered by the courseware could be saved and accessed easily (I,

5.4), so he marked “not applicable” in the comment column instead of checking the score 0,

indicating “very unlikely”. Second, when the evaluators perceived that a criterion was only

partially met, they tended not to score the criterion, indicating “unknown” in the comment

61

column. For instance, for the evaluation item “the content matches learners’ English

proficiency level,” Teacher-evaluator A’s idea was partially yes as he thought the English

proficiency level of his students varied, so he decided not to score it, writing “some yes, some

no” in the comment column instead. When asked the reasons for the uncertainty, Teacher-

evaluator A replied that he had kept his students, who had been using MyET, in mind while

doing the evaluation. With a specific target group of students in mind, he found it difficult to

make judgments unless he had already seen evidence of the effectiveness within the students’

skills. In other words, he did not try to predict the effectiveness of the courseware, but instead

adopted a retrospective view to conduct the evaluation (see section 2.3.1 of the literature

review). Therefore, he could not score quite a few of the evaluation questions.

Another reason for the lack of a score might be unfamiliarity with the courseware.

Although all of the teacher-evaluators had used MyET, they might not have had the chance to

try all of its functions and interfaces; thus, they could not respond to the checklist items. For

example, Teacher-evaluator C had no idea whether MyET offered an entry requirement

verification device or a learner’s file; she indicated this with a question mark in the comment

column and explained this during an interview.

Finally, the evaluators did not give a score because they interpreted the criteria

descriptions differently. Teacher-evaluator C, for instance, responded to item I, 7.1 “Learners

have opportunities to interact with the courseware” by writing her comment that “Isn’t it

interaction to use the courseware?” (“使用本身不就是互動了嗎?” in Chinese.) However, the

interactivity here refers to the interactive opportunities the courseware offers. The way that

the teacher-evaluator interpreted the criteria influenced her judgment; thus she could not

respond to this evaluation question.

These concerns led to a lack of responses, creating complexity in analyzing and

interpreting the research data. Contextual variables affect the use of a checklist. In fact,

62

unable-to-answer or not-sure phenomena are common if the checklist is not tailor-made for a

specific evaluation (Bradin, 1999; Le & Le, 1997). As the aim of the current study is to

explore the learner-centeredness in online CALL tools and develop a learner-centeredness-

based checklist for online courseware evaluation, Stage II served as an application of the

checklist developed, giving all evaluators the actual experience of using the checklist for the

subsequent meta-evaluation of its effectiveness. It should be reiterated that the statistical

results of the MyET evaluation in this study were not intended to be the main findings, nor to

be generalized.

Controversial Issues

All three teacher-evaluators agreed to a great extent on their evaluations. However,

disagreements occurred among a few evaluation criteria (i.e., I, 1.1; I, 3.7; I, 3.8; I, 5.5; I, 8.3;

II, 4.2). For example, two evaluators thought the courseware offered sufficient information on

learning objectives, while the third disagreed (I, 1.1). In addition, when responding to

whether the courseware provides an opportunity for learners to give personal feedback on

their learning experience (I, 5.5) and whether the courseware provides learners a learning

profile (I, 8.3), Teacher-evaluator A held a positive attitude whereas Teacher-evaluator B

disagreed; Teacher-evaluator C did not answer these two questions at all. Still, the findings

indicate that most of the time these evaluators shared mutual points of view on MyET, which

might imply the effectiveness of the checklist, which will be discussed in section 4.4.

4.3.2 Student-evaluators’ Evaluation Results

In addition to the 39 evaluation criteria, these student-evaluators were asked to

comment on MyET in general─“In general, this courseware is helpful for learners; namely, it

is able to meet learners’ needs or enhance English learning” (see item No. 40 in Appendix II).

Ten out of the total 22 student-evaluators thought that MyET was likely helpful for their

learning, and another 10 agreed that MyET was very likely helpful.

63

Generally speaking, the student-evaluators held a positive view of MyET. With 22

student-evaluators involved in this evaluation, the total scores should range from the two

extremes— 0 to 66— making any score below 33 a negative perception. Based on the

statistical results from student-evaluators’ evaluations, the total scores for each evaluation

criterion range from 55 (highest) to 43 (lowest), with a mean score of 2.23 out of 3. The fact

that the lowest score was 43 indicates the students’ positive attitude. (Note: item No. 40 was

not included in the calculation in order to be able to compare the student-evaluators’ mean

score with the mean score calculated from the teacher-evaluators’ evaluation results.)

Strengths of the Courseware

Table 4.8 highlights examples of the high-scoring evaluation items, which are

considered the strengths of the evaluand. For instance, the student-evaluators thought the

content of MyET’s Let’s Talk in English was related to their life experiences. Looking at the

results category by category, freedom received the highest average (52)— similar to the

teacher-evaluators’ evaluation— and learning objectives the second highest (51). This result

shows that most of the student-evaluators were satisfied with the freedom that MyET

provides learners.

64

Table 4.8

The high-scoring evaluation items from the student-evaluators’ evaluation results

Item No. Item Description Total*

I, 3.2 The content is related to learners’ life experience. 55

I, 1.1 The information on the objectives is sufficient for learners. 53

I, 1.2 The objectives of the course are associated with learners’ learning development. 53

I, 6.1 Learners are allowed to choose the learning activities that they want to use. 53

I, 6.3 Learners are able to learn the course independently outside the classroom. 52

II, 2.1 The interfaces are easy for learners to use. 52

Note: * Here the total means the total of the 22 student-evaluators’ responses

Weaknesses of the Courseware

No evaluation items scored below 33, which seems to indicate that there were no low-

scoring evaluation items from the student-evaluators’ evaluation results. Table 4.9 lists a few

examples of the lowest scores according to the student-evaluators. These results may not

actually indicate deficiencies in MyET, but might point to possible weaknesses. For example,

MyET does not provide a wide variety of activities for users with different multiple

intelligences (I, 2.5). Moreover, among the 13 subcategories, support had the lowest average

(45). This finding not only reveals a difference between the two groups of evaluators, but also

exposes a phenomenon that, as the primary user, students expect more sufficient support (e.g.,

online support) when using the courseware. Nonetheless, although these criteria received

lower scores, when divided by the total number of student-evaluators, the average (2)

indicates that the students agreed that MyET likely conforms to the criteria. On the other

hand, the subcategory with the highest average is freedom (with the average 52). Interestingly,

the teacher-evaluators’ evaluation results show the same finding.

65

Table 4.9

The low-scoring evaluation items from the student -evaluators’ evaluation results

Item No. Item Description Total*

I, 2.5 There is a variety of learning activities for learners with multiple intelligences

(e.g., interpersonal, mathematics… ).

43

I, 7.2 Learners have opportunities to interact with other learners. 44

II, 5.1 The technical support offered (e.g., FAQ, online support) is sufficient. 45

I, 3.4 The content is culturally appropriate for learners. 46

I, 3.6 The content helps learners build up strategic ability. 46

I, 5.2 The course offers feedback that is able to motivate learners. 46

I, 5.5 The course offers an opportunity for learners to give personal feedback on their

learning experience.

46

II, 4.1 It is easy to install the courseware. 46

Note: * Here the total means the total of the 22 student-evaluators’ responses

Non-scored Evaluation Criteria

Only one student among the 22 left items unanswered (a total of three unanswered

questions); the others answered all the evaluation questions. It seems that students tended to

answer all the questions they encountered.

4.3.3 Caveats about Stage II

As mentioned earlier, due to the small sample sizes, the statistical results presented

here are not aimed to be generalized and to decide the value of the evaluand; instead, the

purpose of this stage of the study is to give the participants experience in using the checklist

as well as to gather insight as to what actual learners think and need when using courseware

to learn English, since all of the participants had used the evaluand. Another interesting

finding worth mentioning is that among both groups of evaluators, few evaluators gave any

zeros when responding to the evaluation checklist. Teacher-evaluator A gave one zero, and

Teacher-evaluator C gave two; two student-evaluators gave one zero each. Whether this

phenomenon shows the courtesy of Taiwanese when they make judgments or suggests the

66

value of MyET is an issue for further study. Finally, all data were collected in January 2007

based on these evaluators’ previous experiences using the courseware. MyET may have

subsequently issued new editions.

4.3.4 Summary of Stage II

In sum, the total 25 evaluators regarded MyET in a positive light. Although the

evaluators examined two different MyET’s courses, Studio Classroom Standard Course and

Let’s Talk in English, both courses used the same platform and interfaces. Thus, the

evaluation results still present the advantages and disadvantages of the courseware itself.

Teacher-evaluators tended to examine the courseware from a thorough and critical point of

view, resulting in a lower mean (1.66) than the student-evaluators’ (2.23). Contextual issues,

such as professional background (teachers versus students) and evaluation processing time

frame (a week versus an hour), could have caused such differences. Moreover, any evaluation

process embraces subjective judgments; for example, the interpretation of “likely” in “the

course/courseware likely conforms to the description” might be different for different

evaluators.

The results and discussions of Stage II aimed to display the results from the

evaluations of MyET and are expected to provide insights for English teachers, institution

administrators (stakeholders) who may need to make decisions as to what courseware to use

in a language lab or a self- learning center, or even online courseware developers.

4.4 Stage III: Meta-evaluation of the Learner-centered Evaluation Checklist

To answer the second research question, Stage III gathered feedback on the evaluation

checklist through interviews and open-ended questions. The first interview with each teacher-

evaluator during the previous stage (see section 3.3.3) focused on their teaching background

and concepts about learner-centeredness. Two of the teachers had been teaching for

approximately ten years, while the other one for six years─one year in a college setting. The

67

second interview during Stage III, aiming to gather their feedback in regards to the evaluation

checklist, was conducted after they had used the checklist to evaluate MyET. Interview

question four─“Have you used any other evaluation checklist for ESL/EFL materials?”─

aimed to determine these teachers’ experiences using an evaluation checklist for ESL/EFL

materials. Two of them had used several such checklists previously; however, these two

teachers mentioned that the evaluation checklists they had used were for textbooks, not for

courseware. One teacher had never used such an evaluation tool.

Feedback on the checklist from teacher-evaluators and student-evaluators is discussed

in the following two sections.

4.4.1 Teacher-evaluators’ Feedback

Seven primary interview questions were asked (see Table 3.2), along with a few

follow-up questions when necessary during each interview (see Appendix VI for the

interview transcripts with English translations). References to the transcripts are marked in

parentheses, for example, “(A, 100 & 105)”, referring to Teacher-evaluator A’s comments

presented in line number 100─responses in Chinese─and line number 105─English

translation. Categorizing the responses to the interview questions, five aspects are discussed

as follows.

Overall Opinions toward the Evaluation Checklist

Based on the findings generated from interview questions one, six, and seven,

generally speaking, the three teacher-evaluators thought the checklist would be helpful for

teachers in evaluating courseware’s learner-centeredness ( A, 276-277, 280 & 281-282, 286;

B, 343 & 345; C, 475 & 478). All agreed that the developed evaluation checklist focused

teachers’ attention on learners. One felt most teachers were influenced by technical issues

when evaluating software; what attracted teachers might simply be the beauty of the

software’s design and multimedia. When asked whether the checklist would be helpful for

68

English teachers in evaluating and selecting courseware, Teacher-evaluator A replied, “Sure!

Why not? I think this checklist would help teachers see what they should pay attention to

when it comes to selecting courseware” (A, 276-277 & 281-282). He also agreed that this

checklist would focus teachers’ awareness on the need to include language

acquisition/learning issues during the evaluation (A, 279-280 & 284-286). Another evaluator

found the checklist “not bad” (B, 291 & 292). “If I have to grade it, I’ll give it a least an 85.

The questionnaire is not bad,” answered Teacher-evaluator B (B, 296 & 297). He thought the

checklist was helpful for teachers in that it offered quite a few items that most teachers might

overlook (B, 343-344 & 345-346). When asked whether the checklist was helpful for teachers

in evaluating courseware, the third teacher-evaluator replied, “I think so. With this checklist

you could be more objective. And it reminds you of some things that you’ve never thought

of ” (C, 475-476 & 478-479). She also mentioned that the checklist was able to help avoid

selecting software “by first impressions” since it provided different aspects to consider (C,

476-477 & 479-480).

It might be argued that the three teacher-evaluators gave positive comments out of

politeness. Nonetheless, since all indicated a similar attitude toward the checklist to a great

extent, it can be concluded that the learner-centered evaluation checklist serves as a good

evaluation tool for online courseware. In addition, one way to validate findings is to look for

negative evidence (Miles & Huberman, 1994). These teacher-evaluators did actually point out

the evaluation checklist’s deficiencies (see Weaknesses of the Evaluation Checklist below),

revealing that their positive responses in general were not simply made out of courtesy.

Strengths of the Evaluation Checklist

In response to interview question five, the three teacher-evaluators identified several

strengths of the checklist. First, the checklist’s understandable categorization enhances

understanding of the focus. “The most helpful part of the checklist is the categories,” one

69

teacher-evaluator said (A, 237 & 240). He further mentioned, “I should focus on my student

when I try to help him/her to learn. But do I notice some relevant points? I like the

questionnaire because it reminds me of some points” (A, 238-240 & 241-244). Second, it has

sufficient coverage of crucial evaluation facets for online CALL tools. “At least it gives

teachers quite a few criteria for evaluation; teachers may not think of so many themselves” (B,

343-344 & 345-346). According to the third teacher-evaluator, “it covers quite a lot of aspects,

allowing teachers to evaluate courseware from various aspects” (C, 448-449 & 450-452).

Finally, the checklist involves certain pedagogical concerns that might be easily ignored by a

teacher when evaluating courseware; for example, one teacher-evaluator mentioned that her

attention was drawn to the learner— her students— when she saw the evaluation criterion

concerning learning objectives. This teacher-evaluator started to think about the reason for

having her students use the courseware and whether students could accomplish something at

the end. This particular evaluation item reminded this teacher-evaluator to take her students’

uniqueness into account (C, 437 & 440-442).

Weaknesses of the Evaluation Checklist

On the other hand, these teacher-evaluators pointed out several weaknesses or defects

in the checklist when replying to interview question five. First, a few evaluation items were

not easy to answer for two of the evaluators. One possible explanation is that one teacher-

evaluator always kept his students in mind during the evaluation, adopting a retrospective

perspective in evaluating this courseware— in other words, if he could not decide his

students’ potential performance, he chose “unknown.” “The reason I can’t answer this

question is because MyET focuses on pronunciation, and we haven’t finished the course yet.

So I can’t answer it; I wrote ‘unknown’” (A, 183-185 & 189-191). As mentioned earlier, it is

a common phenomenon that in an evaluation process certain items are not easy to respond to

since the evaluation context varies. Bradin (1999) suggests that it is not necessary to finish a

70

checklist from the beginning to the end; wherever there are primary items that do not fit,

there is no need to continue the evaluation. Another possibility is the degree of familiarity

with this courseware. If the teacher-evaluator had not had a chance to test all functions of the

courseware, some evaluation criteria were difficult to rate. For example, Teacher-evaluator C

wrote several question marks due to the fact that she had not used or known of certain MyET

function, such as “learners’ profile.”

Second, the translation of two terms between the English and Chinese was not

consistent or clear. Two of the teacher-evaluators pointed out the deficiency. One problem

was the heading of the subcategory freedom (translated as “自由度” in Chinese). According to

Teacher-evaluator B, the Chinese translation does not interpret the real meanings of the

evaluation criteria under the subcategory, including students’ control over the learning

activities and learning pace (B, 315-318 & 323-326). Another error of translation is that

cognitive development should have been translated as “認知發展” instead of “智能發展” (B,

314-315 & 321-322; C, 384-386 & 388-390). This checklist was first developed in English,

and the subsequent mistakes occurred when it was translated into Chinese; more caution is

required in the translation process.

Suggestions for Further Revisions

With regard to further additions to and deletions from the evaluation checklist, none

of the three teacher-evaluators proposed any suggestions except that one expected to see

more items regarding SLA on an evaluation checklist for CALL tools (A, 258-259 & 261-

263). As for further suggestions for revision, two teacher-evaluators suggested that the

subcategory interactivity be defined first and include specific types of interactivity since it

might be confusing to evaluators (B, 351-359 & 361-372; C, 400-403 & 417-420). Different

types of interactivity exist, such as interaction between the user and the courseware and

interaction between/among users. Finally, one teacher-evaluator suggested separating the four

71

skills (listening, speaking, reading, and writing) under the subcategory skills since different

courseware may focus on different skills (C, 397-399 & 413-417). For example, MyET’s

focus is primarily on pronunciation skills. Revisions to the checklist based on these

suggestions are discussed in Chapter Five.

Teacher-evaluators’ Concepts of Learner-centeredness

Before using the checklist, teacher-evaluators were asked about their concepts of

learner-centeredness (see section 3.3.3 about the first interview). All three expressed similar

viewpoints. Teacher-evaluator A thought that learner-centeredness meant “wherever the

learner is, what he or she is learning must be relevant to himself or herself. It has to be

meaningful; if not, it will be that teachers teach whatever they feel like teaching and students

learn whatever they feel like learning” (A, 76-78 & 91-95). Teacher-evaluator B stated,

“Learner-centeredness means that the focus is on the students when it comes to learning or

preparing teaching activities, right? On one hand, as a teacher, you acknowledge the needs of

the students and try to satisfy them. On the other hand, students are placed in the center of

learning” (B, 128-130 & 138-141). In Teacher-evaluator C’s opinion, learner-centeredness is

“the design as well as the process of a curriculum has the learner as the focus. The teacher,

here, plays a role like an assistant” (C, 168 & 172-173). Although the teacher-evaluators’

descriptions seemed broad, they touched on the characteristics of learner-centeredness to

some degree. It is interesting that two of the teacher-evaluators commented that not all

learners understood what they needed in terms of learning, particularly those with lower

proficiency levels (A, 78-80 & 95-98; B, 130-136 & 141-148), implying that a teacher still

needed to intervene in learning. This idea reveals the reality of the English learning context in

Taiwan.

The teacher-evaluators’ thinking and attitude toward learner-centeredness might have

affected their evaluations. After using the checklist to evaluate MyET, when answering

72

whether the checklist helped focus the attention to learner-centeredness, Teacher evaluator A

replied, “It should!” since his curriculum design for his class focused on self- learning (A,

230-231 & 233-234). Teacher-evaluator C also agreed that the checklist was able to draw her

attention to learner-centeredness; she mentioned her experience of using the checklist, “You

mention a few questions concerning objectives in teaching, which leads me to think why I

would want my students to use such software and think if they can achieve the objectives.

That’s what the checklist did for me” (C, 437-439 & 440-442). As for Teacher-evaluator B, he

did not describe his own experience; however, he thought the checklist “should be a little bit”

able to arouse teachers’ awareness of learner-centeredness (B, 302 & 306). To sum up, based

on these teachers’ responses, it can be concluded that the use of the evaluation checklist

increased their awareness of learner-centeredness in evaluating courseware.

4.4.2 Student-evaluators’ Appreciation

Three open-ended questionnaire questions on the student-evaluators’ version of the

checklist were included to elicit feedback on its use (see section 3.4.3). Their responses were

coded as positive or negative, categorized and counted the number of responses (see section

3.4.4 about the categorization process). The results are shown in Table 4.10 (translated from

Chinese; see Appendix V for the original comments in Chinese).

73

Table 4.10

Frequency of student-evaluators’ feedback on the learner-centered evaluation checklist

Question 1: Generally speaking, do you find these evaluation items easy to answer? Why/Why not?

Response Frequency Percentage

Positive 4 18%

Fairly Positive 8 36%

Partially Positive 1 5%

Negative 3 14%

No Response 5 23%

Responses that do not fit the question 1 5%

Total 22 101%*

Question 2: Do you think this evaluation checklist is able to focus the attention on learners’ needs

and conditions?

Response Frequency Percentage

Positive 10 45%

Partially Positive 1 5%

Negative 1 5%

No Response 9 41%

Responses that do not fit the question 1 5%

Total 22 101%*

Question 3: In addition to these evaluation items, what concerns should be taken into account when

a teacher is evaluating and selecting suitable learning materials to meet learners’ needs?

Response Frequency Percentage

Learners’ proficiency levels 2 9%

Learners’ situations of learning and using of the courseware 3 14%

Learners’ Freedom 2 9%

Content of the courseware (interesting) 2 9%

No further comments (enough items.) 4 18%

No response 7 32%

Responses that do not fit the question 2 9%

Total 22 100%

Note:*Approximate value exceeds 100 because of rounding up

74

Overall Opinions toward the Evaluation Checklist

In general, the student-evaluators thought the questions were easy to answer since the

descriptions of the evaluation criteria were presented in a straightforward manner. They also

agreed that the evaluation checklist was helpful for focusing attention on learners’ needs and

situations when evaluating learning courseware. The following sections discuss the student-

evaluators’ positive and negative comments in regards to the checklist.

Positivity about the Evaluation Checklist

The findings indicate the student-evaluators’ positivity about the checklist as follows.

First, four of the 22 student-evaluators rated the checklist as easy to answer; another 8 found

it fairly easy to answer. Second, nearly 50 percent of the student-evaluators agreed that the

checklist was able to focus their attention on learners’ needs and conditions in evaluating

courseware. Although these students held a relatively positive attitude toward the checklist,

three student-evaluators thought it included too many questions, some of which were too long

or vague. This finding confirms what had been suspected: students prefer a short, concise

evaluation checklist, while teachers prefer a comprehensive one. This outcome proves that

the decision to modify the original six-page bilingual checklist into a two-page Chinese

checklist was sensible (see section 3.2.4).

Negativity about the Evaluation Checklist

The findings reveal the student-evaluators’ negative attitude toward the checklist as

follows. First, three student-evaluators rated the checklist as not easy to answer, citing the

vagueness of the evaluation item descriptions, experience with MyET (they had not being

using it for long), and the coverage of MyET functions (they had not tried all functions).

Second, one student-evaluator disagreed that the checklist functioned as it claimed since the

checklist did not reveal the reality of learners’ learning situations. In addition, a certain

percentage indicated “no response”; five did not give feedback on question one, and nine did

75

not on question two (see Table 4.10).

Additional Comments

When asked what concerns, in addition to the 39 criteria on the checklist, should be

taken into account in evaluating courseware for learners, several student-evaluators proposed

certain evaluation criteria to this learner-centeredness-based checklist. Two mentioned

learners’ proficiency levels and learning abilities, and another three listed learners’ learning

status and use of the courseware. One stated that the option to skip difficult questions in the

courseware is an important item, while another pointed out that the inability to install

courseware on students’ personal computers reduces learning opportunities; these two ideas

conform to the concept of freedom of the courseware— the degree to which a user can control

the process when using the learning tool. In addition, two student-evaluators expected the

courseware to be interesting and to provide more games; such responses hint at students’

needs and expectations for a more learner-centered learning tool.

Four irrelevant comments in total were made by the student-evaluators, revealing their

inability to understand and answer the questions. For example, two student-evaluators

responded “the repeating and recording part” (“跟著讀的部份” in Chinese) to the question: “In

addition to these evaluation items, what concerns should be taken into account when a

teacher is evaluating and selecting suitable learning materials to meet learners’ needs?” The

response seemed irrelevant or unclear.

4.4.3 Summary of Stage III

Based on all 25 evaluators’ feedback on the evaluation checklist, the second research

question has been answered; this checklist can indeed assist in focusing teachers’ (and

students’) attention on learners’ needs in evaluating courseware, although certain revisions

are required. The findings also highlight that a need does indeed exist for a new approach to

courseware evaluation in which learners’ needs are taken into serious consideration.

76

4.5 Summary

This chapter displays and discusses the findings of the three stages of this study as

well as answers the two research questions proposed in Chapter Two: an evaluation checklist

based on the premises of learner-centeredness was developed, and its effectiveness of

enhancing teachers’ awareness of learner-centeredness was also affirmed. The following

chapter will give conclusions.

77

CHAPTER FIVE

Conclusions

This final chapter draws conclusions in regards to this study. In this chapter, a revised

learner-centered evaluation checklist is proposed based on the suggestions and comments

from the evaluators. Besides, the contribution of this study, its limitations, and suggestions

for further research will be presented.

5.1 Revised Learner-centered Evaluation Checklist

Based on the teacher-evaluators’ and student-evaluators’ suggestions and comments,

several revisions were made to the evaluation checklist (see Table 5.1 for all the revised items

and Appendix VII for the revised checklist in Chinese and English). First, in the pedagogical

part, the heading skill in Category Four has been changed to skill-training; in addition, the

four skills (listening, speaking, reading, and writing) were separated into their own items,

shown as item I, 4.1, I, 4.2, I, 4.3, and I, 4.4 (see Suggestions for Further Revisions in section

4.4.1). Second, in Category Six, the heading was changed from freedom to learner control

(“學習者的自主性” in Chinese), which may correspond more with its items (see Weaknesses of

the Evaluation Checklist in section 4.4.1). Learner control/user control allows learners to take

their own path through the courseware. Hoven (1999) argues that offering a certain degree of

control to learners is desirable in order to promote learner autonomy. Little (1991) also states

that the lack of choice and freedom may lead to the problem that learners are reluctant to take

responsibility for their own learning. Third, the translation error in item I, 3.8 has been

modified from the erroneous “智能發展” to the corrected “認知發展”, which is consistent with

the English term cognitive development. Finally, regarding Category Seven, two evaluators

suggested defining the term interactivity (see Suggestions for Further Revisions in section

4.4.1); however, item I, 7.1 and I, 7.2 have actually indicated the differences between “the

interaction between the learner and the courseware” and “the interaction between/among the

78

learner and other learners.” To avoid any misunderstanding, the revised checklist includes

examples in the descriptions of these items (see Table 4.2 for comparison).

79

Table 5.1

Revised learner-centered evaluation checklist for online CALL courseware

I. Pedagogical Criteria 1.1 The information on the objectives is sufficient for learners. 0 1 2 3

1.2 The objectives of the course are associated with learners’ learning development.

0 1 2 3

1.3 The objectives are achievable for learners. 0 1 2 3

1 Learning

Objectives

1.4 The objectives reflect what learners need to know. 0 1 2 3 2.1 The learning activities are able to arouse learners’ interests in participation.

0 1 2 3

2.2 The instructions of the learning activities are clear for learners. 0 1 2 3

2.3 The guidance and models of learning activities are clear for learners.

0 1 2 3

2.4 There is a variety of learning activities for learners with varied learning styles.

0 1 2 3

2 Learning Activities

2.5 There is a variety of learning activities for learners with multiple intelligences (e.g., interpersonal, mathematics… .).

0 1 2 3

3.1 The content matches learners’ English proficiency level. 0 1 2 3

3.2 The content is related to learners’ life experience. 0 1 2 3 3.3 The content covers a variety of topics that are able to attain learners’ interests.

0 1 2 3

3.4 The content is culturally appropriate for learners. 0 1 2 3 3.5 The content accommodates learners with various learning styles. 0 1 2 3

3.6 The content helps learners build up strategic ability. 0 1 2 3 3.7 The content stimulates recall of prior learning. 0 1 2 3

3 Content

3.8 The content enhances learners’ cognitive development. 0 1 2 3

4.1 The content offers listening training that is able to meet learners’ needs.

0 1 2 3

4.2 The content offers speaking training that is able to meet learners’ needs.

0 1 2 3

4.3 The content offers reading training that is able to meet learners’ needs.

0 1 2 3

4

Skill-training

4.4 The content offers writing training that is able to meet learners’ needs.

0 1 2 3

5.1 The course offers immediate and proper feedback on learners’ performance and/or responses.

0 1 2 3

5.2 The course offers feedback that is able to motivate learners. 0 1 2 3

5.3 The course offers meaningful and helpful error handling which is able to help learners improve.

0 1 2 3

5.4 The course offers feedback that can be saved and accessed easily so that learners may realize what difficulties they have in the learning process.

0 1 2 3

5 Feedback

5.5 The course offers an opportunity for learners to give personal feedback on their learning experience.

0 1 2 3

1

3

80

6.1 Learners are allowed to choose the learning activities that they want to use.

0 1 2 3

6.2 Learners have their control over their own pace. 0 1 2 3

6

Learner Control 6.3 Learners are able to learn the course independently outside the classroom.

0 1 2 3

7.1 Learners have opportunities to interact with the courseware (e.g., role-play activities for dialog practice).

0 1 2 3 7

Interactivity 7.2 Learners have opportunities to interact with other learners (e.g., online discussion forum).

0 1 2 3

8.1 The course offers an entry requirement verification device to see if the learner meets the entry conditions.

0 1 2 3

8.2 The course is able to help arouse learners’ awareness of their own learning progress/status.

0 1 2 3

8.3 Learners are able to find where they have been (i.e. a learning profile is offered) and where they are heading during the course.

0 1 2 3

8 Learners’

profile

8.4 The learning profile is easy to access. 0 1 2 3

II. Technical Criteria 1

Instructions 1.1 The instructions on how to use the courseware are easy for learners to follow.

0 1 2 3

2.1 The interfaces are easy for learners to use. 0 1 2 3 2 Interfaces 2.2 The icons/symbols are intuitive and easy to understand. 0 1 2 3

3 Layout

3.1 The overall presentation (colors, font, spacing, etc.) is able to interest learners.

0 1 2 3

4.1 It is easy to install the courseware. 0 1 2 3 4 Accessibility 4.2 It is easy to access the course to be learned. 0 1 2 3

5 Support

5.1 The technical support offered (e.g., FAQ, online support) is sufficient.

0 1 2 3

5.2 Contributions of the Study

5.2.1 Contribution to Theory

As mentioned in Chapter Two, a gap exists between pedagogical theories and CALL

courseware design. Most courseware developers emphasize the technical features instead of

taking instructional theories into consideration. As a result, when language teachers evaluate

and select a CALL tool, the beauty of the interface as well as the integration of audio and

video features might mislead teachers to believing that only colorful Flash files and cool

sound tracks can attract and motivate learners. Current literature on CALL evaluations shows

2

4

4

81

that, although this gap has been noticed among a great number of evaluation approaches and

tools, few studies have attempted to involve specific learning principles— in this case,

learner-centeredness— in a CALL-tool evaluation checklist. The concept of learner-

centeredness in second/foreign language teaching is not new, yet CALL researchers have only

recently started paying more attention to issues of user- friendliness and promoting user-

centered designs. This study sought to fill the research gap and contribute to CALL and

evaluation research by providing an example of the application of a key concept in teaching

English as second or other language (TESOL) to the evaluation of CALL courseware.

5.2.2 Contribution to Practice

Implications for English Teachers and Stakeholders of Institutions

Due to the characteristics of evaluation, which involves a judgment along with

evidence about the worth of the evaluand, evaluation is intended to “have immediate utility

for policy shaping and is expected to be influential in short-term decision making” (Rea-

Dickins & Germaine, 1998, p. 11). The developed learner-centered evaluation checklist

supplements existing evaluation checklists for English learning materials. The learner-

centered-focused checklist provides a series of criteria to follow for language teachers and

stakeholders who make decisions about the learning materials to be used in a classroom or a

self- learning center. This evaluation checklist should also serve to remind teachers and

stakeholders of learners’ individuality and uniqueness. When they evaluate and select

courseware, learners’ needs and expectations should be taken into account. Learning

efficiency will increase a great deal once the learner is placed in the right position— at the

core of the learning process.

Implications for Courseware Developers and Material Providers

It is believed that online courseware has its advantages in assisting language learning.

Numerous English material providers are working with courseware developers in an attempt

82

to provide their learning materials on the Internet. However, Susser and Robb (2004) point

out that online courseware does not mean moving the content in print to a web-based

environment; instead, an online learning tool should follow certain principles of learning and

pay attention to interactivity and authenticity, which can be some of the advantages of a

CALL tool. Applying the checklist to MyET as an example, the study found that the learning

activities need improvement; they should be more interactive rather than focused on

repetition and drill. The evaluand seems to follow the computer-as-pupil metaphor

(Warschauer & Kern, 2000) (see section 2.2). By embracing the principles of learner-

centeredness in the process of courseware evaluation, this checklist can direct courseware

designers and material providers, balancing the design process with a focus on the learner to

avoid the overemphasis on technical characteristics.

5.3 Limitations of the Study

Three limitations need to be acknowledged and addressed regarding the present study.

First, only one courseware was evaluated. Evaluating more than one courseware would have

enabled comparisons that would have assisted and strengthened the analysis of the

effectiveness of the developed evaluation checklist. Second, the samples are small— 3

teachers and 22 students— which caused challenges in dealing with the quantitative data from

Stage II. For example, when one of the three teacher-evaluators did not respond to 14 of 39

evaluation questions, determining how to deal with these non-scored questions was a

dilemma. Therefore, the results from Stage II regarding the evaluations of MyET cannot be

generalized in regards to the strengths and weaknesses of MyET. Finally, the researcher did

not interview the student-evaluators for additional feedback on the use of the checklist due to

practical constraints. The three open-ended questions were insufficient for exploring the

student-evaluators’ opinions in depth.

83

5.4 Suggestions for Further Research

Certain avenues for further research are suggested. First, the learner-centered

evaluation checklist should be applied to more courseware with greater variety to examine its

effectiveness. Second, more participants should be involved, either quantitatively (more

teachers and students) or qualitatively (interview more evaluators to obtain more in-depth

data). In addition to English teachers and students, if possible, stakeholders, courseware

developers, and material providers should be invited to participate in the study to obtain their

feedback and— hopefully— focus their attention on the learner-centeredness in CALL

material evaluation. Moreover, in addition to collecting criteria from existing checklists,

researchers might also attempt to create several criteria of their own based on the principles

of learner-centeredness.

Furthermore, it might be interesting to see the checklist being used in different

perspectives of evaluation, both predictive and retrospective. For instance, it could be used in

the selection process before specific courseware is chosen; it can also be used as a tool to

examine whether the courseware being used is learner-centered or not. Determining the

correlation between learners’ performance using learner-centered CALL materials might be

another possibility for further research. For example, two or more courseware packages could

be compared with one another to evaluate the differences in their degree of learner-

centeredness. The learners could take a pre-test on their language performance and then, after

using both courseware packages for a while, complete a post-test to investigate their

performance; meanwhile, a comparison analysis could be conducted to determine whether

relatively learner-centered courseware enhances learners’ learning effectiveness.

5.5 Conclusion

As an English teacher for 8 years as well as a language learner for 20 years, the

researcher had used MyET for one year before the study was initiated. During this time,

84

while benefiting from the courseware, the researcher came up with specific questions

regarding the courseware design, such as the Automatic Speech Analysis System and the

design of learning tasks. Investigating more studies in CALL and the idea of user-

centeredness more deeply, the researcher became interested in finding out more about the role

of the learner— the primary user of the courseware— in CALL tool development and

evaluation. The user has increasingly been considered an important part in the process of

courseware development. The researcher felt such significance should be considered during

the process of courseware evaluation.

The present study made an effort to develop an evaluation checklist with a specific

focus— learner-centeredness— to incorporate more pedagogical aspects. Based on the

findings of this study, it can be claimed that the established learner-centered evaluation

checklist did function as expected. Nonetheless, the creation of a set of criteria for the

evaluation of CALL courseware is not without limitations. When one area receives more

attention, another might be neglected. Thus, developers and evaluators should remember that

where an evaluation is conducted and where the courseware is used are actually context-

dependent and dynamic. The development of the checklist does not mean that it will be

appropriate in all areas of language instruction; hence, adoption and adaptation of an

evaluation framework or checklist should be flexible and dependent on the context in which

the evaluation is conducted.

It is noted that, although this checklist has a specific focal point on learner-

centeredness, the study does not imply that the evaluation checklist should be solely and

exclusive ly applied. Instead, the checklist can accompany other evaluation instruments as

long as the combination will lead to effective evaluations and benefit users and the quality of

learning. Maddux, Johnson, and Willis (1997) believe that educators should “take a flexible

approach to the use of forms” (p. 344). Any type of checklists may need to be modified to suit

85

the purposes of the setting in which they will be used.

To sum up, effective learning relies on the focus of the learning process being

shifted

from the teacher to the learner. A learner-centered approach views learners as individuals

with

individual needs and rights. Only when the learner’s experience, talents, needs, and

interests are appreciated can the learner’s intrinsic motivation in learning thus be triggered

and increased. Learner-centeredness should always be considered in both the process of

developing learning material and designing courseware, as well as the process of

evaluating and selecting courseware. This study expects to contribute to the field by

putting a missing puzzle piece— awareness of learner-centeredness— into the wider

picture of the CALL field.