ARTICLE IN PRESS

International Journal of Educational Development XX (2004) XXX–XXXwww.elsevier.com/locate/ijedudev

Closing the digital divide: evaluation of the World Linksprogram

Robert Kozma∗, Ray McGhee, Edys Quellmalz, Dan ZallesCenter for Technology in Learning, SRI International, 2151 Filbert Street, San Francisco, CA 94123, USA

Abstract

In response to this digital divide between developed and developing countries in their use of computers to preparestudents for the global economy, the World Bank and, subsequently, the World Links organization provided schoolsin developing countries with networked computers and training that supports integration of ICT into teaching. Thisarticle synthesizes findings from 3 years of evaluative research on the program. The findings are based on surveys ofteachers, students, administrators, and technology coordinators, as well as a field test assessment of student learning.The study examines the services provided, documents their impact, and draws implications for policy in developingcountries. 2003 Elsevier Ltd. All rights reserved.

Keywords: International development; Educational technology; Educational policy

1. Introduction

Information and communication technology(ICT) can make a tremendous contribution tohuman development—but only for those that haveaccess. Technological innovation affects humandevelopment in two ways (United Nations Devel-opment Programme, 2001). It can directly contrib-ute to human capabilities by increasing people’sability to participate more actively in the social,educational, economic and political life of a com-munity. It can also support economic growththrough the productivity gains that it generates.Conversely, human development—particularly the

∗ Corresponding author. Tel.:+1-415-292-2471; fax:+1-415-292-4343.

E-mail address: robert.kozma@sri.com (R. Kozma).

0738-0593/$ - see front matter 2003 Elsevier Ltd. All rights reserved.doi:10.1016/j.ijedudev.2003.11.014

development of a highly skilled workforce—can inturn contribute to technology development.Together, these two developments can create a“virtuous circle” that can reduce poverty andimprove the human condition.

Unfortunately, these same technologicalresources are currently creating significant dispari-ties between developed and developing countriesaccording to aUnited Nations Development Pro-gramme report (1999). For example, the reportpoints out that in South Africa, the best-connectedAfrican country, 75% of the schools have no tele-phone line. For the sake of comparison, there are99 telephones for every 100 people in Monaco, butin Uganda, there are only two for every 1000. Thereport concludes that, “The global gap betweenhave and have-nots, between know and know-nots,is widening” (p. 57).

This growing digital divide is further docu-

ARTICLE IN PRESS2 R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

mented by several more recent reports. Bridges.org(2001) states that while there were 167 millionInternet users in North America and 113 millionin Europe, there were only 16 million in SouthAmerica and 3 million in all of Africa, accordingto data at the time of the report. While there hasbeen a dramatic increase in Internet users indeveloped countries—from an estimated 4% of theinhabitants in 1995 to 28% in 2000—the numberof users in developing countries has only inchedforward to only 3.6% in 2000. Similarly, a studyby the Organization for Economic Cooperation andDevelopment (OECD, 2001b) shows that there hasbeen a dramatic growth of Internet hosts—thesource of content on the Internet. But while thenumber of hosts in developed countries hasincreased from 23 per 1000 inhabitants in 1997 to82 per 1000 in 2000, the number in developingcountries has only increased from 0.21 per 1000 in1997 to 0.85 per 1000 in 2000. This lack of contentoriginating in and oriented to developing countriesis further compounded by the fact that 94% of theworld’s Internet content, as measured by the num-ber of links to pages on secured servers, is inEnglish.

The lack of access to technology reduces theprospect that citizens of developing countries willbe able to participate in the growing global econ-omy and minimizes the potential that technologyhas for improving their health, educational,governmental, and cultural institutions (UnitedNations Development Programme, 1999).

2. The World Links program

In response to this growing digital disparity andto requests from developing countries to assistthem in preparing their youth to enter an infor-mation age and participate in the global economy,the World Bank’s Economic Development Instituteconducted a pilot program, initially called WorldLinks for Development, from 1997 to 2002. Themission of the World Bank is to fight poverty andcontribute to sustained development (World Bank,1999). In its comprehensive development frame-work, the Bank posits that sustainable developmentrequires many social and structural elements in

addition to strong economic performance. Thedevelopment of educational capacity in clientcountries is a key element of the Bank’s strategy.While the primary mechanism for Bank support isthrough its program of strategic loans, the Bank’sEconomic Development Institute (now the WorldBank Institute, http://www.worldbank.org/) con-tributes to capacity building through training andoutreach programs (World Bank, 1997).

The Economic Development Institute initiatedthe World Links for Development in the belief thattechnology could be used to improve educationaloutcomes in client countries and facilitate culturalunderstanding across nations (World Bank, 1997).The program was started by connecting a singlehigh school in Uganda with one in the UnitedStates. By 1999, the program reached over 300schools in 15 countries. In 2000, the World Bankcreated an independent non-profit organizationcalled World Links (http://www.world-links.org/)to continue the program once the pilot was com-pleted. World Links has since grown to serve overa 1000 secondary schools in 26 developing coun-tries in Africa, Latin America, the Middle East, andSouth and Southeast Asia.

The World Links program aims to establish glo-bal, educational on-line communities for secondaryschool students and teachers around the world inorder to expand distance learning opportunities,enhance cultural understanding across nations,build broad support for economic and social devel-opment, and train teachers to integrate informationtechnology into the classroom. There are fivecomponents to the World Links program, eachelaborated below:

� Internet connectivity for secondary schools indeveloping countries.

� Training and educational content to promoteeconomic and social development.

� Regional and global partnerships with public,private, and non-governmental organizations.

� Telecommunications policy advice for the edu-cation sector.

� Monitoring and evaluation support.

Internet connectivity has been an essentialcomponent of the World Links program. For coun-

ARTICLE IN PRESS3R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

tries and schools with the most need, the programprovided up-to-date multimedia computer equip-ment and networking for the school and computerlaboratory. The program provided several alterna-tive networking models, depending on the net-working resources and bandwidth that were avail-able.

Training was also an essential component. Acentral tenet of the program is that the introductionof technology will alone not improve education.Teachers from participating schools received train-ing in the use of equipment and typical software.But more important was training on the integrationof ICT into classroom teaching. The programadopted a constructivist approach to teaching withICT that emphasized a shift away from teacher-centered lecture-based instruction towards student-centered, project-based learning (Carlson, 2002).Among the ultimate goals of the program was toencourage students’ development of higher-orderthinking and information-reasoning skills. Theseskills are cited as particularly important for partici-pation in the knowledge-based global economy(OECD, 1996, 2001a; 21st Century Partnership,2003). To accomplish this goal, World Links train-ing was to provide teachers with pedagogicalapproaches to incorporate ICT into their teachingand support student acquisition of advanced skillsinstructional goals. By 2000, the World Linksteacher training curriculum included the followingfour phases:

Phase I: Introduction to the Internet for teachingand learning.Phase II: Introduction to educational telecollabor-ative projects.Phase III: Integrating technology and curricula.Phase IV: Evaluating and diffusing innovativeclassroom practices.

The program also created a set of regional andglobal partnerships, such as I∗EARN, GLOBE,and ThinkQuest, that created digital content, activi-ties, and training materials for teachers and stu-dents. These partnerships and their services sup-ported teachers in their implementation of ICT andthe pedagogical strategies advocated by WorldLinks.

The program also provided ministries of edu-cation and other governmental agencies with con-sultation and policy materials that supported thedevelopment of ICT and its use to improve edu-cational systems. Among the advice was the advis-ability of drawing on World Bank lendingresources to support additional programs that useICT in light of the results of the pilot program intheir country.

The fifth component of the World Links pro-gram was monitoring and evaluation. To informthe decisions of national policy makers and pro-gram managers, an evaluation was designed toexamine the World Links program on theimplementation and effectiveness of the inter-vention and barriers that need to be overcome toassure its success. The evaluation was conductedby the Center for Technology in Learning at SRIInternational (http://www.sri.com/policy/ctl/).

This article summarizes the findings from evalu-ative research that was conducted over a 3 yearperiod (Kozma and McGhee, 1999; McGhee andKozma, 2000; Quellmalz and Zalles, 1999, 2000).The focus of the evaluation was on documentingthe services actually provided by the program,studying the classroom implementation of the pro-gram and the effect that it had on classroom prac-tice, examining initial indications of student andteacher outcomes and long-term impacts, and ana-lyzing the barriers that may limit the effectivenessof the program. Our findings indicate that the pro-gram provided a variety of intended services toschools and teachers and that these were effec-tively implemented in participating schools andclassrooms. The use of ICT in these schools wasassociated with student-centered, constructivistpedagogical practices. The data we were able toobtain indicate that students acquired the skillsthey would need to participate in the global knowl-edge economy. A range of barriers—from the needfor more reliable connections in schools, to theneed for more time in the curriculum for computer-based activities and national policies that integrateICT into education reform—limited the program’simplementation and impact. But the extent of theprogram’s success despite these barriers has impli-cations for the use of ICT to support education

ARTICLE IN PRESS4 R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

improvement in developing countries and for thepolicies that can increase its impact.

3. Study design

As mentioned above, the goal of our evaluationwas to document the implementation of the WorldLinks program and assess its effectiveness. Thefocus of the evaluation was on the services pro-vided by the program, the extent to and ways inwhich these services were implemented in theclassroom, and student and teacher outcomes andthe extent to which they could be attributed to theprogram. In addition, we were concerned with thebarriers that teachers encountered in implementingthe program. Our purpose was to provide programstaff with information that could be used toimprove the program and to provide governmentswith information about the value of ICT invest-ments in education and polices that may be neededto increase the impact of these investments.

The evaluation of the World Links program wassituated within a conceptual framework in whichthe impact of educational improvement efforts isinfluenced by a number of variables, in addition tothe intervention itself, which mediate the impact ofthe program (see Fig. 1). The ultimate goal of the

Figure 1. Conceptual Framework for World Links Evaluation

World Links program was to contribute to suchoutcomes as improving the technological andinformation skills of students and teachers andlong-term impacts such as improved test scores,increased cross-cultural understanding, andimproved job prospects in the developing knowl-edge economy. However, these ultimate goals aremediated by a variety of intermediate factors, mostsalient being the extent to which the services pro-vided by the program are implemented and sup-ported by the participating schools. In addition,there are other “exogenous” variables unrelated tothe program which may influence the implemen-tation of the program and its outcomes andimpacts, factors over which the program and theBank may have very little influence. We used thisconceptual framework to guide the construction ofour research instruments. Within this framework,our focus was on the services of the program, theways their implementation affected classroompractice, and the impact of these practices, parti-cularly on the skills students would need to partici-pate in the knowledge economy.

Previous research suggests that the services pro-vided by the World Links program—specificallythe ICT infrastructure and teacher training—wouldhave an effect on classroom practice, particularlythe training of teachers to integrate ICT into their

ARTICLE IN PRESS5R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

classes. Classroom studies (Means and Olson,1995; Sandholtz et al., 1997; Means et al., 2001;Schofield and Davidson, 2002) have shown thatsuch interventions can have a direct effect on thepedagogical strategies used by teachers, the extentto which computers are used within these stra-tegies, the amount of time students are engaged inthe use of computers, and the kinds of projects thatstudents conduct with computers. The examinationof this relationship was a primary focus of theevaluation.

The classroom implementation, so influenced,may have a subsequent effect on short-term out-comes, such as the acquisition of certain knowl-edge, skills, and attitudes. Ultimately, the inter-vention may have longer-term impacts on studentretention rates or their preparation for the workworld. The goal often cited in the literature is toprovide students with the ability to access, analyze,evaluate, communicate, and use information tosolve problems and create new knowledge(Educational Testing Service (ETS), 2002; Inter-national Society for Technology in Education(ISTE), 2000; OECD, 2000; Quellmalz andKozma, 2003)—skills that are needed to participatein the global knowledge economy. However, therelationship between the use of ICT and variousoutcomes and impacts is an area where researchresults are less conclusive, with some studies find-ing negative results (Pelgrum and Plomp, 2002;Wenglinski, 1998) and some finding positiveresults (NCES, 2001; Wenglinski, 1998). The con-flicting results suggest that the mere introductionof computers is insufficient to affect outcomes; theway computers are used in the classroom matters.In our evaluation, we examined this relationshipbetween the use of computers, the use of specificpedagogical practices, and certain student out-comes. We designed this as a comparative studyso that some attributions could be made to the ser-vices provided by the program.

The evaluation was also situated within a practi-cal context of limited time and budget. As in manyevaluations, these pragmatic constraints influencedthe way in which we were able to assess the goalsand examine the relationships imbedded in ourconceptual framework. We had to rely on the self-report of participants for much of our data. How-

ever, we triangulated findings by the use of mul-tiple sources (students, teachers, technology coor-dinators, and administrators), periods (the programin 1999 and 2000), participating countries, andmethods (Tashakkori and Teddlie, 1998; Shavelsonand Towne, 2001). Survey instruments weredesigned to collect information on the services pro-vided by the program, teacher and student class-room practices and the extent to which computerscontributed to these, the effect of the program andthe use of computers on teacher and student knowl-edge, skills, and attitudes. We collected data fromstudents, teachers, technology coordinators, andadministrators. Also, sets of performance assess-ments were specifically designed to directly meas-ure the impact of the program on student learning.To address certain research questions related torelative effects of the program vs. the mere useof computers, we included comparison groups ofteachers and students who used computers but didnot participate in the program, as described below.

3.1. Surveys of students, teachers, andadministrators

The purpose of the surveys was to collect self-reported responses about the program servicesreceived, the implementation of these services inthe classroom, and the outcomes and impact of theprogram on students and teachers. Survey datawere collected during the 1998–1999 and 1999–2000 operational years. During 1998–1999, surveydata were collected from samples of adminis-trators, teachers, technology coordinators, and stu-dents from five countries. During 1999–2000, sur-vey data were collected from more countries (12in all) but only from teachers.

In 1998–1999, the evaluation was conducted infive of the 13 currently participating countries:Chile, Paraguay, Peru, Senegal, and Uganda. Ineach of the five countries in which the evaluationwas conducted, six World Links schools wereasked to participate in data collection. Because theresponses were self-reports, the data were triangu-lated with multiple types of respondents. At eachschool, samples of students and teachers were sur-veyed along with the headmaster or headmistressand the school’s technology coordinator. In total,

ARTICLE IN PRESS6 R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

26 World Links schools participated in the evalu-ation. There were nearly 20,000 students served bythese schools, more than 12,000 girls and nearly8000 boys. There were over 1200 teachers in theseschools. A total of 661 World Links students and83 World Links teachers responded to the surveyfrom these schools. Also 25 administrators and 23technology coordinators responded. The responserate was at or above 80% for all schools. Becausewe wanted to assess the extent to which studentand teacher outcomes could be attributed to theWorld Links program, two other groups wereidentified for comparison purposes. One was agroup of non-participating or “ least participating”teachers and students within World Links schools.These were teachers and students in participatingschools who did not participate in the program orparticipated in a minimal way; respondents in thisgroup included 191 teachers and 441 students. Asecond comparison group was administrators, tea-chers, and students from nine schools that did notparticipate in the program but were asked to par-ticipate in the evaluation based on their compar-ability to World Links schools. Within theseschools, 90 teachers, 378 students, and nine admin-istrators responded to surveys. To assure parti-cularly useful comparisons, both computer usingand non-computer using teachers and studentswere surveyed in non-World Links schools. Of thetotal respondents in non-World Links schools, 32teachers and 247 students were computer users.1

Relating to pedagogical practices and outcomes,survey questions were constructed so that identicalresponses could be collected from both WorldLinks participants and non-participants and bothcomputer users and non-users. For example, ques-tions were asked of all respondents about the useof practices such as collaborating with other stu-dents on a project, gathering data for a researchproject, writing project reports, and so on. If the

1 The total number of respondents by country in 1998–1999was: Students—Chile (255), Paraguay (222), Peru (390), Sen-egal (262), and Uganda (350); teachers—Chile (48), Paraguay(47), Peru (109), Senegal (76), and Uganda (84); technologycoordinators—Chile (6), Paraguay (4), Peru (6), and Uganda(6); administrators—Chile (6), Paraguay (6), Peru (7), Senegal(8), and Uganda (6).

respondents used computers, they would then beasked what role computers played in these prac-tices. If they were participants in the World Linksprogram, they would then be asked the extent towhich their participation contributed to the prac-tices. Consequently, the occurrence of certain prac-tices could be compared between program parti-cipants and non-participants and between computerusers and non-users to see if there were differ-ences. Subsequent questions could provideadditional information about the extent to whichthe respondents felt that the use of computers orthe participation in the program contributed to anydifferences that existed.

In 1999–2000, 12 of 15 countries that were part-icipating in the World Links program at the timewere included in the study. These were: Brazil,Chile, Colombia, Ghana, Mauritania, Mozam-bique, Paraguay, Peru, Senegal, South Africa,Uganda, and Zimbabwe. The intent of this studywas to see if 1998–1999 findings held for newcountries that had subsequently joined the pro-gram. As mentioned above, data were collectedonly from teachers. The design included bothWorld Links and non-World Links schools: 98 and42, respectively. In World Links schools, four tea-chers were included for each school. To examinethe impact of the program on a broader range ofteachers in participating schools, two of the fourwere teachers who had received formal trainingfrom the program; one had received training froma colleague; and one teacher had not received train-ing. A total of 383 World Links teachers partici-pated in the survey and 158 teachers responded innon-World Links schools; the response rate was97%.2

3.2. Assessment of student learning

Because we wanted to go beyond self-report ofthe impact of the program on student outcomes,we devised a direct measurement of student

2 The total number of teacher respondents by country in1999–2002 was: Brazil (56), Chile (40), Columbia (42), Ghana(40), Mauritania (36), Mozambique (32), Paraguay (36), Peru(56), Senegal (56), South Africa (56), Uganda (55), and Zim-babwe (36).

ARTICLE IN PRESS7R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

achievement. There were some unique challengesin the design of student learning assessments forthis study. First, the World Links program wasimplemented in a wide range of subject areas,including science, social studies, language, andcomputer science. Furthermore, different countrieshad different curricula and expectations for studentlearning. Both these factors made it problematic todesign an instrument that would measure the learn-ing of specific school subject matter. However, anoverriding consideration in designing the studentassessment was the interest in measuring one ofthe most important outcome goals of the program:to prepare students for work in the knowledge-based global economy. Consequently, rather thanmeasure the learning within the multitude of sub-ject areas in which the program was implemented,the assessment of student learning focused on ICTskills and the use of ICT to search for, organize,and communicate information. These are the kindsof skills that are most likely to be influenced by theprogram, regardless of the subject area in which itis implemented, and they are the skills identifiedin policy documents (ISTE, 2000; OECD, 1996,2001a; World Bank, 2002) as those needed forworkers in the global knowledge economy.

With this goal in mind and drawing on the Edu-cational Technology Standards for Studentsdeveloped by the International Society for Edu-cational Technology (ISTE, 2000), the SRI teamidentified key skill components for three outcomeareas: reasoning with information, communication,and technology use. The SRI team then designedsets of performance assessments that engaged stu-dents in complex tasks of searching for, organiz-ing, and analyzing information and designed indi-cators that measured skill components, such asformulating a search query and communicating anargument (Quellmalz and Zalles, 1999, 2000). As aculminating task in one of the assessments, WorldLinks students were asked to write a newsletter forother students about the plight of two endangeredspecies. In response to this task, students gatheredinformation from Web pages about the twoendangered species, specified a line of inquiry forfurther research and a Web search query phrase,and then prepared a news article making evidence-based predictions about the likelihood that the two

species would survive. Students were also asked toinsert and annotate a relevant graphic. A parallel“paper-and-pencil” version of the task wasdesigned for students in the comparison group whodid not use ICT. The assessments were pilot testedin a World Links school in Uganda and in Parag-uay. Scoring rubrics were developed and scorerswere trained to assess the quality of studentsreasoning with information, communication, andtechnology skills.

The student assessment field test conducted in2000 involved six World Links schools and fourcomparable non-World Links schools in Uganda.There were a total of 200 students assessed; 121from World Links schools and 79 from non-WorldLinks schools. Students in non-World Linksschools were not assessed on their technologyskills.

4. Results

4.1. Services provided

Our evaluation began with an analysis of the ser-vices that the World Links program actually pro-vided to participating schools. The World Linksprogram is a distinctive blend of infrastructure andhuman capacity development. Perhaps, the mostvisible contribution of the World Links program isthe donation of computer hardware and software.Equipment was provided only to those schoolswhere the current lack of equipment limited theirparticipation in the program. At the time of the1998–1999 survey, the technology coordinators in60% of the schools reported that their schools hadreceived equipment from the program. The coordi-nators reported having received an average ofnearly nine computers for each school. The largemajority of these computers were located in a com-puter laboratory along with other computers thatthe school had previously acquired.

One of the outcomes of the World Links pro-gram—one that significantly increases itsefficiency—is the additional resources that thedonation of equipment and participation in the pro-gram fostered. The relatively modest donationsfrom the program were leveraged to get additional

ARTICLE IN PRESS8 R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

resources, as they were often matched by signifi-cant local or national contributions. For example,the computer laboratories were often built withfunds from a country’s Education Ministry or fromlocal resources. In Senegal, the Ministry of Edu-cation built or remodeled laboratories at each ofthe World Links schools at a cost in excess of $20,000 for each school. Contributions of additionalequipment in Uganda were often made by old girlsand old boys clubs, as well. Other local funds, typi-cally student fees that ranged from $ 1 to $ 50 ayear, covered operating costs, such the cost ofpaper, ink, and dial up access. In most cases, stu-dents were able to use the resources even if theycould not afford to pay the fee. In some schools,parents and students supplemented computerresources through fund-raising events. Forexample, in a poor, all-girls school in Peru, stu-dents raised funds by performing musical and the-atrical shows for parents and neighbors.

While equipment donation was, perhaps, themost visible component of the program, it wasthrough the professional development of teachersthat the World Links program made its most sig-nificant contribution. In the 1998–1999 survey,World Links technology coordinators reported that336 of the teachers in their schools, or an estimated28% of their entire teaching staff, had receivedtraining as part of their school’s participation in theprogram. During the first year, most of the trainingtopics focused on the use of technology; 92% ofthe teachers said they received training on the useof the Internet and 75% received training on thedevelopment of Web pages. Teachers also saidthey received training on the use of computer hard-ware (70% of the participating teachers) and appli-cations software (64%). The workshops alsofocused on constructivist approaches to learningthat emphasized project-based learning and studentcollaboration. Nearly, 90% of the teachers saidthey received training on how to design and leadcollaborative student projects; 70% received train-ing on how to use student groups in their teaching;and 59% received training on how to collaboratewith other teachers on the development of instruc-tional materials.

These percentages were smaller for the largergroup of teachers in 12 countries that responded to

the survey in 1999–2000, due in part to theinclusion of a broader range of teachers in theWorld Links schools. In these schools, 62% of theteachers said they received training on designingand leading collaborative groups; 52% receivedtraining on using student groups in teaching; and45% received training on collaboration with teach-ers to develop instructional materials.

As a result of their experience, 93% of WorldLinks teachers in the five countries studied in1998–1999 expressed satisfaction with the way theprogram was implemented. In the 1999–2000 sur-vey, 90% the teachers in the 12 countriesexpressed satisfaction.

Administrators also received training as part oftheir school’s participation in the World Links pro-gram; only two of the 18 administrators whoresponded to the question in the 1998–1999 surveysaid they did not. Training in the use of Internetsoftware and in the design of collaborative studentprojects were mentioned most often as the topics ofthe training. Correspondingly, all 21 administratorswho responded to the question said they were sup-portive of the program and 91% said they weresatisfied with its implementation to date.

The school technology coordinator played a cru-cial role in teacher professional development.Technology coordinators were frequently teacherswho had some prior computer experience. Oftenthey taught mathematics or science courses. Thecoordinators served a large number and a wide var-iety of functions with 70% or more saying that theytrained teachers in the use of hardware, appli-cations software, the use of the Internet, and in thedesign of collaborative projects and ways to inte-grate technology into the curriculum. The coordi-nators also worked extensively with students. Morethan 80% of the coordinators said they trained stu-dents in the use of hardware, software, and theInternet. Many coordinators also serviced the hard-ware, with 57% saying that they maintained thecomputer equipment and 52% saying they adminis-tered the school’s network. These teachers oftenprovided these special services while continuingtheir teaching duties.

In summary, while many schools received com-puters from the World Links program and lever-aged these donations to obtain more resources, the

ARTICLE IN PRESS9R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

primary service provided by the program wasteacher training. Training covered both the use ofhardware, software, and the Internet, as well as theuse of computers within new, constructivist-ori-ented pedagogical strategies.

4.2. Implementation

The second focus of the evaluation was on theextent to which these services were used in theclassroom. We found that World Links training andthe availability of networked computers were asso-ciated with significant differences in what WorldLinks teachers and their students did in theirclassrooms, compared to computer-using teachersand students in non-World Links schools. In the1998–1999 survey, World Links students in fivecountries were much more likely than computer-using students in non-World Links to report thatthey engaged in a number of pedagogical practicesin their classrooms (see Table 1). Some of thesedifferences related to the use of various techno-logies. For example, while 52% of the World Linksstudents said they used the Internet and 40% saidthey used a search engine, only 16% of the com-puter-using non-World Links students said theyused the Internet and 19% said they used a searchengine. Similarly, while 34% of the World Linksstudents said they used email, only 21% of thecomputer-using non-World Links students saidthey did so. But some of the most dramatic differ-ences relate to other learning activities. Forexample, 38% of the World Links studentsreported that they gathered data for a research pro-ject, 31% collected information about anothercountry or culture, and 23% collaborated on a pro-ject with students from another country. On theother hand, 22% of computer-using non-WorldLinks students reported that they gathered data fora research project, 18% collected informationabout another country, and only 4% collaboratedon a project with students from another country.These differences held up when World Links tea-chers were asked parallel questions about the class-room practices of their students in the 12 countriesparticipating in the 1999–2000 study (see alsoTable 1).

As a result of the program’s emphasis on collab-

orative, project-based learning, World Links stu-dents frequently conducted projects with other stu-dents, often with students from other schools andother countries. The titles of some of the studentprojects in Uganda were Women in Mathematics,The Democracy Schools Project, The UtopianVision, Refugees, The Wetlands, Faces of War, andCulture and Technology. In Senegal, topicsincluded Technology and the Generation Gap,Oppression and Human Violence, Women andDevelopment, and Girls’ Education in Senegal.These projects were shared with students in Eur-ope, Canada, the United States, and South Koreafor their reaction. Box 1 describes a few of theWorld Links ecology-related student projects in abit more detail. In these projects, students workedtogether while using computers, software appli-cations, and the Internet to collect information,analyze it, write up reports, and post their resultson the Internet or communicate with students else-where.

In summary, participation in the World Linksprogram lead to the classroom use of a variety ofcomputer applications and resulted in significantdifferences in the use of collaborative, project-based pedagogical practices, relative to their useby non-participating schools.

4.3. Outcomes

The goal of the World Links program was toinfluence the learning outcomes of secondaryschool students and teachers in developing coun-tries and, ultimately, to have an impact on othervariables such as school attendance, examinationscores, graduation rates, and the employment ofstudents. While outcome and impact variables arethe most difficult to measure and most susceptibleto a range of influences apart from the program,we attempted to get an initial indication of the pro-gram’s effect by asking students, teachers, andadministrators questions about them. In addition,we compared reported outcomes in participatingschools to those in non-participating schools. Aswell, we directly measured student outcomes in asample of World Links and non-World Linksschools in one country. We used the triangulationof these responses as in indication of their veracity.

ARTICLE IN PRESS10 R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

Table 1Classroom practices as reported by students and teachers

How often did you use computers to do each of the following: Percentage of each group who responded “1–3 times amonth” or “once a week or more” a

World Links Non-World Links World Links teachersstudents 1999 computer-using 2000b

students 1999

Use applications software 52c 42 71Use CD-ROMs 34c 21 50Use e-mail 47c 25 62Use a search engine to find information on the Web 40c 19 56Use bulletin boards or listservs 26c 15 35Use the Internet 52c 16 59Use drill and practice materials 51 46 56Use spread sheets to analyze data 34c 31 28Produce a Web page 23c 9 16Collaborate on a project with other students in the same class 47c 30 53Collaborate on a project with students from another school in 25c 13 27your own countryCollaborate on a project with students from another country 23c 4 23Exchange information with another students from another country 28c 8 29Gather and analyze resource materials on a problem or topic 41c 25 44Gather data for a research project 38c 22 40Gather evidence to argue a position about an issue 34c 25 39Write project reports 35c 24 35Use graphics in a report 34 28 35Collect information about another country or culture 31c 18 26Draw conclusions or make predictions using data you gathered or 35c 28 34obtained from resource materialsCommunicate with your parents or other members of the 41c 31 28community about what you do in school

a The response options provided to respondents were: not at all, 1–5 times during the year, more than five times during the yearbut less than once a month, average of 1–3 times a month, and average of once a week or more.

b Teachers were asked about the extent of student practices and were provided with the same response option as students.c World Links and non-World Links students are statistically different (pair-wise t-test, p � 0.05) for full range of responses.

Nonetheless, these findings must be consideredpreliminary until direct measures of learning canbe systematically and widely collected.

4.3.1. Student outcomesIn the 1998–1999 survey, a majority of students

in the World Links program said that it had a sig-nificant effect on a variety of knowledge, skills,and attitudes (see Table 2). Perhaps, it is not sur-prising that 66% of students rated the program veryhighly for its impact on their attitudes toward tech-nology and over 60% said their technology skillsimproved. However, the most highly rated out-

come of the program was on students’ opinionabout their ability to get better jobs upon gradu-ation; 77% of the World Links students rated thisoutcome as very high. In addition, 71% rated theprogram very high on its effect on their communi-cation skills and 61% on its effect on their abilityto reason with information. A large majority saidthat the program improved their attitudes towardschool (66%) and their school attendance (64%).And 60% rated the program very highly in itseffect on their knowledge about other cultures asa result of their participation. In the 1998–1999survey, an overwhelming 92% of the students in

ARTICLE IN PRESS11R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

Table 2Student outcomes attributed to World Links

How much impact the participation in the World Links Percentage of each group that responded “very much” a

program has on students:

World Links World Links World Links World Linksstudents 1999 teachers 1999 administrators 1999 teachers 2000

Improved technology skills 63 88 65 69Improved attitudes towards technology 70 86 71 78Improved ability to reason with information 61 71 65 63Improved communication skills 71 74 73 70Increased knowledge or awareness of other cultures 60 67 47 61Improved attitudes toward school 66 78 76 67Improved school attendance 64 31 50 44Improved ability to get bettor jobs upon graduation 77 48 25 59

a The response options provided to the respondent were: not at all, somewhat, and very much.

World Links schools expressed satisfaction withtheir participation in the program.

Teachers and administrators generally agreedwith students’ opinions of the program’s impact(see also Table 2). In the 1998–1999 survey, 88%of the teachers and 65% of the administrators ratedthe program very highly for its positive impact onstudent technology skills and 86% of the teachersand 71% of the administrators rated it highly onits impact on students’ attitudes toward technology.Similarly, these respondents rated the programvery high on its effect on students’ ability to reasonwith information (71% of the teachers and 64% ofthe administrators), on their communication skills(74% of the teachers and 73% of theadministrators), and on their attitudes towardschool (78% of the teachers and 76% of theadministrators). Two thirds of the teachers ratedthe program very highly on its impact on students’knowledge of other cultures and on their schoolattendance. Teachers were less enthusiastic thanstudents on their assessment of the program’simpact on student school attendance and on stu-dents’ ability to get better jobs. Administratorswere less optimistic about the program’s impact onstudents’ job prospects, on their school attendance,and on their knowledge of other cultures.

These positive assessments of the program’simpact were shared by the larger group of teachersfrom 12 countries that responded to the survey in

1999–2000 (Table 2). A large majority of the tea-chers rated the program very highly on its impacton all the outcomes, especially its impact on stu-dents’ attitudes toward technology, their communi-cation skills, their technology skills, and their atti-tudes toward school.

4.3.2. Student outcomes comparedWhile the questions reported above asked for

opinions about outcomes specifically linked to theWorld Links, the questions described in this sec-tion asked respondents to report on learning out-comes related to their use of computers but withoutreference to the program. Consequently, the ques-tions could be asked of students and teachers whodid not participate in the program as well as thosewho did.

Table 3 compares the responses of three othergroups of students from the 1998–1999 survey:students who participated in the World Links pro-gram, computer-using students who were in thesame schools but did not participate in the WorldLinks program, and computer-using students innon-World Links schools. On a wide range ofknowledge, skills, and abilities, World Links stu-dents rated their learning high as a result of usingcomputers and higher than did students in the sameschool who did not participate in the program. Amajority of World Links students rated their learn-ing highly and had higher end-of-year ratings on

ARTICLE IN PRESS12 R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

Table 3Student learning outcomes as reported by various groups of students

Use of computers has improved your knowledge, skill, and Percentage of each group that responded “very much” a

abilities in the following areas:

Participating World Non-participating Non-World LinksLinks students 1999 World Links computer-using

students 1999 students 1999

Knowledge about academic subjects 40b,c 33 36Planning, regulating and monitoring of your own learning 54b,c 43 35Finding, comparing, and evaluating information 53b,c 52 42Analyzing and interpreting information 57b,c 49 42Writing reports 67c 54 57Communicating with others 60c 60 55Using images, drawings, or graphics to represent ideas 52 54 51General knowledge about current events 53c 44 37Knowledge about other cultures, countries, or languages 53b,c 44 34Collaborating with others 63c 58 51Using computers 77b 67 73Using computers software 58c 52 49Using Internet 66b,c 58 45

a The response options provided to respondents were: not at all, somewhat, very much.b World Links and non-participating World Links students are statistically different (pair-wise t-test, p � 0.05) for full range

of responses.c World Links and non-World Links computer-using students are statistically different (pair-wise t-test, p � 0.05) for full range

of responses.

their ability to plan their own learning; their abili-ties to analyze and to interpret information; theirabilities to communicate, to write reports, and touse graphics to represent ideas; their ability to col-laborate with others; their knowledge of other cul-tures; and, of course, their skills in using com-puters, software, and the Internet. Only 40% of theWorld Links students gave a high rating to theimpact of computers on their knowledge of aca-demic subject matter, but this was still higher thanstudents in the two other groups. And only for theuse of graphics did World Links students report nomore learning than one or both of the other groups.

Similarly, teachers who participated in theWorld Links program were significantly morelikely than computer-using teachers in otherschools to report higher ratings for their students’learning and attitudes in a number of areas. Table4 shows that a majority of the World Links teach-ers from the five countries participating in the1998–1999 survey rated the use of computers ashaving a high impact on student learning in all but

two of the indicators—student planning of theirown learning and their analysis of information.They rated the impact of computers on studentlearning higher than teachers in one or both of theother groups for communication skills, knowledgeof other cultures, collaboration with others, andInternet skills. Similarly, World Links teachersfrom the 12 countries who responded to the 1999–2000 survey were more likely than non-WorldLinks teachers to highly rate the impact of com-puters on student communication skills, knowledgeof current events, knowledge of other cultures, col-laboration skills, and Internet skills.

4.3.3. Student outcomes assessedWhile the survey responses of students, teachers,

and administrators all attested to the impact ofcomputers and the World Links program on stud-ent learning, this is not as compelling as a directmeasure of student learning. In 2000, the perform-ance of 121 students in six World Links schoolsand 79 students in four comparable non-World

ARTICLE IN PRESS13R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

Tab

le4

Stud

ent

lear

ning

outc

omes

asre

port

edby

vari

ous

grou

psof

teac

hers

Use

ofco

mpu

ters

has

help

edyo

urst

uden

tsim

prov

eth

eir

Perc

enta

geof

each

grou

pth

atre

spon

ded

“ver

ym

uch”

a

know

ledg

e,sk

ills,

and

abili

ties

inth

efo

llow

ing

area

s:

Part

icip

atin

gW

orld

Non

-par

ticip

atin

gN

on-W

orld

Lin

ksW

orld

Lin

ksN

on-W

orld

Lin

kste

ache

rs19

99W

orld

Lin

kste

ache

rsco

mpu

ter-

usin

gte

ache

rs20

00L

inks

teac

hers

1999

teac

hers

1999

2000

Kno

wle

dge

abou

tac

adem

icsu

bjec

tsb

5648

3739

34Pl

anni

ng,

regu

latin

g,an

dm

onito

ring

thei

row

nle

arni

ng47

5231

4330

Find

ing,

com

pari

ng,

and

eval

uatin

gin

form

atio

n53

5442

5054

Ana

lyzi

ngan

din

terp

retin

gin

form

atio

n47

5036

4855

Wri

ting

repo

rts

5654

7355

59C

omm

unic

atin

gw

ithot

hers

79c

5542

67d

46U

sing

imag

es,

draw

ings

,or

grap

hics

tore

pres

ent

idea

s53

6174

5656

Gen

eral

know

ledg

eof

curr

ent

even

ts65

5948

62d

36K

now

ledg

eab

out

othe

rcu

lture

s,co

untr

ies,

orla

ngua

ges

75c,

e46

2253

d38

Col

labo

ratin

gw

ithot

hers

80c,

e51

4865

d35

Usi

ngco

mpu

ter

soft

war

e81

7177

6458

Usi

ngth

eIn

tern

et75

c,e

4435

67d

35

aT

here

spon

seop

tions

prov

ided

tore

spon

dent

sw

ere:

not

atal

l,so

mew

hat,

very

muc

h.b

The

wor

ding

ofth

isst

atem

ent

was

slig

htly

diff

eren

tfo

rth

e19

99–2

000

surv

ey.

Itst

ated

“per

form

ance

onte

sts

ofac

adem

icsu

bjec

ts”.

cW

orld

Lin

ksan

dno

n-W

orld

Lin

ksco

mpu

ter-

usin

gte

ache

rsar

est

atis

tical

lydi

ffer

ent

(pai

r-w

ise

t-te

st,

p�

0.5)

for

full

rang

eof

resp

onse

s.d

Wor

ldL

inks

and

non-

Wor

ldL

inks

teac

hers

are

stat

istic

ally

diff

eren

t(p

air-

wis

et-

test

,p

�0.

5)fo

rfu

llra

nge

ofre

spon

ses.

eW

orld

Lin

ksan

dno

n-pa

rtic

ipat

ing

Wor

ldL

inks

teac

hers

are

stat

istic

ally

diff

eren

t(p

air-

wis

et-

test

,p

�0.

05)

for

full

rang

eof

resp

onse

s.

ARTICLE IN PRESS14 R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

Links schools in Uganda were measured, scored,analyzed, and compared (Quellmalz and Zalles,2000). Table 5 shows the number of performancesof each component that were exhibited in theseopen-ended tasks and the percent that was rated as“adequate” or “above adequate” by raters.

The World Links schools out-performed thenon-World Links schools on all components ofreasoning with information and communicationthat were measured by the assessment. The differ-ences were greatest in the components of reasoningwith information that involved finding and categor-izing relevant information (85% for World Links,59% for non-World Links), making comparisons(63% for World Links, 46% for non-World Links),and using information to make supported predic-tions (62% for World Links, 50% for non-WorldLinks). Overall, the percentage of “adequate” or“above adequate” responses by students in WorldLinks schools on reasoning with information was14% higher than the ratings of student responsesfrom non-World Links schools and 5% higher oncommunication.

The lowest scores for both World Links andnon-World Links schools were for ratings of howwell the students’ news articles presented and sup-ported their predictions. Consequently, the differ-ences between the two groups were least pro-nounced for this outcome area. While slightlyhigher than the non-World Links group, only 37%of the World Links students’ news articles wererated as presenting an “adequate” or “aboveadequate” predictions.

World Links school students out-performed non-World Links students on ratings of the logicaldevelopment of their news articles. Seventy-fourpercent of the news articles written by World Linksstudents were rated as “adequately organized” or“well organized” , in comparison to 65% of thenews articles written by students in the non-WorldLinks schools.

The students in World Links schools did quitewell in their display of technology skills, as scoredby raters. Seventy-four percent of the World Linksstudents received ratings of “adequate” or “above”on tasks requiring searching for information on theWeb. Sixty-three percent received “adequate” or“above adequate” ratings on either formulating or

identifying an appropriate phrase for a Web searchquery on a specific topic.

4.3.4. Teacher outcomesStudents were not the only ones to benefit from

participation in the World Links program; teachersbenefited in a number of ways. As mentioned earl-ier, a major component of the World Links pro-gram is its emphasis on the professional develop-ment of teachers and on improved pedagogy.

As a result of their participation in the WorldLinks program, a large majority of the World Linksteachers in the 1998–1999 survey felt that they gre-atly improved not only their ability to use computerhardware (70%), applications software (67%), andthe Internet (73%) but their ability to use studentgroups in their teaching (63%) and to design andlead collaborative student projects (68%), as shownin Table 6. A majority of teachers (52%) indicatedthat the program increased their ability to collabor-ate with other teachers on the development ofmaterials.

A very large percentage of administrators feltthat the World Links program greatly affected tea-chers’ computer skills (82%), their ability to usethe Internet (80%), their attitudes about technology(85%), and their attitudes about their own teaching(90%). A majority felt that the program improvedteachers’ software skills (67%) and their ability touse student groups in their teaching (60%). Bothteachers and administrators in the 1998–1999 sur-vey agreed that program had limited impact on tea-chers’ ability to integrate computers into curricu-lum integration and assessment.

Teachers from the 12 countries participating inthe 1999–2000 survey agreed with their colleaguesin the earlier survey on most questions (also inTable 6). These teachers rated the program’simpact as higher on integration of computers intothe curriculum but they also felt that impact onassessment design was limited.

4.3.5. SummaryIn summary, data from administrators, teachers,

and students indicate that students who participatedin the World Links program learned a range of newskills, more so than those in non-participatingschools. These findings were confirmed by a larger

ARTICLE IN PRESS15R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

Tab

le5

Sum

mar

yby

outc

ome

area

and

skill

com

pone

nts

for

the

stud

ent

asse

ssm

ent

inU

gand

a

Out

com

ear

eaSk

illco

mpo

nent

Wor

LD

scho

ols

Non

-Wor

LD

scho

ols

Num

ber

ofre

spon

ses

Perc

enta

geof

resp

onse

sN

umbe

rof

resp

onse

sPe

rcen

tage

ofre

spon

ses

“ade

quat

e”or

“abo

ve”

“ade

quat

e”or

“abo

ve”

“ade

quat

e”or

“abo

ve”

“ade

quat

e”or

“abo

ve”

(tot

alre

spon

ses)

(tot

alre

spon

ses)

Rea

soni

ngw

ithFi

ndin

gan

dca

tego

rizi

ngre

leva

nt14

2(1

67)

8567

(114

)59

info

rmat

ion

info

rmat

ion

Com

pari

sons

178

(282

)63

89(1

93)

46Pr

edic

tions

135

(217

)62

76(1

53)

50Fo

rmul

atin

gor

iden

tifyi

nga

rese

arch

83(1

07)

7851

(71)

72qu

estio

nSu

ppor

ting

apr

edic

tion

43(1

15)

3728

(77)

36T

otal

s58

1(8

88)

6531

1(6

08)

51C

omm

unic

atio

nO

rgan

izat

ion

85(1

15)

7450

(77)

65A

rgum

ent

43(1

15)

3728

(77)

36T

otal

s12

8(2

30)

5678

(154

)51

Tec

hnol

ogy

use

Sear

chin

gfo

rin

form

atio

non

the

Web

147

(198

)74

–a–a

Form

ulat

ing

orid

entif

ying

ase

arch

quer

y64

(102

)63

–a–a

Tot

als

211

(300

)70

–a–a

aT

echn

olog

yus

eou

tcom

eda

taw

ere

not

gath

ered

from

non-

Wor

ldsc

hool

s.

ARTICLE IN PRESS16 R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

Table 6Impact of the World Links program on teachers as reported by various groups

Impact of the program on teachers’ skills, abilities, knowledge, and Percentage of each group that responded “very much” a

attitudes:

World Links World Links World Linksteachers 1999 administrators 1999 teachers 2000

How to use computer hardware 70 82 76How to use software applications 67 67 71How to use the Internet 73 80 72How to develop Web pages 47 53 50How to use student groups in teaching 63 60 63How to design and lead collaborative student projects 68 58 63How to design and use student assessment materials 46 35 45How to collaborate with other teachers in developing materials 52 44 46How to integrate computers into the curriculum 39 40 53Attitudes about technology 72 85 84Attitudes about teaching 75 90 82

a The response options provided to respondents were: not at all, somewhat, very much.

sample of teachers in many more countries thatparticipated in the program during the second yearof the evaluation. Even more important, WorldLinks students performed better than non-WorldLinks students on direct assessments of ICT skillsand their use to reason with information and com-municate ideas. Teachers too acquired significantICT and pedagogical skills as a result of their par-ticipation in the program. All these results, takentogether, affirm the positive—although still pre-liminary—effects of the World Links program.

4.4. Barriers and differences between countries

While the primary focus of the evaluation wason program practices and outcomes, we felt thatan assessment of the problems and barriers thatconfronted teachers would provide information forWorld Links program improvement and for the for-mulation of national policies that would increasethe effectiveness and impact of such ICT-basedprograms and investments.

Teachers were asked about the reasons that theymay have used computers in their teaching lessthan they had planned. Of the 83 WorLD teacherswho responded to the survey in 1998–1999, 16 (ornearly 20%) indicated that they had not yet

implemented computer-related activities with theirstudents, and 53 (or 64%) mentioned that they hadencountered one or more problems that reducedtheir ability to use computers in their classes.

Among the 16 teachers who said they were notable to implement computer-related activities, themost often cited major barrier was not the lack ofequipment but the lack of time; 62% of these tea-chers indicated that, given curriculum and testingrequirements, difficulty in finding time for com-puter activities was a major barrier (Table 7). Simi-larly, many of these teachers said that finding timeto complete computer activities within the school’sdaily schedule (59%) and finding time in their ownschedule to prepare for implementing technologyin their classes (56%) were major barriers. Relatedto these issues, 42% said that a barrier was the lackof a national policy on the use of computers inschools. Overall, relatively few of these teachersidentified infrastructure problems, such as the lackof computers in working condition, unreliable elec-tricity, or lack of access to the Internet, althoughthese varied by country, as described below.

The 53 teachers who said they were able toimplement computer-related activities butimplemented fewer than planned responded simi-larly. Of these teachers, 51% indicated as a major

ARTICLE IN PRESS17R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

Tab

le7

Bar

rier

sto

impl

emen

ting

com

pute

r-re

late

dcl

assr

oom

prac

tices

To

wha

tex

tent

wer

eth

efo

llow

ing

barr

iers

:Pe

rcen

tage

ofea

chgr

oup

who

resp

onde

d“m

ajor

barr

ier”

a

Wor

ldL

inks

teac

hers

who

Wor

ldL

inks

teac

hers

Wor

ldL

inks

did

not

yet

impl

emen

tw

hodi

dim

plem

ent

teac

hers

2000

com

pute

rs19

99co

mpu

ters

1999

Lac

kof

com

pute

rha

rdw

are/

soft

war

e32

2960

Lac

kof

com

pute

rsin

good

wor

king

cond

ition

3632

56L

ack

ofre

liabl

eel

ectr

icity

2418

18L

ack

ofad

equa

tete

leph

one

line

orse

rvic

esfo

rac

cess

toth

eIn

tern

et38

5352

Lac

kof

inte

rnet

acce

ssfo

rot

her

reas

ons

3124

52L

ack

ofth

ein

suffi

cien

ttr

aini

ngfo

rus

ing

com

pute

rsan

dso

ftw

are

3224

23L

ack

ofor

insu

ffici

ent

tech

nica

lsu

ppor

tfo

rus

ing

com

pute

rsan

dso

ftw

are

3124

23L

ack

ofor

insu

ffici

ent

trai

ning

orsu

ppor

tfo

rin

tegr

atin

gco

mpu

ters

into

the

curr

icul

um54

3938

Lac

kof

orin

suffi

cien

ttr

aini

ngor

supp

ort

for

usin

gco

mpu

ters

inco

llabo

rativ

est

uden

t38

2235

proj

ects

Dif

ficul

tlyfin

ding

time

topr

epar

efo

rim

plem

entin

gte

chno

logy

inyo

urcl

assr

oom

orat

your

5643

44sc

hool

Dif

ficul

tyco

mpl

etin

gco

mpu

ter

activ

ities

with

inth

esc

hool

’sda

ilysc

hedu

le59

4251

Dif

ficul

tyfin

ding

time

for

com

pute

rac

tiviti

es,

give

not

her

curr

icul

uman

dte

stin

g61

5149

requ

irem

ents

Com

pute

rac

tiviti

esdo

not

mat

chw

ell

with

your

inst

ruct

iona

lgo

als

and

met

hods

1510

10C

once

rnab

out

whe

ther

tech

nolo

gyw

ould

beva

luab

leto

your

stud

ents

210

7C

once

rnth

atus

ing

tech

nolo

gyin

your

scho

olw

illaf

fect

the

stab

ility

ofyo

urcu

rren

tjo

b4

04

Lac

kof

acce

ssto

exis

ting

com

pute

rs12

1220

Lac

kof

supp

ort

from

othe

rco

lleag

ues

2220

15L

ack

ofad

min

istr

ativ

esu

ppor

tin

scho

ol37

1715

Lac

kof

ana

tiona

lpo

licy

onth

eus

eof

com

pute

rsin

scho

ols

4229

36

aT

here

spon

seop

tions

prov

ided

tore

spon

dent

sw

ere:

not

aba

rrie

r,m

inor

barr

ier,

and

maj

orba

rrie

r.

ARTICLE IN PRESS18 R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

barrier the difficulty in finding time for computeractivities, given other curriculum and testingrequirements; 43% cited the difficulty in findingtime to prepare for implementing technology intheir classes; and 41% mentioned the difficulty infinding time within the school’s daily schedule. Aswith WorLD teachers who were not able to usecomputers at all, relatively few computer-usingteachers identified infrastructure problems, such asthe lack of computers in working condition, unre-liable electricity, or lack of access to the Internet,per se. However, 53% did say that a major barrierto using the computer as much as they had plannedwas the lack of adequate telephone lines for usewith the Internet. Indeed, this was the most fre-quently cited major barrier for this group of teach-ers.

There were differences among the five countriesin the kinds of barriers that teachers encounteredin the 1998–1999 study. Teachers in Senegal(53%) and Chile (90%) most often identified thelack of time to integrate ICT into the curriculumas a barrier. Teachers in Senegal (53%) also mostfrequently identified lack of computers in goodworking order. On the other hand, teachers inUganda (78%) and Paraguay (60%) most oftenidentified inadequate telephone or Internet serviceas a barrier. Finally, 67% of the teachers in Peruidentified the lack of training and support for inte-grating computers into the curriculum, the barriermost often mentioned by teachers in this country.

In contrast with the 1998–1999 teachers, WorldLinks teachers from 12 countries who respondedto 1999–2000 survey were most likely to identifythe lack of computer hardware (60%), software(56%), and reliable Internet connections (52%) asmajor barriers to their use of ICT in their classes.These findings suggest that as the World Linksscaled up the program, the organization wasinvolving countries and schools within countriesthat were less technologically prepared and in moreneed of assistance. But even in these schools, theteachers agreed with their colleagues in the 1998–1999 study that a major barrier was finding time inthe school’ daily schedule to implement computer-related activities in their classes (51%) and thatthey had difficulty finding time given curriculumand other testing requirements (49%).

There were also differences among countries inthe 1999–2000 study. Teachers in Mauritania(100%), Zimbabwe (67%), Ghana (74%), Chile(59%), and Peru (53%) most often mentioned thelack of hardware and software among the barrierslisted in Table 7. All the teachers in Mauritaniaalso mentioned the lack of computers in workingorder as a barrier. Inadequate phone lines or otherInternet connection problems were most oftenmentioned by teachers in Brazil (88%), Colombia(65%), Mozambique (62%), and Paraguay (62%).Teachers in Senegal mentioned the lack of trainingand support for integration into the curriculum asa barrier; 78% of them mentioned this problem.And teachers in South Africa (62%) and Uganda(55%) were most likely to mention lack of time tointegrate computers into the curriculum as a bar-rier.

The barriers to teachers’ use of ICT were some-times significant. However, it is also important toidentify those factors that teachers did not consideras barriers. As important as those things that teach-ers considered to be barriers are those they did not.Regardless of country or year of the survey, fewof the World Links teachers felt that computers didnot match their instructional goals or methods (ascontrasted to curricular goals). Few were con-cerned that the use of computers would have novalue for their students. And very few were con-cerned that the introduction of computers inschools would affect their jobs status or stability.

5. Summary and conclusions

The findings from this study indicate that whentechnological infrastructure is developed in con-junction with appropriate teacher training, signifi-cant educational change can be achieved indeveloping countries. Despite the identified bar-riers and problems encountered by World Linksteachers, we can infer from our data that studentsin World Links schools were more likely thancomputer-using students in non-World Linksschools to use a variety of technologies, such asemail, search engines, and the Internet. They werealso more likely to engage in classroom practicesthat are often cited as important for preparing stu-

ARTICLE IN PRESS19R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

dents for the global knowledge economy, activitiessuch as gathering data for a research project, col-lecting information about another country or cul-ture, and collaborating on a project with studentsfrom another country.

By all indications, students not only engaged inthese practices but acquired the skills that they willneed to enter the global workforce. Students, teach-ers, and administrators agreed that the World Linksprogram contributed to student outcomes such asimproved skills in reasoning with information,communication skills, knowledge of other cultures,better attitudes toward school and technology, and,of course, improved technology skills. Further-more, teachers and students (in the case of the1998–1999 study) in the World Links programwere more likely than those in non-World Linksclassrooms to say that students improved theircommunication skills, knowledge of currentevents, knowledge of other cultures, collaborationskills, and Internet skills. Finally, and mostimportantly, World Links students scored higherthan non-World Links students in direct assess-ments of reasoning with information and com-munication skills.

Our findings support the argument thatdeveloping countries can benefit from investmentsin equipment, software, and network infrastructure.The findings also demonstrate the benefit of train-ing teachers not only in the operation of hardwareand software but in the integration of technologyinto classroom instruction. Finally, the findingsdemonstrate that they payoff of these investmentsis increased skills of both students and teachers.Taken together, these findings support the value ofthe World Links program and suggest that WorldLinks and other programs which aim to build thetechnological infrastructure and teacher skills indeveloping countries can contribute to theimprovement of education and reduction of thedigital and perhaps economic divide betweendeveloping and developed countries.

But the barriers cited by World Links teachers,as well as findings from other studies (Means andOlson, 1995; Sandholtz et al., 1997; Means et al.,2001; Schofield and Davidson, 2002; Kozma,2003) suggest that there are other actions and poli-cies that if employed could increase the impact of

World Links and other such programs. On the tech-nology side, better ways are needed to connectschools in developing countries to the Internet,especially low-cost, high bandwidth solutions thatmight benefit schools in remote areas that do notcurrently have reliable phone lines.

But the most-often cited barriers were not tech-nological. World Links teachers consistently citedthe lack of time for computer activities, given cur-riculum and examination requirements, the lack oftime in the school day, and the lack of time forpreparation. These findings suggest that teachers indeveloping countries are encountering the samebarriers as teachers in developed countries are(Means and Olson, 1995). In turn, these time-related barriers suggest an underlying lack of pri-ority for the use of ICT in current national andlocal policies. The lack of national ICT policy wasalso a barrier often cited by teachers. This barrieris most vividly illustrated by an incident during afield site visit by one of the authors (Kozma) to aWorld Links school in Uganda. When asked whya computer lab was empty during the school day,even though it was packed after classes, the teachersaid that there was little fit between the use of com-puters and the national curriculum and examinationsystem in Uganda. The use of computers was notin the curriculum nor would it be tested on thenational exam. Consequently, it was difficult forthe teachers of this school to justify using com-puters during regular school time.

Developing countries can draw on the experi-ences of other countries where ICT has come tohave a more important role in education. In a grow-ing number of developed countries, national poli-cies identify a high priority for the use of ICT ineducation; see for example Singapore (Ministry ofEducation and Singapore, 2002), Finland (Ministryof Education and Finland, 1999), and US (USDepartment of Education, 2000). These policiesand priorities are often expressed as an educationtechnology master plan. These plans provide avision for the use of technology in education andinstitute programs to support the realization of thisvision (Jones, 2003). To maximize the impact ofinvestments in educational ICT, the ICT policyneeds to be coordinated with the policies of otherministries (such as telecommunications, human

ARTICLE IN PRESS20 R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

resources, and science and technology) and withother policies within the ministries of education onmatters related to teacher training, curriculum,and assessment.

Teacher training—both in-service and pre-ser-vice—needs to incorporate the use of ICT in class-room teaching. World Links teachers identified thelack of sufficient training and support for the inte-gration of computers into the curriculum as anotherbarrier. If computers are going to have a significantimpact on the learning of students, it will bebecause they are integrated into the curriculum, notjust as a separate subject such as computer literacyor keyboarding, but as tools to support the learningof science, mathematics, social studies, language,and creative arts. In the work world, technology isdramatically changing the way these school sub-

Box 1. Selected Examples of WorLD School Projects on Environmental Issues

Ciencia a Conciencia (Conscientious Science)Participating schools in: Paraguay, MexicoDescription: Students learn collaboratively about various environmental issues including: defor-estation; pollution; ozone layers; toxic wastes. They share results of findings and explore sol-utions to these problems. Findings from the collaborative learning activities are posted on theWeb site.Web site:

http://www.enlaces.edu.py/cndelm/131198/circulos.htm

Wetlands Project

Participating schools in: Uganda, Australia, USADescription: The project brings together students around the theme of learning about and pro-tecting local wetlands and waterways. Reports are prepared by students on: wetland conditions;legislation to protect wetlands; public awareness raising on the impact of human activities onthe ecology; water levels and water quality of wetlands/waterways; and steps to take to ensurefuture protection of wetlands. Findings are shared among the participating students on the web-sites.Web sites:

http://www.viser.net/gs21/ugdepletion.htm (school)

http://www.camperdowncoll.vic.edu.au/global/wateryworlds/wetlands.htm (project)

Flora y Fauna

Participating schools in: Peru, Chile, Brazil, Argentina, U.S.A.Description: An interdisciplinary project where students learn about “Flora and Fauna” fromvarious approaches including: mathematics (statistical/geometric/measurement aspects); litera-ture (metaphoric use of plants & animals in literature); language (translation).Web site:

http://www.geocities.com/Athens/Atlantis/6126/proyecto.htm

jects are applied to solve everyday problems. Theintegration of ICT into the learning of these schoolsubjects can make a significant contribution to thepreparation of students in developing counties forparticipation in the global knowledge economy andinformation society (21st Century Partnership,2003; ISTE, 2000).

Classroom integration will depend on the inte-gration of ICT into the formal national curriculumand testing program. Policies in an increasing num-ber of developed countries (Jones, 2003) areincorporating the use of ICT throughout the cur-riculum as part of an explicit effort to reform edu-cation and improve student learning. Often thesechanges are coordinated with student-centeredpedagogies. Some countries, specifically Singaporeand Norway, are beginning to review their

ARTICLE IN PRESS21R. Kozma et al. / International Journal of Educational Development XX (2004) XXX–XXX

approach to national assessments and examinationsso as to incorporate the use of and learning fromICT.

If developing countries set their national pri-orities to emphasize the use of ICT in educationand these are coordinated with other policiesrelated to teacher training, curriculum, and assess-ment, the foundation will exist for teachers to inte-grate computer activities into their daily practice.This in turn will prepare students for the globalknowledge economy and contribute significantly toclosing the digital divide between developed anddeveloping countries. Without such national poli-cies, priorities, and resources, in all likelihood thedigital divide will continue to grow.

References

21st Century Partnership 2003. Learning for the 21st Century.21st Century Partnership, Washington, DC.

Bridges.org. 2001. Spanning the Digital Divide: Understandingand Tackling the Issues. Bridges.org, Washington, DC.

Carlson, S., 2002. The missing link in educational technology:trained teachers. TechKnowLogia 4 (6), 7–11.

Educational Testing Service (ETS) 2002. Digital Transform-ation: A Framework for ICT Literacy. ETS, Princeton, NJ.

International Society for Technology in Education (ISTE) 2000.National Educational Technology Standards for Students:Connecting Curriculum and Technology. InternationalSociety for Technology in Education, Eugene, OR.

Jones, R., 2003. National and local policies. In: Kozma, R.(Ed.), Technology, Innovation, and Educational Change: AGlobal Perspective. International Society for Technology inEducation, Eugene, OR, pp. 163–194.

Kozma, R., 2003. Technology, Innovation, and EducationalChange: A Global Perspective. International Society forTechnology in Education, Eugene, OR.

Kozma, R., McGhee, R., 1999. World Links for Development:Accomplishments and challenges. Monitoring and evalu-ation annual report: 1998–1999. SRI International, MenloPark, CA.

Means, B., Olson, K., 1995. Technology’s Role in EducationReform: Findings from a National Study of InnovatingSchools. Washington, DC, US Department of Education,Office of Educational Research and Improvement.

Means, B., Penuel, W., Padilla, C., 2001. The ConnectedSchool: Technology and Learning in High School. Jossey-Bass, San Francisco.

McGhee, R., Kozma, R., 2000. World links for Development:Accomplishments and challenges. Monitoring and evalu-ation annual report: 1999–2000. SRI International, MenloPark, CA.

Ministry of Education, Finland 1999. Education, Training andResearch in the Information Society: A National Strategyfor 2000–2004. Ministry of Education, Olso, NO.

Ministry of Education, Singapore, 2002. Masterplan 2(http://www.moe.edu.sg/edumall/mp2/mp2—overview.htm).

National Center for Educational Statistics (NCES) 2001. TheNation’s Report Card: Science 2000. National Center forEducational Statistics, Washington, DC.

Organization for Economic Co-operation and Development(OECD) 1996. The Knowledge-Based Economy. OECD,Paris.

Organization for Economic Co-operation and Development(OECD) 2000. Literacy in the Information Age. OECD, Paris.

Organization for Economic Co-operation and Development(OECD) 2001a. Education Policy Analysis. OECD, Paris.

Organization for Economic Co-operation and Development(OECD) 2001b. Understanding the Digital Divide. OECD,Paris.

Pelgrum, W., Plomp, T., 2002. Indicators of ICT in mathemat-ics: status and covariation with achievement measures. In:Beaton, A.E., Robitaille, D.F. (Eds.), Secondary Analysis ofthe TIMSS Data. Kluwer Academic Press, Dordrecht.

Quellmalz, E., Kozma, R., 2003. Designing assessments oflearning with technology. Assessment in Education 10 (3),389–407.

Quellmalz, E.S., Zalles, D., 1999. World Links Student Assess-ment Report, 1998–1999. SRI International, Menlo Park, CA.

Quellmalz, E., Zalles, D., 2000. World Links for Development:Student Assessment Uganda Field Test, 2000. SRI Inter-national, Menlo Park, CA.

Sandholtz, H., Ringstaff, C., Dwyer, D., 1997. Teaching withTechnology: Creating Student-Centered Classrooms. Teach-ers College Press, New York.

Shavelson, R., Towne, L., 2001. Scientific Inquiry in Education.National Academy Press, Washington, DC.

Schofield, J., Davidson, A., 2002. Bringing the Internet toSchool: Lessons from an Urban District. Wiley, New York.

Tashakkori, A., Teddlie, C., 1998. Mixed Methodology: Com-bining Qualitative and Quantitative Approaches. Sage,Thousand Oaks, CA.

United Nations Development Programme 2001. Human Devel-opment Report 2001. UNDP, Paris.

United Nations Development Programme 1999. Human Devel-opment Report 1999. UNDP, Paris.

US Department of Education 2000. E-learning: Putting aWorld-Class Education at the Fingertips of all Children. USGovernment Printing Office, Washington, DC.

Wenglinski, H., 1998. Does it Compute? The RelationshipBetween Educational Technology and Student Achievementin Mathematics. ETS, Princeton, NJ.

World Bank 2002. The Education Sector Strategy Report.World Bank, Washington, DC.

World Bank 1999. The Education Sector Strategy Report.World Bank, Washington, DC.

World Bank 1997. The Economic Development InstituteAnnual Report. World Bank, Washington, DC.