Higher Education 5(1976) 253-275 �9 Elsevier Scientific Publishing Company, Amsterdam - Printed in the Netherlands

EVALUATION CRITERIA AND THE EFFECTIVENESS OF INSTRUCTIONAL TECHNOLOGY IN HIGHER EDUCATION*

STUART WELLS

Assistant Professor, School of Business, San dosd State University, San JosO, California

ABSTRACT

"No significant differences" is the most frequently quoted conclusion of any survey comparing the effectiveness [1] of alternative instructional technologies [2,3]. While this effectiveness conclusion does not differ for the research reported upon in this paper the other purposes of this paper are to discuss alternative criteria for measuring effectiveness and to analyse potential causes of the finding of no significant differences.

The paper is divided into three major sections. In the first section an analysis of the impact of comparability of technologies, organizational factors, student characteristics, and material quality on technology effectiveness is presented. The second section is devoted to a description of cognitive (measurements of knowledge gains) and non- cognitive (attitudes and time to complete material) criteria for evaluation. The third section contains a summarization of research findings for a variety of technologies including: television, radio, programmed instruction, computer-assisted instruction and effects of alterations in class size, changes in course length, and utilization of graduate students as instructors.

P rob lems in the Eva lua t ion o f In s t ruc t i ona l Techno log ies

COMPARABILITY OF TECHNOLOGIES

To de t e rmine the relat ive cogni t ive ef fec t iveness of ins t ruc t iona l tech-

no logy c o m p a r i s o n s are usual ly m a d e wi th conven t iona l ins t ruc t ion . Un-

f o r t u n a t e l y there is no clear def in i t ion o f conven t iona l ins t ruc t ion . I t is also

diff icul t to spec i fy precise e x p e r i m e n t a l cont ro ls for the studies. F o r

example , in a c o m p a r i s o n o f televised and conven t iona l c lass room instruc-

t ion, should we use the same ins t ruc to r wi th the same mater ia ls in b o t h

media? Or should we al low to ta l f lexibi l i ty in course cons t ruc t ion to util ize

the d i f fe rences in the med ia? What size o f class should be a l lowed for the

conven t iona l i n s t ruc t ion a l ternat ive? Briggs et al. (1967) po in t ed to o the r

diff icul t ies in c o m p a r i s o n studies including: the use o f in tac t classes and the

+'~ Prepared for the Economics Education Project, Heriot-Watt University, Edinburgh, Scotland. The author would like to thank the referees for Higher Education for their helpful comments on an earlier draft of this paper.

254

lack of r a ndom assignment to t r e a tmen t groups; the use o f investigator-

cons t ruc ted evaluat ion tests and the subsequent lack o f a t t en t ion to reliabili- ty and validi ty; the possibil i ty tha t the test was no t power fu l enough to

uncover differences; and the f requen t lack o f con t ro l for t eacher effects. F o r example, Costin et al. (1971) in a survey o f s tuden t ratings r epor t ed tha t the mos t f requen t ly ci ted criteria of teacher success were: knowledge level o f teacher ; organizat ion o f course con ten t ; interest in s tudents and course; and, abil i ty to motivate .

One might feel tha t a successful use o f t echno logy would take advan- tage o f the unique possibilities o f the various media (e.g. color television, animat ion) . However , in analyzing a var ie ty o f program formats for instruc-

t ional television Chu and Schramm (1967, p. 180)gene ra l ly conc luded that:

. . . effective use of television grows out of attention to the basic require- ments of good teaching, rather than to any fanciness that might be peculiar to television... . . . qualities like simplicity, good organization, motivation, practice, knowl- edge of results, rest pauses at appropriate points, cues that direct the pupil to the essential things he is to l ea rn . . .

One poten t ia l exp lana t ion for the d i f f icul ty in compar ing technologies may be the relative advantage o f one t echno logy for some e lements o f instruct ion. As Briggs et al. (1967, p. 24) s tated:

When a lengthy course or sequence, representing several kinds of learning, is prepared in two different media and the results analyzed, the most frequent result is a failure to demonstrate a significant difference. One reason for such a finding could be that each of the media compared was more effective for some elements of instruction and less effective for other elements, so that the differences in effectiveness among media were cancelled in the overall~ analysis.

The op in ion tha t new research should be d i rec ted toward uncover ing specific instances for which a par t icular t e chno logy might be significantly effect ive is no t new.

The problem that must be solved is not the question 'Is Method A better than Method B or Method C?' but rather, 'What are the conditions under which Method A produces more effective results? What are the situations where Method B is best, etc.'

Spence (1928, p. 462)

In quot ing this remark Dubin and Taveggia (1968, p. 17) fu r the r conc luded : " I t is no tab le tha t this ques t ion persists t h rough the l i terature and always remains unanswered in te rms o f empir ical da ta . "

255

The elements of instruction may be related to the type of learning. For example, Gagn6 (1965) has theorized that learning proceeds according to a hierarchy of learning tasks from signal learning (a diffuse response to a signal) to problem solving (the ability to use two or more chains of concepts to produce a new calSability that can be shown to depend on a "higher- order" chain). In a variety of experiments (Gagn6, 1962; Gagn6 and Paradise, 1961 ; and Gagn6 et al., 1962) the advantage of dividing a task into a hierarchy of learning tasks was demonstrated although the research did not deal with alternative technology presentations.

Bloom's (1956) taxonomy of learning may also be used to determine relative advantages of technology. His taxonomy is divided into six basic categories moving from knowledge (recall ability), the lowest level, to evaluation (ability to assess value of materials), the highest level.

ORGANIZATION OF PROJECT

Other favorable conditions which might insure the successfulness of a technology project might relate to the organization of the project. Armsey and Dahl (1973) stressed organizational objectives such as the existence of a recognized and generally agreed-upon need and the desire to meet the need through the use of technology; the participation and support of teachers in the project; and adequate resources for the duration of the project. Briggs et al. (1967) suggest four basic steps in the planning of a course using multi- media presentation:

1. Identify each belaavioral objective. 2. Relate behavioral objective to type of learning (according to Gagn6's

hierarchy). 3. Choose media which provide the best stimuli for types of learning. 4. Look at overall choices to determine most effective sequencing for

multi-media use.

Bretz (1971) mentioned several objectives for determining appropriate technology including: motivating learner; reinforcing learning; stimulating discussion; providing for drill and practices; and inducing teacher-learner interfaces.

The importance of stating instructional objectives cannot be under- estimated. This step is critical in evaluating the relative effectiveness of instructional technologies. For example, Forsythe (1970) mentioned a highly successful creative arts program which was originally broadcast on radio and later transferred to television. The program was returned to radio when it was discovered that its effectiveness had diminished since students were copying the artist. It is evident in this instance that creativity was the

256

instructional objective. Had the objective been technical competence in drawing the television program might have been deemed the more effective medium.

Without a careful rereading of each individual research study or dis- cussions with researchers it is extremely difficult to isolate particular organizational structures which led to successes or failures in various in- stances. This difficulty might be evidenced by taking Dubin and Hedley's (1969, p. IX) conclusion regarding two-way television at face value. The general result of their survey of instructional television in higher education was that conventional instruction was superior to instructional television.

Overall the results appear to favor face-to-face instruction. However, when we introduce the appropriate controls on variation in the television medium by parceling out the effects of one-way and two-way television instruction, the differences favoring face-to-face instruction are wiped out. It is two-way television instruction which is markedly inferior to face-to-face instruction!

This conclusion is frequently mentioned in their s tudy to support a continuing discussion on the merits of two-way (one-way video with audio talk-back facilities) television and the importance of not having one medium presentation follow another medium presentation. However, Dubin and Hedley point out that virtually all of the studies on two-way television came from one research project (Los Angeles City SchoolDistr ict , 1959). Since in describing the statistical procedures employed in this project they stated: "We are satisfied that this is one of the best studies in the literature from a methodological standpoint and that the only influence of this s tudy has been to contribute data in which we have a high level of confidence" (p. 10), one concludes that there might well be factors in the organization of this project which contr ibuted to its obvious failure. Other studies of a two-way tele- vision system (discussed in the section on effectiveness) reached conclusions of no significant differences between this medium and conventional instruc- tion.

QUALITY OF MATERIAL

The quality of the instructional material is also an important dimension in measuring the success of a given project. Chu and Schramm (1967) concluded (on the basis of research by Gropper and ~umsdaine, 1961; and Gropper et al., 1961) that showing, testing, and revising an instructional program would substitute for the lack of feedback and improve the quality of a televised course. Wells (1973) in .a survey of economics education discussed two studies on programmed instruction which demonstrated the need to account for differences in quality. Fels and Starleaf (1963) in a

257

comparison of lectures and programmed instruction reported significant differences favoring the lecture students in their first experiment but no significant difference in the second experiment when the programmed materials were revised. Attiyeh et al., (1969) conducted one of the largest studies of programmed instruction to date involving over 4100 students at 48 colleges. Two different programmed books were used and the students were divided into three groups: programmed book only, conventional instruction only, and conventional instruction supplemented by the pro- grammed book. Students using one of the programmed books as the only source of instruction performed significantly worse than all other groups, indicating a difference in quality of programmed material. Wells (1974b) found significant differences in achievement of economics students in the United Kingdom depending upon the textbook which they used indicating a difference in quality.

STUDENT CHARACTERISTICS

There is a possibility that instructional technology has differential impacts on students. These differential impacts might be determined by certain characteristics of the learner such as, basic intelligence or ability, initial knowledge of the subject, family background, or personality. If we had been comparing two technologies and randomly assigned students to groups we would not be concentrating on an analysis of these relationships. Unfortunately, the results of research studies in this area do not reveal any consistent findings and we have not yet reached the point of recommending specific instructional technologies for specific students.

Kelley (1972) determined that his system of computer-managed in- struction had a greater impact on lower ability students. For televised instruction Dreher and Beatty (1958) found significantly higher scores for low-ability and high-ability students receiving televised instruction and no significant differences for medium-ability levels. Wells (1974b) found no difference in the effects of a variety of course and instructor qualities on the performance of college economics students between students who had advanced secondary economics and students who did not.

Therehave been several interesting studies analyzing the r e l a t i onsh ipo f personality types and performance on programmed instruction. Doty and Doty (1964) used a programmed unit for psychology students and found a significant negative relationship between social need and creativity and performance on programmed instruction. There was no significant relation- ship with achievement need.

Lublin (1965) reported that programmed instruction in introductory psychology was more effective for students with a high need for autonomy than for students with a lower need.

258

Kight and Sassenrath (1966) reported that students with a high level of achievement motivation completed a program in less time and with fewer errors and had higher scores on a short-term retention test. There were no significant differences on a long-term test of retention.

MacPherson (1967) reported that persons with a high anxiety level required significantly less time to complete a program in mathematics.

Shrabel and Sassenrath (1970) performed an experiment with college upperclassmen. They defined an easy program as one in which the probabili- ty of success of a response was nearly 100%. A hard program had a probability of success of less than 50%. Basing their results on quicker completion rates, lower error rates, and higher retention rates they found that easy programs were better for persons with low need for achievement and high fear of failure. Hard programs with longer steps [4] were better for persons with high need for achievement and low fear of failure. They also concluded that the easy program provided insufficient motivation for students with high need for achievement.

As these studies indicate, there is a definite advantage to analyzing the relationship between student characteristics and type or presentation of technology to improve the effectiveness of the technology.

With respect to students, perhaps one of the most important reasons for findings of no significant differences has been ignored: the allocation decisions of students with respect to their own time. If the student deter- mines a deficiency in his classroom instruction (regardless of the technology used) he may decide to compensate by increased studying and reduced attendance. Wells (1975) in a study of economics education in the United Kingdom found tha t students in classes with lower-quality lecturers (measured by status and experience) tended to attend fewer classes and study more hours than students in classes with better lecturers. Students in lectures with higher enrollments studied more often and attended fewer lectures. As this study indicates, a lack of control or information regarding student activities and decisions may lead to a conclusion of no significant differences when in fact there are very real differences in the productivity of two instructional methods or technologies.

Evaluation Criteria

COGNITIVE MEASURES

The most frequently used measures for evaluating the relative effec- tiveness of instructional technologies have been internally generated tests (usually final examinations and frequently multiple-choice questions) at

259

higher education level and standardized achievement tests at the elementary and secondary levels. For example, Dubin and Taveggia (1968) only included studies which utilized a final examination to measure the relative effec- tiveness of a variety of teaching methods. They justified this criterion since educators are very willing to use final examination scores to judge student performance. Anderson and Greenburg (1972, p. 7) made the same comment regarding achievement scores:

Educators generally seem qui te willing to permit decisions about s tudents ' lives to be made on the basis of "unsa t i s fac to ry" achievement measures but are unwill ing to permit those same measures to be used as a basis for evaluating their own performance.

This is ironic, since the tests are much more sat isfactory and reliable as a basis for group-to-group comparisons where standard errors are low than for judgements about individual s tudents where errors of measurement are ex t remely high.

However, there are definite disadvantages to the use of final examina- tions. As mentioned previously, Briggs et al. (1967) pointed out thattlaese examinations are often not tested for reliability and validity and are often not powerful enough to uncover differences. Additionally, since the test is a final examination and used for grading, the student, as pointed out by Wells (1975), is likely to increase his study time if he feels that the presentation method is inadequate.

It is also not evident from a simple analysis of research results which type of learning the final examination was structured to test. Bloom's (1956) taxonomy of learning or GagnCs (1965) hierarchy of learning may be used to help determine the level of instruction which is achieved.

Most examinations probably test learning through the application level only. For example, several multiple-choice examinations are commonly used in economic education (Test of Understanding of College Economics and Test of Economics Comprehension). These tests have been formulated by panels of distinguished economists and have questions to test certain abilities: (1) recognition and understanding - questions which could be answered from a textbook, (2) simple application - the ability to use some economic principle or concept which is clearly specified, and (3) complex application - the ability to use some economic principle or concept which is not clearly specified or to combine more than one concept in a new manner. Lewis and Dahl (1971) tested the validity of one of these examinations and found a higher correlation with critical thinking (as measured by t h e Watson-Glaser Critical Thinking Appraisal Test) and simple application problems than with complex application problems.

260

NONCOGNITIVE MEASURES

Noncognitive measures may be used to evaluate the effectiveness of a given technology. Kelley (1972) found that students in a section using computer-managed instruction had a 23% higher rate of continuation than students in a conventional class for college economics. Suppes and Morning- star (1969) reported that the dropout rate for students using computer- assisted instruction was less than half t h e dropout rate of students in a conventional course for introductory Russian at the college level. Attiyeh et al. (1969) reported that students using a programmed book learned as much economics with only 12 hours of studying as students in a lecture section during 7 weeks (approximately 40-60 hours of instruction and studying). Wells (1974b) regarded the students' opinions about the "usefulness" of economics as an output and, using multiple regression analysis found that opinions and cognitive achievement were affected by different qualities of the instruction.

Krathwohl et al. (1964) delineated a taxonomy in the affective domain which is structured in a manner similar to Bloom's (1956) cognitive taxono- my. The affective taxonomy tends to be more ambiguous than the cognitive taxonomy but the authors do suggest test items. The taxonomy is quite complex but was structured in a manner to form a hierarchy of development from receiving (sensitive to phenomena), to consistent behavior with respect to a set of attitudes. It is quite evident that single courses could not encompass the entire taxonomy. Additionally most research in the affective domain has involved attitudes regarding course, teacher, and instructional technology characteristics. The investigator is primarily concerned with gauging student opinions.

The few attempts at experimentation in the affective domain seem to be aimed at changing attitudes. The Wisconsin Research Project in School Broadcasting (1942, p. 90) reported that students in high school receiving social studies courses via radio showed significantly more changes in the direction of the program goals which were to increase:

1. tolerance toward the interests of various economic groups; 2. cooperation with the member of one's own group in solving

common problems; 3. cooperation with other groups in solving common problems; and 4. a sense of responsibility in furthering the interests of one's own

group.

As related work on attitude change a study by Willis (1940) could be cited. Willis compared the relative effectiveness of three forms of radio presentations: straight talk, dramatization, and a combination of talk and

261

dramatization. The subject areas treated were: the treatment of criminals, freedom of speech and press, and the attitudes students should hold toward the German people. An attitude test was administered to high school and college students prior to the program, at the conclusion of the program, and after two weeks. There were nonlistening control groups for each education level.

The following general conclusions were noted:

1. A 15-minute radio program significantly shifted the attitudes of high school and college students;

2. The attitude changes were still significant after two weeks al- though there was a tendency for the attitudes of the high school students to return to original attitudes;

3. For high school students the dramatization was more effective, the combined form next, and the talk the least effective in changing attitudes (all methods produced significant attitude changes);

4. For college students all three methods were equally effective.

Radio has also been successfully used to inculcate socially desired values in Thailand (Jamison et al., 1974) and Korea (Forsythe, 1970).

At a more mundane level, Lloyd (1970) used role playing to teach a course in labor-management relations. He tested students' attitudes before and after the role playing and found a greater post-test association with attitudes associated with the role played.

Effectiveness of Alternative Technologies

The relative effectiveness of technologies is now examined. Most of the studies reported below utilize some test of knowledge. Where possible, the potential shortcomings of the research are mentioned in accordance to the discussions in the first section of this paper.

INSTRUCTIONAL METHODS

The most traditional types of variation in instructional technology have been changes in course size, method of instruction, variation in length of the course, and the use of lower status lecturers or discussion section leaders (e.g. graduate students).

The most comprehensive survey of literature on variation in instruc- tional methods is provided by Dubin and Taveggia (1968). They subdivided instructional methods into two basic categories: face-to-face instruction and independent study. Face-to-face instruction included: lecture - lecturer

262

directed class with some questions; discussion - active interchange between teacher and students; a combination of lecture and discussion; and tutorial - direct one-to-one interchange. Independent study included: supervised - regular guidance and instructor provides motivation; and unsupervised - student is given bibliography and materials and proceeds on his own.

From over 300 different comparisons Dubin and Taveggia concluded that there were no significant differences among all of these methods. Interestingly enough, the studies which they surveyed were approximately evenly divided between the superiority of a given method in all of the two-way comparisons. This result indicates that there may be some advan- tage in investigating technologies to determine if there are unique factors which lead to success.

Since lectures are commonly associated with large classes and dis- cussions are associated with small classes these results would seem to indicate that class size does not affect student learning. This result is consistent with the result reported by Jamison et al. (1974) regarding class size at higher education levels. Wells (1973) reported on a number of studies at the higher education level in economics education and concluded that the size of the class was not important. However, Attiyeh and Lumsden (1972) and Wells (1974b) reported negatively significant relations between performance and discussion section class size in economics courses in the United Kingdom in which a combination of lectures and discussions was utilized.

MASTERY LEARNING

Mastery learning is a teaching method in which a subject is subdivided into many smaller units. The subdivision usually involves progression from simpler to more complex tasks but these are not necessarily subdivided according to a taxonomy involving kinds of learning. Peterson (unpublished) surveyed twenty-eight experiments in mastery learning and found twenty- four cases with significant differences in favor of mastery learning.

Baker (1971) surveyed a variety of experiments which involve the basic principles of mastery learning but utilize a computer to analyze student responses to questionnaires submitted after some segment of a course or curriculum. Based upon the responses of the student, the student is directed to certain assignments. However, he did not assess the impact of learning. Kelley (1968, 1972) reported on a system of computer-managed instruction in economics at the college level. Several times during the course of the semester the students received a 10 to 15 item multiple-choice examination. Students were then given a group of assignments (required and optional) based upon their responses. Reports were also prepared for the lecturer and discussion section leader describing student progress and assignments. Students in the section utilizing the computer-managed instruction (TIPS -

263

Teaching Information and Processing System) performed significantly higher on a mid-term examination than students in the conventional section.

There would appear to be some advantage to allowing students to proceed at their own pace and structuring assignments for the ability and knowledge of different students.

LENGTH OF COURSE

The research reported below raises a serious question regarding the planning of course content and course length as it would appear that one cannot necessarily conclude that a longer course would lead t o more learning.

Bach (1970) compared the performance of students in a one-semester experimental course and students in the one-year course of the previous year. No significant differences were reported for the TEC (Test of Economics Comprehension) or a university exam. However, the experimen- tal course contained only twenty-five students per section and the small class size may account for the results.

Klos and Trenton (1969) assigned students to a one- or two-semester course. All topics were covered in the one-semester course and the two- semester course included more depth in the technical and theoretical aspects of economics. They reported no significant difference in total test score although the results varied for subject components of the test. It was not clear in this study if the test which was used would fully test the potential advantage of students in the year-long course.

Paden and Moyer (1971) compared the performance of students in a two-semester videotaped TV course with students in a one-semester course. Both groups had the same lectures but the two-semester students had two discussion sections per week. Regression analysis revealed that the two- semester students scored significantly higher on 189 multiple-choice questions, had better attitudes toward method of instruction and had no difference in attitudes toward the course.

GRADUATE STUDENT INSTRUCTORS

The use of graduate students as instructors for principles courses is a common practice at many universities. McConnell (1968) found no signifi- cant differences in test scores for students in the teaching assistant sections compared to students in videotaped lecture sections or faculty led large or small lecture sections.

However, Lamphear and McConnell (1970) compared the performance of the students in the TA sections in McConnell (1968) with the students in the "lectureless" and videotaped lecture sections in a study by McConnell

264

and Lamphear (1969) and revealed an advantage to the latter two groups of students for the university multiple-choice examination and each of its components. There was no significant difference on the TEU (Test of Economics Understanding).

Oates and Quandt (1970) compared student performance in graduate- student- and faculty-led sections for eight separate semesters at Princeton University. Using the final examination as the measure of performance and regression analysis, they found that students in faculty-led sections were favored in two semesters, students in graduate-student-led sections were favored in two semesters, and there were no significant differences in four semesters. Based on the size of the significant regression coefficients, they concluded that students in faculty-led sections would, in general, be favored.

Saunders (1971) analyzed the performance of students in faculty- and graduate-student-led sections for several semesters at Carnegie. Regression analysis revealed no difference for the TEC (Test of Economics Comprehen- sion) or the TUCE (Test of Understanding of College Economics) and its components, although a comparison of means revealed lower instructor ratings and higher grades for students in the graduate student sections.

The evidence on the effectiveness of graduate student instructors is mixed. While each study found some advantage to students in faculty-led sections the variation in experience and educational level of the graduate students might account for some of the differences. Oates and Quandt used graduate students in their third or fourth year and had regular meetings with all instructors while Lamphear and McConnell evaluated graduate students in their second year with no meetings. Saunders claimed that the situation of graduate student instructors in his study was not as favorable as in the study by Oates and Quandt but more favorable than in the one by Lamphear and McConnell. Again it would appear that we cannot simply examine different technologies without an investigation of other factors which might affect technology effectiveness.

TELEVISED INSTRUCTION

The research into the use of television for educational purposes has been voluminous and there have been several excellent surveys of this literature. As expected, the major conclusion that one reaches is that educational television is as effective as conventional instruction.

Pflieger and Kelley (1961) reported the results of a three-year study which involved over 200,000 students in 800 public schools. The majority of the comparisons resulted in no significant differences; there were significant differences favoring the TV-taught students in 119 cases and the conven- tionally taught students in 44 cases. Kelley (1964) evaluated over 300 matched achievement test comparisons from 1956 to 1961. He found

265

significant differencesin favor of television in 25% of the cases. Schramm (1962) evaluated 393 studies and found no significant differ-

ences in 255 of the studies, significant differences favoring television in 83 cases and significant differences favoring conventional instruction in 55 c a s e s .

Chu and Schramm (1967) evaluated 207 studies which involved 421 separate comparisons. Of these comparisons there were no significant differ- ences in 308 instances. Significant differences favoring televised instruction were found in 63 comparisons and 50 cases favored conventional instruction. Chu and Schramm also analyzed program format and generally concluded that effective television teaching was more a result of attention to the basic requirements of good teaching than of the use of fancy production tech- niques.

Stickell (1963) analyzed 250 comparisons. He applied strict criteria to experimental design and determined that 217 were not interpretable. Of these 217 cases, 59 showed significant differences, evenly divided in favor of televised instruction and of conventional instruction. Twenty-three studies were partially acceptable (usually because of nonrandom assignment) and of these there were significant differences in favor of television in three cases. The ten acceptable studies were all undertaken by Carpenter and Greenhill (1955, 1958) at Pennsylvania State University and resulted in no significant differences.

The criteria were:

1. Experimental and control groups of at least 25; 2. which had been randomly assigned from the same population; 3. were taught by the same instructor - either the two instructors

exchanged classes in the middle of the term or one group was seated in the room from which the class was being televised to the other group; and

4. were evaluated by acceptable statistical procedures.

While Stickell's technique eliminates many of the potential difficulties encountered in comparison studies it does not provide information regarding the potential unique advantages of televised and conventional instruction.

Duhin and Hedley (1969) surveyed research in college instruction by television. In order to be included in their survey the study had to be of an American college course for credit of at least one term duration and to include a report of group mean scores on identical tests for students in a televised course compared with students in a conventional course. On this basis they found 102 studies which favored televised instruction, 89 which favored conventional instruction, and 2 with no significant differences.

As previously mentioned, Dubin and Hedley concluded that conven-

266

tional instruction was superior to two-way television but did not have an advantage over one-way television. Many other studies contradict this result for two-way television and this lends further credence to the claim that the lack of success of two-way television in the Dubin and Hedley study is likely to be attributable to some other deficiency of the project which they analyzed.

Chu and Schramm (1967) concluded that student learning was not impaired by the lack of two-way communication although a talk-back facility might be more important for more complex material or more advanced students.

Wolgamuth (1961) found no significant differences on attitudes, learning, or retention (after four weeks) for 80 college students divided into four groups: a) studio class, b) class with talk-back facility, c) class which could signal for pace of instruction, and d) class with no possibility of feedback.

Greenhill (1964) reported no significant differences between students having a talk-back facility and students without this possibility. As in the Wolgamuth study all students could hear all discussions and this might account for the lack of significant differences.

Jamison and Lumsden (1973) reported no significant differences among graduate business students in an economics course who were allowed to choose to attend the studio class, an overflow room with talk-back facilities, or a videotape replay on a final examination or an average of cases submitted during the quarter.

Dubin and Hedley (1969) summarized several studies regarding the attitudes of students and faculty to television at the college level rather than the effect of television on social or personal attitudes. For student attitudes their general conclusions may be summarized as follows:

1. Students receive ETV favorably and are even more favorable after they have experienced it;

2. Students are more favorable to ETV in general than they are toward their particular televised courses;

3. Given a choice between a televised and a conventional course, most students do not prefer television;

4. Faced with a choice between a televised course and a large lecture course, students modify their attitudes in favor of television;

5. If students feel they can get a better lecturer on TV, they will choose the televised course;

6. Most students feel they learn at least as much from a televised course as they do from a conventional course.

267

As an example of the impact of experience on attitudes, Wells (1974b) reported on a course which was broadcast over television with audio feed- back. At the beginning of the course 40% stated a preference for conven- tional instruction while 27% favored televised instruction. At the end of the course these preferences became 31% and 54% respectively. Wells concluded that the experience of the course may have altered preconceived notions regarding televised instruction.

The attitudes of faculty members might best be summarized as, "I have nothing against television courses in general but I wouldn't want my child to attend one." From several studies Dubin and Hedley reported that faculty members were highly favorable to TV courses in general (11-28% unfavor- able), less favorable to teaching a TV course (15-49% unfavorable), even less favorable to teaching TV courses on a regular basis (31-63% unfavorable), and least favorable to having their children attend a university using TV for elementary courses (59-75% unfavorable). Faculty with TV teaching experience were much more favorable to TV course attendance by their children (13-15 % unfavorable). One factor contributing to the response of faculty with respect to teaching televised courses may be the high cost in preparation time. Macomber and Siegel (1960) reported .that average prepa- ration time was 1.7 hours for conventional instruction, 5.3 hours for closed circuit courses, and 9.6 hours for open circuit television at Miami University.

PROGRAMMED INSTRUCTION

Programmed instruction is a general description for a type of instruc- tional process and may involve the use of books, teaching machines, radio or television. The use of a computer allows for much more complicated branching programs. The effectiveness of computer-assisted instruction is discussed in the following section. The programmed material usually allows for individual learning of a subject by presenting the subject in a well- ordered sequence of steps, allowing for student responses, and reinforcing those responses.

The general conclusions from several recent surveys of programmed instruction are: programmed instruction is as effective as conventional in- struction and usually involves less time. For example, Attiyeh et al. (1969) in an analysis of programmed instruction for college economics found no significant differences in economics knowledge between programmed and conventionally taught students. However, students with the programmed books spent only three weeks (an average of 12 study hours) compared to seven weeks for the lecture course. Time savings of 20% to 47% were also evident in three other studies (Hughes, 1962; Hughes and McNamara, 1961 ; and Hough, 1962) which revealed no significant differences in learning between groups using programmed instruction and students receiving con-

268

ventional instruction. Schramm's (1964) annotated bibliography includes 36 studies which

compared programmed and conventional instruction. He reported no signifi- cant differences in 18 studies, 17 studies significantly favoring programmed instruction, and only one favoring conventional instruction.

Silberman (1962) reviewed 15 experiments in programmed instruction and found significant differences favoring programmed instruction in nine of the cases and no significant differences in the remaining six cases. He also reported a time savings with the use of programmed materials in each of the experiments.

Lange (1972) reported the results of 112 studies of programmed instruction which were undertaken between 1960 and 1964. No significant differences were reported in 49% of the cases. Significant differences favoring programmed instruction were reported in 41% of the studies and the remaining 10% of the cases favored traditional instruction.

Zoll (1969) reviewed research on the use of programmed instruction for mathematics instruction in 35 studies (mostly doctoral dissertations). Thirteen of the studies compared programmed and traditional instruction with no significant differences in seven of the studies. Of the remaining six, three favored programmed instruction and three favored traditional instruc- tion.

Wells (1973) surveyed the use of technology in economics education (mostly at the college level). Of the 36 studies included in his survey, seven analyzed the use of programmed instruction. Programmed instruction provided significantly higher performance in two of the cases and there were no significant differences in the remaining five studies.

The surveys and studies analyzed did not contain any information on social attitude changes. Atti tude data in these studies only measured student attitudes toward programmed instruction. Zoll (1969) reported that ten of the mathematic studies he surveyed reported favorable attitudes toward programmed instruction and in three of the studies interest decreased with time spent on the program. Schramm (1964) reported that students were generally favorable although they were more often bored with long programs than with short ones and with programs which used short steps than with programs which had increasing step size or used long Steps.

Schramm (1964, p. 8) pointed out the common assumptions regarding an effective program design are:

(1) an ordered sequence of items, through which the student works in (2) "short steps," therefore (3) making few errors, as he records to each item (4)a constructed response, and receives (5)immediate knowledge of results. The student (6)works at h i s own pace, and (7)receives reinforcement for each correct response.

The vast majority of experiments (with the exception of Carpenter and Greenhill, 1963) which Schramm cited to test various aspects of the above

269

assumptions were of short duration (one to several hours). His conclusions basically supported the assumptions with the following notable exceptions: shorter steps required more time and individual pacing (a primary motivation for using programmed instruction) did not contribute to learning.

COMPUTER-ASSISTED INSTRUCTION

Computer-assisted instruction is a relatively new and expensive tech- nology which allows for more individuality in progression toward program goals than other technologies. Research into the effectiveness of CAI is recent and has amply demonstrated its effectiveness. The CAI programs for elementary students have usually been drill-and-practice mathematics or reading programs and have been supplemental to regular instruction. At the college level CAI has been used as a substitute for conventional instruction rather than as a supplement and the results have also been favorable.

Jamison et al. (1974) have provided an extensive analysis of CAI experiments at the elementary and college level and the discussions on the effectiveness of CAI in this section closely follows their discussion.

Hansen et al. (1968) reported on the use of CAI for physics and found that the group receiving CAI instruction performed significantly better than the groups receiving traditional instruction only or a combination of CAI and traditional instruction.

Adams (1969) and Morrison and Adams (1969) reported on the use of an hour of CAI instead of an hour of language laboratory as supplements to the lecture class in college German. CAI students were superior in reading and writing ability. No significant differences were uncovered in listening or speaking abilities.

Suppes and Morningstar (1969) reported on the use of CAI as a replacement for the normal classroom hours of an introductory college Russian course. Five hours of lectures were replaced with five CAI hours each week. The dropout rate for the CAI section was less than half the dropout rate of the regular section. In addition, the CAI students performed better on the final examination.

Three studies demonstrated the reduction in time which occurs for CAI usage. Axeen (1967) used the Plato system for teaching library use and found that CAI students took less time.

Bitzer and Boudreaux (1969) used the Plato system for a nursing course and found that the average student using CAI completed the materials in 50 hours compared with 84 lecture hours.

Homeyer (1970) used CAI for computer programming. Although he found no significant differences on examinations or grades for programs written by the students he did find significant time savings. CAI students used an average of 13.75 hours compared with 24 hours for the lecture groups.

270

In general it would appear that students using CAI learn more in the same amount of time or take less time to learn the same amount compared with students in conventional courses. These results point to the important need to consider more than one criterion in comparing effectiveness.

C o n c l u s i o n s

The effectiveness of any educational technology or instructional method can be evaluated against a variety of criteria. The majority of the studies surveyed in this paper have used some measure of student knowledge at the end of a course or course segment. Many of the studies have utilized locally generated final examinations. It is likely that these exams are not tested for reliability or validity and that they are not powerful enough to uncover differences among the methods tested. Furthermore, a final exam is likely to generate extra studying on the part of students to compensate for any perceived deficiency in the technology or method of instruction for the courses they attended. Without analyzing the particular test measure it is not possible to know the type of learning which was tested. It was suggested in this paper that the learning hierarchy proposed by Gagn6 (1965) or the taxonomy proposed by Bloom (1956) might be a logical basis for sub- dividing the tasks of a course and testing for its effectiveness. Other criteria used to evaluate a technology might include dropout rates, percentage of students continuing in a given subject area, average completion time for course material, and student ratings of courses, technology, methods, and instructors.

The overall conclusion which one reaches after an examination of the studies is that there are no significant differences (on cognitive measures of student learning) among a variety of technologies and instructional methods - television, programmed instruction, computer-managed and assisted in- struction, large lecture classes, small discussion groups, independent study, and graduate student instructors as substitutes for full-time faculty members. As Chu and Schramm (1967) have found: Given favorable conditions, pupils can learn from any instructional medium now available.

The most consistent finding has been the savings of time realized in the use of technologies amenable to individualized instruction: programmed instruction (e.g. Attiyeh et al. 1969; Hough, 1962; and Hughes, 1962), computer-managed instruction (e.g. Kelley, 1968, 1972), and computer- assisted instruction (e.g. Axeen, 1967; Bitzer and Boudreaux, 1969; and Homeyer, 1970).

Most of the individual research studies and surveys of studies cited in this paper have compared some technology such as television (with no qualifications) with conventional instruction (with no qualifications). The

271

problem with this type of research model is that it does not uncover any variations in software, organization, or interactions among media, instructor characteristics, and learner characteristics. Furthermore, it does not aid in an understanding of relationships between technology and instructional objec- tives. It may well be that a given instructional technology is especially successful for a given instructional objective or type of learner.

It should be clear from this paper that future research should be directed towards uncovering unique advantages and accounting for the many factors which affect our ability to discriminate among technology effec- tiveness. As Allen (1971) has stated:

The study of this three way interaction of stimulus, task, and learner is extremely c o m p l e x . . . The folly of assigning generalized and all-inclusive attributes to specific classes ~af media under all conditions is finally being appreciated, and we should observe more intensive research efforts to discover how to design and manipulate the media so as to enhance their effectiveness under specific instructional conditions. Such research will occupy our attention for some time, leading to the evolution of taxonomies of unique media effects so that we can predict that the use of a particular instructional medium will lead to specified learning outcomes with different kinds of learners.

Notes

1 The term "effectiveness" in this paper refers to the impact of different technologies on a variety of evaluation criteria such as examinations and attitudes of students. While no significant differences may appear among technologies with respect to these criteria any decision involving a choice of technology should include an analysis of costs. This paper does not include cost analysis. The interested reader is referred to other studies (General Learning Corporation, 1968; Miller, 1970; Witmer, 1972; Layard, 1973; and Jamison et al., 1975) for detailed analysis of technology costs.

2 The term "technology" is used in this paper in the economic sense which includes any course methodology rather than being restricted to the common usage which would include machinery only.

3 Among the surveys which reach the general conclusion of no significant differences are: Schramm (1964); Chu and Schramm (1967); Dubin and Taveggia (1968); Dubin and Hedley (1969); Jamison et al. (1974); and Wells (1974a).

4 A step or frame in programmed instruction is the material presented before a response by the student is necessary.

References

Adams, E. (1969). "Field Evaluation of the German CAI Lab," in Atkinson, R. C. and Wilson, H., eds., Computer-assisted Instruction: A Book of Readings. New York: Academic Press.

272

Allen, W.H. (1971). "Instructional Media Research: Past, Present, and Future," A V Communication Review 19.

Anderson, B.D. and Greenburg, E. (1972). "Educational Production Functions for Teacher-Technology Mixes: Problems and Possibilities." Program on Application of Communications Satellites to Educational Development, Washington University.

Armsey, J.W. and Dahl, N.C. (1973). An Inquiry into the Uses of Instructional Technology. New York: The Ford Foundation.

Attiyeh, R. and Lumsden, K.G. (1969). "The Efficiency of Programmed Learning in Teaching Economics: The Results of a Nationwide Experiment," American Economic Review. 59:217-224.

Attiyeh, R. and Lumsden, K. G. (1972). "The Production of Economics Understanding: An Analysis of the First Year University Economics Course." Economics Education Project, Heriot-Watt University, (mimeographed).

Axeen, R. E. (1967). "Teaching the use of Library to Undergraduates: An Experimental Comparison of Computer-Based Instruction and the Conventional Lecture Method." Report No. R-362, University of Illinois, Coordinated Science Laboratory.

Bach, G. L. (1970). "Student Learning in Basic Economics: An Evaluated Experimental Course," in Lumsden, K. G., ed., Recent Research In Economic Education. Englewood Cliffs, New Jersey: Prentice-Hall.

Baker, F.B. (1970). "Computer-Based Instructional Management Systems: A First Look," Review of Educational Research 41:51-70.

Bitzer, M. and Boudreaux, M. (1969). "Using a Computer to Teach Nursing," Nursing Forum 8.

Bloom, B.S. (1956). Taxonomy of Educational Objectives: Handbook I, Cognitive Domain. New York: David McKay.

Bretz, R. (1971). The Selection of Appropriate Communication Media for lnstruction: A Guide for Designers of Air Force Technical Training Program. Santa Monica, Cal.: RAND.

Briggs, L. J., Campeau, P. L., Gagn6, R. M. and May, M. A. (1967). InstructionalMedia: A Procedure for the Design of Multi-Media Instruction, a Critical Review of Research, a~d Suggestions for Future Researeh. Pittsburgh: American Institute for Research.

Carpenter, C. R. and Greenhill, L. P. (1955). "Instructional Television Research Project Number One: An Investigation of Closed-Circuit Television for Teaching University Courses." University Park, Pa.: Pennsylvania State University.

Carpenter, C.R. and Greenhill, L.P. ~1958). "An Investigation of Closed-Circuit Television for Teaching University Courses: Instructional Television Research Report Number Two." University Park, Pa. : Pennsylvania State University.

Carpenter, C. R. and Greenhill, L.P. et al. (1963). "Comparative Research in Methods and Media for Presenting Programmed Courses in Mathematics and English." Univer- sity Park, Pa.: University Division of Instructional Services, Pennsylvania State Univer- sity.

Chu, G. C. and Schramm, W. (1967). Learning from Television: What the Research Says. Washington, D.C.: National Association of Educational Broadcasters.

Costin, F., Greenough, W.T., and Menges, R. J. (1971). "Student Ratings of College Teaching: Reliability, Validity, and Usefulness," Review of Educational Research 41 : 511-535.

Doty, B. A. and Doty, L. A. (1964). "Programmed Instructional Effectiveness in Relation to Certain Characteristics," Journal of Educational Psychology 55: 334-338.

Dreher, R. E. and Beatty, W. H. (1958). Instructional Television Research, Project No. 1: An Experimental Study of College Instruction Using Broadcast Television. San Francisco: San Francisco State College.

273

Dubin, R. and Hedley, R.A. (1969). The Medium may be Related to the Message: College Instruction by T. V. Eugene: University of Oregon Press.

Dubin, R. and Taveggia, T. C. (1968). The Teaeher-Learning Paradox: A Comparative Analysis of College Teaehing Methods. Eugene: University of Oregon.

Fels, R. and Starleaf, D. R. (1963). "Controlled Experiments in Teaching Techniques," Southern Economic Journal 30: 68-73.

Forsythe, R.O. (1970). "Instructional Radio," in Tickton, S.G., ed., To Improve Learning: An Evaluation of Instructional Technology. New York: R. R. Bowker Co.

Gagn~, R. M. (1962). "The Acquisition of Knowledge," Psyehology Review 69: 355-365. Gagn6, R. M. (1965). The Conditions of Learning. New York: Holt, Rinehart and Wilson. Gagn~, R.M., Mayer, J.E., Garstens, H.L. and Paradise, N.E. (1962). "Factors in

Acquiring Knowledge of a Mathematical Task," Psychology Monographs 76. Gagn~, R.M. and Paradise, N.E. (1961). "Abilities and Learning Sets in Knowledge

Acquisition," Psychology Monographs 75. General Learning Corporation. (1968). Cost Study of Educational Media Systems and

Their Equipment Components, 3 volumes. Washington: General Learning Corporation. Gropper, G.L. and Lumsdaine, A.A. (1961). "An Experimental Evaluation of the

Contribution of Sequencing, Pre-testing, and Active Student Responses to the Effec- tiveness of 'programmed' TV Instruction," Studies in Televised Instruction, Report No. 3. Pittsburgh: Metropohtan Pittsburgh Educational Television Stations WQED- WQEX, and American Institute for Research.

Gropper, G. L., Lumsdaine, A. A. and Shipman, V. (1961). "Improvement of Televised Instruction based on Student Responses to Achievement Tests." Pittsburgh: Metro- politan Pittsburgh Educational Television Stations WQED-WQEX, and American Institute for Research.

Greenhill, L. P. (1964). "Penn State Experiments with Two-Way Audio Systems for CCTV," NAEB Journal 23: 73-78.

Hansen, D.N., Dick, W. and Lippert, H. T. (1968). "Research and Implementation of Collegiate Instruction of Physics via Computer-assisted Instruction." Volume 1. Technical report No.3, Florida State University, Computer-Assisted Instruction Center.

Homeyer, F.C. (1970). "Development and Evaluation of an Automated Assembly Language Teacher." Technical Report No. 3, Computer-Assisted Instruction Laborato- ry, University of Texas.

Hough, J.B. (1962). "Research Vindication for Teaching Machines," Phi Delta Kappan 42: 240-242.

Hughes, J. L. (1962). "Effect of Changes in Programmed Text Format and Reduction of Classroom Time on the Achievement and Attitudes of Industrial Trainees," Journal of Programmed Instruetion 1 : 43-54.

Hughes, J. L. and McNamara, W. L. (1961). "A Comparative Study of Programmed and Conventional Instruction in Industry," Journal o f Applied Psychology 45 :225-231 .

Jamison, D. and Lunsden, K. G. (1973). "Television and Efficiency in Business Evaluation: Methodology and Application to Instructional Technology. Princeton, N. J.: Economics and Educational Planning Group, Educational Testing Service.

Jamison, D., and Lumsden, K. G. (1973). "Television and Efficiency in Business Education," in Lumsden, K.G., ed., Efficiency in Universities: The La Paz Papers. Amsterdam: Elsevier.

Jamison, D., Suppes, P. and Wells, S. (1974). "The Effectiveness of Alternative Instruc- tional Media: A Survey," Review of Educational Research 44: 1-68.

Kelley, A.C. (1968). "An Experiment with TIPS: A Computer-Aided Instructional System for Undergraduate Education," American Economic Review 58: 446-457.

274

Kelley, A. C. (1972). "TIPS and Technical Change in Classroom Instruction," American Economic Review 62: 422-429.

Kelley, C. F. (1964). "The Efficacy of Television in the Schools," Dissertation Abstracts 24: 224.

Kight, H.R. and Sassenrath, J. M. (1966). "Relation of Achievement, Motivation and Test Anxiety to Performance in Programmed ' " " Instruction, Journal of Educational Psychology 57: 14-24.

Klos, J. J. and Trenton, R. W. (1969). "One Semester or Two," Journal of Economic Education 1: 51-55.

Krathwohl, D., Bloom, B. S. and Masia, B. (1964).Handbook of Educational Objectives: Handbook II, Affective Domain. New York: David McKay.

Lamphear, C., and McConnell, C. R. (1970). "A Note on the Use of Teaching Assistants in the Principles Course," Journal of Economic Education 1 : 139-142.

Lange, P. C. (1972). "Today's Education," National Education Association 61, 59. Layard, R. (1973). "Cost-Effectiveness of the New Media in Higher Education," in

Lumsden K.G., ed., Efficiency in Universities: The La Paz Papers. Amsterdam: Elsevier.

Lewis, D. R. and Dahl, T. (1971). "The Test of Understanding in College Economics and Its Construct Validity," Journal of Economic Education 2:155-166 .

Lloyd, J.W. (1970). "Role Playing, Collective Bargaining, and the Measurement of Attitude Change," Journal of Economic Education 1 : 104-110.

Los Angeles City School District, Division of Extension and Higher Education. (1959). "An Evaluation of Closed-circuit Instructional Television in Los Angeles City College and Los Angeles Valley College: Final Report" (mimeo).

Lublin, S. C. (1965). "Reinforcement Schedules, Scholastic Aptitude, Autonomy Need, and Achievement in a Programmed Course," Journal of Educational Psychology 56: 295-302.

Macomber, F.G. and Siegel, L. (1960). "Final Report of the Experimental Study in Instructional Procedures." Oxford, Ohio: Miami University.

MacPherson, E.D. (1967). "Some Correlates of Anxiety in Learning Programmed Mathematics," Dissertation Abstracts 27: 2948-A.

McConnell, C. R. (1968). "An Experiment with Television in the Elementary Course," American Economic Review 58 : 469-482.

McConnell, C. R. and Lamphear, C. (1969). "Teaching Principles Of Economics without Lectures," Journal o f Economic Education 1 : 20-32.

Miller, J. G. (1970). "Deciding whether and how to use Educational Technology in the light of Cost-Effectiveness Analysis," in Tickton, S. G., ed., To Improve Learning: An Evaluation of Instructional Technology. New York: R. R. Bowker Co.

Morrison, H. and Adams, E. (1969). "Pilot Study of a CAI Laboratory in German," in Atkinson, R. C. and Wilson, H., eds. Computer-assisted Instruction: A Book of Readings. New York: Academic Press.

Oates, W.E. and Quandt, R.E. (1970). "The Effectiveness of Graduate Students as Teachers of the Principles of Economics," Journal of Economic Education 1: 130-138.

Paden, D. W. and Moyer, M. E. (1971). "Some Evidence on the Appropriate Length of the Principles of Economics Course," Journal of Economic Education 2:131-137.

Peterson, P. "A Survey of Studies in Mastery Learning." Unpublished paper, School of Education, Stanford University.

Pflieger, E. F. and Kelly, F. C. (1961). The National Program in the Use of Television in the Public Schools. New- York: The Ford Foundation and the Fund for Advancement of Education.

275

Saunders, P. (1971). "More on the use of Graduate Student Instructors in the Intro- ductory Economics Course," Journal of Economic Education 3: 36-40.

Schramm, W. (1962). "Learning from Instructional Television," Review of Educational Research 32: 156-167.

Schramm, W. (1964). The Research on Programmed Instruction: An Annotated Biblio- graphy. Washington, D.C.: U.S. Department of Health, Education and Welfare.

Shrable, K. and Sassenrath, J. M. (1970). "Effects of Achievement, Motivation and Test Anxiety on Performance in Programmed Instruction," American Educational Research Journal 7: 209-220.

Silberman, H.F. (1962). "Characteristics of some recent studies of instructional methods," in Coulson, J. E., ed., Programmed learning and computer-based instruc- tion. New York: Wiley.

Spence, R.B. (1928). "Lecture and Class Discussion in Teaching Educational Psycholo- gy," Journal o f Educational Psychology 19: 454-462.

Stickell, D. W. (1963). "A Critical Review of the Methodology and Results Comparing Televised and Face-to-Face Instruction." Unpublished doctoral dissertation, Pennsyl- vania State University.

Suppes, P. and Morningstar, M. (1969). "Computer-assisted Instruction," Science 166: 343-350.

WeUs, S. (1973). "An Evaluation of the Effectiveness of Economics Education." Un- published paper, Stanford University.

Wells, S. (1974a). "The Impact of Instructional Technology," in Kincaid, H.V., McEachron, N. B. and McKinney, D., eds., Technology in Public Elementary and Secondary Education: A Policy Analysis Perspective. Menlo Park, California: Education Policy Research Center, Stanford Research Institute.

Wells, S. (1974b). "Technology, Efficiency and Educational Production." Unpublished doctoral dissertation, Stanford University.

Wells, S. (1975). "The Relationship among Student Decisions, Educational Resources, and Resource Effectiveness." Working Paper No. 303, Faculty of Commerce, Univer- sity of British Columbia.

Willis, E. E. (1940). "The Relative Effectiveness of Three Forms of Radio Presentation in Influencing Attitudes." Unpublished doctoral dissertation, University of Wisconsin.

Wisconsin Research Project in School Broadcasting (1942). Radio in the Classroom. Madison: University of Wisconsin Press.

Witmer, D.R. (1972). "Cost Studies in Higher Education," Review of Educational Research 42: 99-127.

Wolgamuth, D. (1961). A Comparative Study of Three Techniques of Student Feedback in Television Teaching: The Effectiveness of an Electrical Signal Feedback System. NDEA Title VII Project No. 453. Washington, D.C.: U.S. Office of Education.

Zoll, E.J. (1969). "Research in Programmed Instruction in Mathematics," The Mathe- matics Teacher 62:103-110 .