CRITICAL PERSPECTIVES ON EDUCATIONAL … · approach to evaluation can be consistent ... concern...

LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME 111The Canadian Journal of Program Evaluation Special Issue Pages 111–135ISSN 0834-1516 Copyright © 2000 Canadian Evaluation Society

CRITICAL PERSPECTIVES ON EDUCATIONALEVALUATION

Michael CollinsUniversity of SaskatchewanSaskatoon, Saskatchewan

This article examines what a critically informed orientation toevaluation in formal and non-formal education settings entails.A substantial critique of the technical rationality pervading con-ventional approaches to evaluation in education is illuminatedby comparing functionalist models (exemplified in competency-based education and behaviourist variations on that theme) withparticipatory approaches, and through a corollary account ofthe way system imperatives intrude on lifeworld values. How-ever, the author argues that a commitment to critical evalua-tion in education calls for systematic engagement withconventional, mainstream initiatives as well as with alterna-tive approaches. Some guiding principles, consistent with theemancipatory aims of a critical orientation, are identified inmaking the case for the feasibility and necessity of critical evalu-ation in education. Questions are raised about the efficacy ofthe continuing debate on quantitative versus qualitative meth-ods and the effects on critical evaluation of postmodern dis-course.

Cet article examine la signification de l’évaluation d’une opti-que critique dans les mileux éducatifs formels et informels. Unecritique considérable de la rationalité stricte qui se répand dansdes approches conventionelles de l’évaluation dans le domainede l’éducation est mise en lumière en faisant une comparaisonentre les modèles fonctionnels (exemplifiés par l’éducation ba-sée sur la compétence et des variations comportementalistes surce theme-là) et les approches participatives, et par l’analyse con-séquente de la façon dont les impératifs de système influencentnos valeurs de vie. Cependant, l’auteur soutient qu’un engage-ment à l’évaluation critique en éducation exige une synthèsesystématique des approches conventionelles dominantes avecdes approches alternatives. En faisant l’argument pour la pos-sibilité et la nécessité d’une évaluation critique dans le domainede l’éducation, cet article met en lumière des principes saillan-tes inhèrentes aux buts principaux d’une approche axée surl’émancipation. De plus, on pose des questions sur l’efficacité

Abstract:

Résumé:

THE CANADIAN JOURNAL OF PROGRAM EVALUATION112

du debat continu au sujet des méthodes quantitatives vis-à-visdes méthodes qualitatives, ainsi que sur les effets de l’évalua-tion critique du discours postmoderne.

A thought-provoking article by VanderPlaat (1997) pro-vides important insights of direct relevance for critical approaches inboth formal and informal educational settings. As in a previous articlefor a British publication, VanderPlaat (1995) draws on Jurgen Haber-mas’s theory of communicative action, feminist studies, and post-modern thought to engender an understanding of how a criticalapproach to evaluation can be consistent with emancipatory aims.Such an understanding is intended “to address the evaluative con-cerns of feminists and other critical thinkers who are politically com-mitted to an agenda for social change” (p. 93). Yet, it is noteworthy thatVanderPlaat warns against “the widespread assumption that onlyqualitative data can serve an emancipatory practice” (1997, p. 149).

In terms of its relevance for educational initiatives, VanderPlaat(1997) provides an instructive summary of Habermas’s concept ofcommunicative action. From this critical perspective, the technical(instrumental) rationality steering traditional mainstream evalua-tion projects serves to sustain “the oppression that results from theunequal distribution of and access to resources” which the accompa-nying evaluative discourse on intervention seeks “to compensate orameliorate” (p. 145). In this Habermasian view, which provides criti-cally informed pedagogy with a useful explanation of how lifeworld(aesthetic, moral, practical, and political) interests are colonized bythe system (Collins, 1991, 1998), “the only legitimate rationale forsystemic action is one that is ‘communicatively secured’ through pub-lic discussion and agreement” (VanderPlaat, 1997, p. 145).

Clearly, the level of participatory democracy envisaged by aHabermasian viewpoint does not characterize the way conventional,and even non-conventional, formal educational evaluations are con-ceptualized and carried out. The concept of an “ideal speech situa-tion,” which Habermas posits to account for such a possibility, isexactly that — an idealized situation. In a sense, VanderPlaat isonto this shortcoming of the Habermasian project, and she arguesconvincingly that “empowering research techniques cannot, in andof themselves, effectively inform and support the political aspira-tions of emancipatory intervention” (VanderPlaat, 1997, p. 143).

The purpose of this article is to engage critically with these con-cerns and insights raised by VanderPlaat and, in accordance with a

LA REVUE CANADIENNE D'ÉVALUATION DE PROGRAMME 113

notion she espouses of evaluation as critique (1995), to enlarge onand substantiate implications that, as evaluators and as stakeholdersin the field of education, we can connect our critical understandingto sensible pedagogical practice that is motivated by emancipatoryaspirations.

CRITICAL TURN IN EVALUATION

The shift in emphasis VanderPlaat proposes (1997) from a preoccu-pation with technique to the agency of relevant stakeholders is con-sistent with the discourse on critical pedagogy (Collins, 1991; Giroux,1988). In particular, VanderPlaat’s critical observation on the im-portance of “the interplay between human agency and systemic struc-tures” (p.144) nicely reinforces what Collins (1994) had to say inthis regard about program planning and evaluation in modern adulteducation practice:

There is a better way to approach program planning andevaluation than that which now predominates withinmodern adult education practice. This alternative ap-proach entails giving up the current obsession with meth-ods, models, and techniques in favor of putting ourselvesinto practice. (p. 95)

From this alternative viewpoint, which extends to other educationalcontexts, such as public schools and universities, relevant stake-holders for a critical evaluation discourse include the evaluated, asactive participants with some potential to influence the course ofevents, as well as designated evaluators and influential stakeholders.As for Guba and Lincoln (1992), Patton (1982), and Eisner (1986),the deployment of technique in evaluation is subordinated to a priorconcern for the practical dimension and phenomenological investi-gations (incorporating moral and aesthetic concerns). However, acritical approach to evaluation goes beyond privileging the evalua-tor as agent or “instrument” (Guba and Lincoln, 1992) and aims,instead, to engender a concern for collective responsibility in theemergence, design, dissemination, and political effects of the evalu-ative process. In any event, critical commitment entails purposefulinvolvement in educational evaluation as a political process thatinfluences the nature of educational programs and policy, institu-tional arrangements, teaching practices, and learning processes.

In this critical view, the political intent of influential stakeholders,especially where they manage to place themselves outside of the


evaluative gaze, is taken into account, along with the spurious rheto-ric on participation, accountability, and transparency that often ac-companies bureaucratically imposed models of evaluation. Thiscritical understanding is particularly relevant where the evaluativerhetoric masks predetermined management agendas to reallocateresources, downsize program offerings, and legitimize layoffs. At thesame time, a critical perspective on educational evaluation is alertto contradictions and possibilities, in both institutional and com-munity-based settings, that make strategies toward more genuinelydemocratic participation a reasonable aspiration.

Thus, the entire discourse on educational evaluation in its variousforms describes a vitally important arena for the purposeful involve-ment of those committed to a critical pedagogy. This critical com-mitment can manifest itself in any style of evaluation in traditionalmainstream approaches to evaluation that claim a dubious legiti-macy as being scientific while privileging the role of evaluator asprofessional expert, in responsive evaluations that acknowledge thesubjective dimension of the evaluative enterprise and give greaterconsideration to stakeholder interests, and in the kinds of partici-patory evaluations that intend from the outset to focus on the poli-tics of empowerment.

While participatory approaches declaring emancipatory interests aremore clearly aligned with the practical and theoretical concerns ofcritical pedagogy and critical approaches to evaluation, it is impor-tant to acknowledge the potential in educational settings for em-powerment strategies, even in the forms of resistance, that canemerge from systematic involvement in traditional, expert-drivenevaluations as well as in more responsive evaluations in which po-litical implications that are typically avoided could be identified asadvancing the interests of key stakeholders. For example, the evalu-ation of curriculum initiatives and educational policy formation thatare deployed without meaningful teacher participation and areshaped by prescriptive technicist designs, such as narrowly conceivedcompetency/objectives-based formats (Collins,1994), provides an op-portunity to raise concerns about the de-skilling of educators. Inthis view of the critical tendency in educational evaluation, all formsof evaluation, whether bureaucratically imposed or derived fromvoluntary initiatives, are characterized by contradictions and possi-bilities that are relevant to an emancipatory project. A tendencyamong some radical educators to exclude institutional settings, inparticular our schools (Holt, 1976; Illich, 1970), as relevant contexts


for an emancipatory practice is, from a critical perspective, politi-cally naive and misdirected (Collins, 1998).

EMANCIPATORY POTENTIAL: EDUCATION OF A FREE PEOPLE

Under present ideological circumstances, and still in the wake of aneo-conservative (now neo-liberal and neo-social democratic) ascend-ancy which normalizes educational discourses characterized by anoverriding concern for efficiency, economic competitiveness for a glo-bal economy, and coping skills, it has become increasingly difficultto talk realistically in moral and political terms about the quest forempowerment and social justice. While such talk is tolerated as amarginalized discourse in educational settings, especially in ouruniversities, it is quickly put back into place, amid charges of “po-litical correctness,” when political strategies to which it gives riseseem to be making some headway. This describes the current situa-tion for discourses on anti-racist pedagogy, feminist issues, ecologi-cal concerns, and systemic poverty. (There remains, of course, thepolitically defused curriculum discourse on critical thinking, or de-bate, about such issues as equity, individual rights, multi-culturalism, the environment — “let’s plant more trees” — and the“disadvantaged” that pose little threat to existing relationships ofpower).

Yet, despite this defusing of the moral and political language asso-ciated with emancipatory pedagogy, an aspiration to advance thereal interests of less empowered stakeholders, and not only of thosewho clearly belong to social groups subject to systemic margin-alization and repression (Fanon, 1988; Freire, 1981, 1997), is a ma-jor feature of the critical tendency in the evaluation of educationalprograms.

Thus, the interests of teachers as employees, students (particularlyadult students who bring mature experience to educational settings),and parents as potentially major decision makers become very sig-nificant in the context of a critical approach to evaluation. In thisregard, VanderPlaat (1997, p. 144) acknowledges the “importantcontributions to the concept of ‘empowering’ evaluation strategies”made by Fischer (1985), Kemmis (1993), Everitt (1996), and Fetter-man, Kaftarian, and Wandersman (1996). However, while commend-ing the work of these contributors, VanderPlaat observes that “westill have a long way to go in developing an approach to evaluationthat truly supports empowerment-oriented aspirations” (p. 144).


VanderPlaat is on the right track and, to see how this might playout, some discussion of an “emancipatory ethic” as an aspiration forthe education of a free people under current conditions (that is,against the compelling claims of capital) is in order.

An adequate discussion of the concept of freedom, even in the con-text of a focus on education, is beyond the scope of this article. Yet acritical engagement with the discourse of emancipatory pedagogy,as a collective experience and going beyond a necessary preoccupa-tion with the defence of individual rights (a more readily acceptableproposition), is a prerequisite to learning participatory democracy.

The discourse on emancipatory pedagogy goes back a long way. ForThomas Hodgskin, co-founder of the mechanics’ institutes movementin 1823, “the education of a free people will always be directed morebeneficially for them when it is in their own hands” (Halévy, 1956, p.86). Author of Labour Defended Against the Claims of Capital, whichfirst appeared in 1825, Hodgskin (1963) was aware of how the objec-tive conditions of his time, including powerful interest groups, wouldinevitably work against his aspirations for an emancipatory pedagogythrough an organization which, nevertheless, became an interna-tional movement in adult education. Hodgskin’s aspiration for genu-inely democratic participation in the development and governance ofeducational institutions prefigures the aims of emancipatory or em-powerment-oriented initiatives in education today.

Clearly, contemporary forms of governance that shape our educa-tional systems, formal curriculum, and social learning processes stillpreclude emancipatory aims for an education in the hands of thepeople. The research of Bowles and Gintis (1976) and Bourdieu andPasseron (1977) remain instructive in the way they explain how oureducational institutions still serve to sustain an unequal distribu-tion of cultural capital through the credential system and therebyreproduce existing social relations. Yet there are important formsof elective representation in the field of education (school boards,for example) which, though falling short in terms of exemplifyinggenuine participatory democracy, prefigure democratic structureswhere the prospects for “education in the hands of the people” canbe realized. In the meantime, an emancipatory pedagogy calls forcritical engagement with institutional evaluations deployed underthe auspices of these quasi-representative governing bodies, eventhough they appear, in these times, to be increasingly in the grip ofa business corporate ethos. This critical orientation sets the stage


for a discourse on the prospects for participatory evaluation in edu-cation that is in keeping with an emancipatory ethic. The followingsection posits, in summary form, some of the guiding principles of acritically informed approach to educational evaluation.

SOME GUIDING PRINCIPLES

A critical approach to educational evaluation wedded to an emanci-patory ethic, though intended to counter the effects of evaluativeprocesses that are steered by a one-sided obsession with technique(instrumental rationality), does not of necessity abjure the use ofconventional methods. On the contrary, it is important to know, forexample, how to prepare carefully designed questionnaires, checkedfor internal validity, which will elicit relevant data in an economi-cal fashion from specifically identified audiences. An ability to in-terpret the significance of the data, their limitations, and the extentof their informative potential is of legitimate concern to both desig-nated evaluators and respondents. The same level of careful atten-tion is also needed in the construction and application of guidedquestionnaires which support more meaningful responses to begleaned from less formally structured interviews (including conver-sation, group discussion, and narrative), pointing the way to pros-pects for a problem-posing dialogue on critical issues. In all of this,critical evaluations are alert to the need for systematic preparationand follow-through procedures, for validity checks through triangu-lation, and for well-strategized negotiations with and among influ-ential stakeholders.

The ongoing quantitative versus qualitative debate in educationalevaluation and research, in which staunch advocates of the formerorthodoxy still regard phenomenological data as hopelessly subjec-tive while many proponents of the latter position simply deny therelevance of qualitative data, is viewed as dysfunctional from a criti-cal perspective. Nor, from a critical viewpoint, is it worthwhile tomediate the differences between a spurious quest for “scientific ob-jectivity” (the orthodox quantitative standpoint) and a disdainfulavoidance by the qualitative side of any significant representationthrough data expressed in quantitative terms. Attempts to mediatethe qualitative versus quantitative positions entails falling betweentwo positivistically posed paradigms.

On the question of method, a useful guideline for a critically ori-ented approach to educational evaluation can be derived from the


observation by Farber (1968): “A diversity of methods is required bythe endless diversity of problems. If methods are instituted to solveproblems or to answer questions, we must be prepared to enlargeour conception of methods accordingly” (p. 10). This is also the ma-jor theme of Feyerabend’s study (1984) in which he argues that weshould keep our options open with regard to method rather thanrestricting ourselves in advance.

In addition to this view on the use of method, a critical perspec-tive on educational evaluation is further distinguished fromexclusionary concerns of the quantitative and qualitative para-digms since it begins with an understanding that education is in-evitably political, infused at all levels and in various contexts withideological struggle. This critical understanding, a key aspect ofcritical pedagogy exemplified in the much-cited work of Freire(1981) and Gramsci (1971), has an effect on the way those whoshare it approach the evaluative task. With a realistic expectationthat any educational situation is inevitably shaped, to a greater orlesser extent, by political and ideological factors (it comes with theturf), the critically informed are able to avoid both the indignantposturing and cynical resignation of otherwise progressive educa-tors who support the idea of education for a more just society, butnow view the way unequal power relationships affect current in-stitutional arrangements as being set, and already played out,within the parameters of an “end of history” (Fukuyama, 1992)ideological discourse. An understanding that education is a dy-namic political discourse, and that politics is the art of the possi-ble, enables a critical approach to evaluation for empowerment toidentify reasonable grounds for its commitments in a principle andpedagogy of hope (Bloch, 1995; Freire, 1994).

As previously indicated, the longstanding aspirations for the educa-tion of a free people in their own hands can be viewed as reasonableto the extent that meaningful and substantial participation is pre-figured in existing arrangements. In this regard, it makes sense thata critical approach to evaluation should be concerned with ways inwhich various groups (and most particularly the least empowered)affected by educational initiatives are involved in the evaluativeprocess. Thus, the issue of participation — the question of whoseinterests are served, who determines the categories to be reviewed,and who sets the level of investigation to be undertaken — is cen-tral to a critically informed orientation to educational evaluation.In a subsequent section, on participatory evaluation, the nature of


this concern for substantive democratic involvement on the part ofvarious stakeholder groups is more fully explored. At the same time,however, it is suggested that an over-determined concern for repre-sentation should be resisted where it incapacitates sensible strate-gies for political emancipation.

A critical perspective on evaluation is guided by a commitment insupport of publicly funded education, ready accessibility formarginalized groups to the best of what it has to offer, and opposi-tion to initiatives intent on reproducing privileged forms of school-ing. Accordingly, a tendency toward responsive evaluations that aimto identify improvements to be made in publicly funded education isfavoured over narrowly defined accountability approaches that con-centrate on deficiencies. Publicly funded education, burdened withunreasonable expectations that it should be able to address most ofthe ills of modern society, is seriously undermined by evaluationsthat contribute to a discourse on how it falls short rather on whattransformations are needed for it to further its role in the interestsof the majority of ordinary men, women, and children. In times when“blame the schools” discourse is accompanied by ideologically driventendencies that support cutbacks in public education, effect a trans-fer of public funds to private-sector education, enthrone businesscorporate values, and erode teacher autonomy, the issue of how edu-cational evaluations are constructed is of critical concern. In defenceof publicly funded education, formal evaluations should be regardedas occasions for critical engagement and contestation.

Critical pedagogy recognizes the importance of the community asboth a socializing influence and a primary pedagogical context wheresocial learning processes occur. Educational evaluations from thisperspective are concerned not only with the emancipatory potentialof community development initiatives, but also with the nature andextent of the linkages between educational institutions and com-munity. Hence, to overcome the division between our educationalsystems and community, critically informed evaluations have a roleto play in assessing the extent to which community interests arerealistically represented in the educational programming and cur-riculum of our schools and universities. This focus on the linkagesbetween our educational systems and community, already endorsedin the aims of designated community schools, university extension,and such quintessential Canadian organizations as Frontier Col-lege (Cook, 1987), is relatively easy to legitimize and can be intensi-fied as a relevant aspect of the evaluative discourse in education.


Aligned with this valuing of the community’s role, a critical approachto evaluation looks for how co-operation is facilitated not only inrelationships between educational institutions and community, butalso in classroom learning processes, curriculum, and institutionalarrangements. There are significant contradictions to be observedin this regard when the emphasis is on both the merits, even thenecessity, of competition in education contexts and on rhetoric aboutthe virtues of co-operation (in the discourse on multicultural educa-tion, for example). In particular, there should be an awareness ofthe way co-operative pedagogical endeavours can foster the learn-ing of communicative competence, a requisite for participatory de-mocracy, whereas a reinforcement of individualistic competitivebehaviours works against its realization. The critical tendency whichendorses the collective empowering potential of co-operative initia-tives extends to how the issue of stakeholder representation anddemocratic participation in educational program development andevaluation is viewed.

Though not subscribing to the values of a competitive, market-drivenethos, a critical perspective places a high value on initiatives thatconnect education with the world of work. Work experience in edu-cation is valued, but not in the way that merely serves job-marketdemands (that is, the immediate and narrowly defined requirementsof employers), nor in preparing people for specific future career rolesas either managers or manual workers. Rather, a critically orientededucation aims to strengthen people’s capacities to take on relevantand fulfilling work as productive members of society.

Specific training, which in publicly funded education should includethe acquisition of practical skills that will retain their usefulnessbeyond short-term jobs, comes at the stage when employment inter-ests are identified. In any event, the right and responsibility to earna decent living are presupposed. The present tendency of businesssector interests and the state to off-load responsibility for unem-ployment and labor market problems on the educational system andthe skill deficiencies of individuals is brought into question.

It is further understood from a critical perspective that educationshould attend to the all-round development of the personality, pre-paring the way for creative engagement with aesthetic, cultural, andsocially interactive aspects of everyday life. Thus, there is a needfor critical assessment of criteria advanced to determine what is in-cluded in, and what is significantly absent from, the educational


provision for personal development. In this regard, the status of thearts (music, drama, painting, and poetry) in school curricula, forexample, is a matter for critical evaluation.

FUNCTIONALIST PARAMETERS: CBE AS PARADIGM

Typically, the forms of evaluation deployed in education are notguided by the commitments and guiding principles of critical peda-gogy. Rather, educational programs, and the interests they are in-tended to address, call for more narrowly defined models ofevaluation with which critical approaches have to contend. Theprevalence of a behaviourist discourse on objectives-based learning,and so-called competency-based education (CBE) formats, exemplifya prescriptive mainstream tendency to needs assessment, curricu-lum design, and evaluation that runs counter to the emancipatoryaims of critical pedagogy but tends to find favour with managementand corporate interests.

Competency-based education, also designated competency-basedinstruction (CBI), competency-based learning (CBL), and compe-tency-based training (CBT), is paradigmatic of the continuing influ-ence of behaviourism in education (Skinner, 1976; Tyler, 1942, 1949)and an exemplar of the most recent preoccupation with objectives-based learning. Taking the term competence on board the behav-iourist agenda is a nicely gauged strategic move. Who would arguesensibly with a quest for competence? The critical evaluative ques-tion for CBE and the variations on its behaviourist theme, however,address the issue of whether measurable outcomes obtained fromnarrowly defined statements of objectives have much to do with ac-tual competent performance (with the ability to read, write, quan-tify, and experience the world as an autonomous individual).

Behaviourist ideology has been around for a long time, and yet thefamiliar complaints about the failures of public education which thelatest manifestations of behaviourism still claim to address remainwith us. A seemingly impressive focus on precision through meas-urable outcomes has as much to do with issues of management andcontrol, including the production of manageable evaluations for ad-ministrative purposes, as with a concern for education that fostersself-directed lifelong learning.

Here is how the emergence of CBE was described in a report of theUnited States Office of Education (1978):


Beginning in the 60s the term became popular for manytypes of education. We had competency-based readinginstruction, teacher education, etc. More recently, CBEhas acquired a more specific meaning and has becomeassociated with the acquisition of skills needed by thegeneral public to survive socially and economically in oursociety. (p. 1)

And in the same report, a prominent exponent of the model at thattime, William Spady, defines CBE in the following terms:

A data-based, adaptive, performance oriented set of in-tegrated processes that facilitate, measure, record andcertify within the complex of flexible time parametersthe demonstration of known, explicitly stated and agreedupon learning outcomes that reflect successful function-ing in life roles. (pp. 7–8)

To a large extent, CBE curriculum, including assessment formats,is neatly prepackaged into standardized modules.

In the twenty years that have elapsed since the USOE report, theCBE model of behaviourism has been embraced as a virtual pana-cea for many levels of education throughout the USA, Canada, andother countries. (In Canada, the Alberta Education Project, whichwas based on a profile of 103 competencies needed for adults livingin that province, and Saskatchewan’s province-wide adoption of CBEfor its community colleges and technical institutions are typical ex-amples.) Yet the concerns CBE claimed to address, tying togethereducation and training, are more pressing than ever if typical com-plaints by employers about the deficiencies of prospective employ-ees are given credence. What is important to note, in the context ofthis article, is how CBE language, in terms of its focus on controland management, aligns itself with the values of business and in-dustry. Thus, CBE discourse can be viewed as in line with govern-ment policy formulation, which places a high priority on educationand training to serve the needs of employers.

CBE formats and other variations on the behavioural objectivestheme which CBE exemplifies appeal to influential stakeholders whoexpect that the evaluations they authorize will provide data in nu-meric form to fit preordained and narrowly defined categories thatare amenable to measurement. Thus, CBE’s shortcomings have to


do with the artificial parameters, prescriptiveness and, from a criti-cal viewpoint, the absence of relevant stakeholder participation,which an excessively reductionistic approach to curriculum design,program development, and evaluation entails. Educational events,issues, and learning experiences that cannot be readily dealt within numerical form and assigned to preordained categories are notseriously taken into account. From a critical perspective, however,they may be disclosed as “significant absences” and as relevant tothe evaluative context.

With regard to the problem of excessive reductionism that is exem-plified in CBE, it is useful to cite John Dewey (1960): “Solution bythe method of partition is always unsatisfactory to minds with anambition for comprehensiveness” (p. 61); and “Standards and testsof validity are found in the consequences of overt action, not in whatis fixed prior to it and independent of it” (p. 73).

William James (1971), the twentieth century’s other prominent phi-losopher of pragmatism, who, like Dewey, was also concerned witheducation and the tasks of teachers, insisted that “out of no amountof discreteness can you manufacture the concrete” (p. 247).

And in his classic, The Meaning of Adult Education, EduardLindeman (1961) said: “The falsest view of life, as in the fable of theblind men and the elephant, is one which rests on some particularismas its point of reference” (p. 172). The pundits of CBE, and of thebehaviourist project it represents, offer us the narrow, drasticallyreduced certainty of the blindfold.

Critical evaluation of the behaviourist foundations of CBE, in par-ticular behaviourism’s spurious claims that align its project withthe natural sciences along with its ideological tendency to reject, onthe grounds of their subjectivity, meaningful reports of human ex-perience, is crucial. For CBE, along with behaviourist approachesin general, renders the process of educational evaluation a technicistdiscourse that sidelines relevant moral, practical, and political con-siderations. Invoking Marshall McLuhan (1967), “the medium” here(that is, the way data are defined, collected, assigned, and reported)“is the message” (of evaluation as a narrow technicist discourse).

The CBE model provides a high degree of certainty, or predictabil-ity, for needs assessment, curriculum formation, program develop-ment, and evaluation, that sustains existing power relations


(among teachers, students, and administrators, for example) anda conventional view of the way knowledge is transmitted from edu-cator as expert to student as passive recipient (Freire, 1981). YetCBE’s deterministic intent prevents the flexibility and creativevariations needed to take account of individual differences and foreffective continuing evaluation. Further, given the authority resid-ing in competency-based statements and definitive categories, com-bined with an evaluative focus on the deficiencies of the individuallearner, it is apparent that CBE precludes a level of meaningfulparticipation envisaged in practical and emancipatory discourseson education.

Recent variations of the CBE model, in attempts to overcome itsinability to address the problem of actual performance, now purportto deal with practical (subjective) dimensions of the learning expe-rience. However, it is still the prescriptive technical rationality ofbehaviourist underpinnings that shapes the evaluative process. Thistendency, in which the predominant steering influence of technicalrationality prevails, is still evident in any approach to curriculumdesign, program development, and evaluation that endeavours toeffect a rapprochement between technicist formulations such as CBEand practical approaches more concerned with qualitative investi-gations and greater stakeholder participation. The prevalence ofpreordained educational objectives and specific competency state-ments, and their steering effects, allows influential stakeholders toentertain a discourse on greater participation without jeopardizingtheir controlling interests.

PARTICIPATORY EVALUATION

For a critical perspective on educational evaluation, the extent andquality of stakeholder participation is of paramount concern. In anarticle on “Participatory Evaluation” Garaway (1995) acknowledgesthat the responsive evaluation approaches of Stake (1983) and Gubaand Lincoln (1992) “come closest to giving full consideration tostakeholder interests” but still “leave evaluation as an externallycontrolled process” (p. 85). The professional evaluator in responsiveevaluation at this level remains the primary investigator. ForGaraway, participatory evaluation in its various forms is concernedwith how it “can be part of an internalized process, an inward and agroup process dialogue.” He identifies a number of studies, includ-ing Cousins and Earl (1992), Brunner and Guzman (1989), andGreene (1988) which confirm the commitment to an approach in


educational evaluation where participants are significantly involvedin shaping the evaluative process and “the external evaluator goesbeyond being the primary investigator and participant observer tobecoming a facilitator” (Garaway, 1995, p. 87).

There are, Garaway suggests, a number of rationales for opting touse a participatory approach. These he addresses under the head-ings of Utilization, in which the criteria for choosing participants“center around not only potential for making use of findings, butalso having the power to do so” (p. 88); Empowerment, which “fo-cuses on including low status/low power stakeholders” (p. 86); Deci-sion Making, to bring about “participation that accurately representsthe decision making process and thus promotes representative deci-sions” (p.90); and Interactive Environment, where the aim is to pro-mote organizational learning.

In all of these situations, the issue of stakeholder interests and lev-els of involvement are problematized, not just taken for granted. Inthis regard, participatory evaluation as characterized by Garaway,and the sources on which he draws, are an advance on the respon-sive approaches of Guba and Lincoln and Stake. But issues of whogets to choose the participants and the significance of the privilegedrole of the professional evaluator as facilitator remain. Privilegingthe evaluator, even in this less-intrusive manner, poses a problemin the case of evaluation for empowerment if by participatory evalu-ation is meant an initiative that is engaged in an emancipatoryproject with the disempowered.

From a critical perspective on participatory evaluation, informed byHabermas’s theory of communicative action and Freirean pedagogy,experts, whether from within the evaluative context or brought infrom outside, would not direct or even assume overall responsibilityfor facilitating the project. The usefulness of the knowledge theybring to the dialogical process would have to be assessed accordingto collective decision making. Participatory evaluation in this veinis far easier to envisage than to enact, especially in formal educa-tional settings, but it does call for an evaluative process that is col-lective, dialogical, educative, and emancipatory. To a very largeextent, this form of participatory evaluation is closely aligned, asGaraway suggests, with the emancipatory aspirations of participa-tory research which likewise have to deal with the problems of therole of the expert within an “internalized process.” Richer (1998)describes how the issue of the expert’s role was addressed in a


schools-based participatory research project which included bothacademics and teachers on the team:

For us, then, democratizing the research process did notnecessarily mean the involvement of both teachers andacademics in every phase of the project. It did mean,though, that team members had to agree to whateverdivision of labor was arrived at. Further, everyone wasinformed about what was going on, had input to key de-cisions and was offered the opportunity to participate ineach research phase. (p.12)

Clearly, the discourse on participatory evaluation and, by the sametoken, participatory research is relevant to the emancipatory aspi-rations of critical pedagogy. Yet it would be a mistake to embraceparticipatory evaluation, in any of its forms, as the method for achiev-ing the emancipatory aspirations outlined by Freire and Hodgskin.Rather, from a critical perspective, it is more useful, and prudent,to foster participatory evaluations as relevant events at which thepolitics of emancipation can be more openly and systematically ad-dressed. At the same time, a critical perspective should alert us toways influential stakeholders, representing the interests of senioradministration for example, can co-opt the discourse on participa-tory evaluation as a means to legitimize its own predetermined po-litical agendas. This tendency is apparent where the senioradministration in educational settings remains outside the evalua-tive gaze while setting the parameters within which less empow-ered stakeholders are to participate. In such circumstances, thefindings, while imbued with a rhetoric on participatory evaluation,can be incorporated, manipulated, or ignored to suit the politicalaims of influential stakeholders who are not themselves account-able to the evaluative process.

SYSTEM AND LIFEWORLD

VanderPlaat’s insight about the dysfunctional effects of attempts tomethodize emancipatory initiatives is supported by her understand-ing of Habermas’s distinction between lifeworld and system inter-ests. The concepts and the distinction he makes between them haveimportant implications for critical pedagogy. Briefly stated, and atthe risk of oversimplification, the system represents the pervasivedimension of modern life in which the imperatives of technical effi-ciency and bureaucratic management hold sway. These imperatives


are viewed as colonizing the lifeworld which represents community-based values, sites for public debate and citizen participation, op-portunities for aesthetic and spiritual expression, and so on.Habermas (1984, 1987) shows how the effects of system imperatives(in the form of increasing legal, political, bureaucratic, and corpo-rate regulation) on education, family life, and other vital sites ofcultural reproduction make us aware of problems on the bordersseparating system and lifeworld.

Thus, the system/lifeworld distinction is helpful in drawing atten-tion to how initiatives such as CBE serve current bureaucratic andcorporate agendas at the expense of aspects of human experiencethat are vital to the quality of life. At the same time, in a borderzone between lifeworld and system, there are opportunities as wellas problems to be addressed from the perspective of critical peda-gogy. For example, the new technologies from which the Net andthe Web have emerged may be providing a means for ordinary peo-ple to gain access, share, and create knowledge in ways that en-hance dialogue and community values while, at the same time,undermining existing lifeworld locations which foster these values.In addition to defending the lifeworld (Welton, 1997), then, criticalpedagogy needs to discover proactive strategies for enlarging thelocations in which a careful assessment of the new communicationtechnology’s emancipatory potential makes sense.

For an emancipatory approach, VanderPlaat maintains that “theevaluative gaze should be from lifeworld to system” (1997, p. 150)reversing the conventional approach. The relationship betweenlifeworld and system is more dynamic and dialectical than is per-haps to be inferred from Habermas’s and VanderPlaat’s represen-tation. In viewing this relationship, where lifeworld values andsystem interconnect, a critical approach is interested in what as-pects of the lifeworld should be fostered and defended. However,the notion that the evaluative gaze should be described as originat-ing from lifeworld concerns rather than from immediate issues ofclass, gender, and race that call for strategic action characteristic ofthe system is questionable.

RATIONALITY, POSTMODERN SENSIBILITIES, AND REALPOLITIK

An important aspect of Habermas’s theoretical project for criticalpedagogy, and for a critical perspective on educational evaluation,resides in the way it lays out rational grounds for emancipatory prac-


tice. Technical rationality, exemplified in this article by CBE andother objectives-based approaches to program planning and evalua-tion, is limited and limiting at best. The level of rationality thatencompasses practical interests, and manifests itself in evaluativeefforts to gain an understanding of what is at stake in a particularsituation (meaning in context), seeks to avoid the steering and dis-torting effects of technical rationality. This form of rationality pro-vides theoretical grounds for the practically oriented approachesespoused by Guba and Lincoln (1992), Stake (1975), and Patton(1982). Emancipatory interests oriented toward non-distorted formsof communication and participatory decision making are sustainedby a more comprehensive form of rationality that incorporates bothtechnical and practical interests.

Thus, the emancipatory politics accompanying a critical approachto evaluation can derive theoretical foundations from the criticaltheory of Habermas. By the same token, Habermas’s theory of com-municative action (1984, 1987) can be invoked to show that the eman-cipatory pedagogy of Freire is a rationally grounded as well as ahopeful undertaking. It is in this rapprochement between the dia-logical and reflective orientation of Freire’s pedagogy and Habermas’stheory of communicative action, which highlights the pedagogicalsignificance of communicative competence, that the case for a criti-cal approach to education evaluation is grounded. The acquisitionof communicative competence, which prefigures an ideal speech situ-ation, emerges as collective experience in efforts toward genuine(non-distorted) participatory decision making. Problem-posing dia-logue provides a pedagogical context for the development of a criti-cal evaluative discourse.

Having located in the theoretical work of Habermas and the peda-gogy of Freire reasons why a critical approach to evaluation shouldbe adopted as an emancipatory project, it makes sense to addressquestions about representation which is a major preoccupation forthe postmodern discourse in critical pedagogy.

If the interests of the oppressed and dispossessed are central to theemancipatory project, it makes sense that their voices should be heardwithin evaluative and decision-making processes. However, a pri-mary ethical concern about representation, which is also central tothe theoretical project of Habermas and to Freire’s pedagogy, doesnot entail a purposeful avoidance of systematic efforts, flawed asthey may be in terms of adequate representation, to incorporate


participatory strategies within the systematic mandates of insti-tutional settings. To refrain from purposeful engagement with con-ventional curriculum development, program planning and evalua-tion, and policy formation initiatives, because they are situated, andlargely defined, within bureaucratized institutional mandates is toback away from strategic sites where emancipatory politics for amore just society can be most effectively enacted.

The discourse on representation referred to by VanderPlaat is in-formed by a postmodern analysis which focuses our attention onhow authoritative mandates establishing evaluative criteria in ourschools and other institutions are linked to the maintenance of ex-isting power relationships. In this regard, a postmodernist under-standing is in accord with a perspective emanating from criticaltheory. Unfortunately, while the insights of postmodern analysiseffectively expose, through a process of deconstruction, the false foun-dations on which doubtful claims for power and privilege are based,its deconstructionist agenda also serves to undermine both the ra-tional concerns of a theoretically informed critical pedagogy and therelevance of institutional contexts (publicly funded education, forexample) in which a reasonable case can be made for developingeducative strategies toward human emancipation.

An adequate account of the fundamental differences between a criti-cal evaluation which draws on the theoretical work of Habermasand insights provided by postmodern discourse is well beyond thescope of this article. However, the distinction does have to be made,because attempts to afford a rapprochement between relativisticpostmodern sensibilities, however insightful, and approaches toevaluation informed by critical theory can be dysfunctional. In par-ticular, the politics of difference engendered by postmodern discoursehas tended, somewhat perversely, to privilege the condition of mar-ginality and marginal locations in its support for disempoweredgroups. Rather than constituting a set of circumstances to beovercome through collective action, which is not distracted by a di-versionary politics of difference, the postmodern discourse on margin-ality defines for itself a disadvantageous location from which to viewand struggle for a more just distribution of resources. Further, whilea theoretically informed critical perspective on educational evalua-tion seeks validity in rationally grounded discourse and a moral/political commitment to the interests of the constituency identifiedin the work of Freire, for example, postmodern analysis, whether ornot its contemporary exponents acknowledge the historical connec-


tion, is linked to a Nietzschean worldview which would scorn suchan emancipatory endeavour.

It is Nietzsche’s courage, not his enthronement of difference and at-tacks on rational discourse that we need to adopt for a critical peda-gogy in these times.

A systematic critical approach to educational evaluation “within theconfines of pre-determined systemic mandates” (VanderPlaat, 1997,p. 157) is both feasible and necessary for the advancement of eman-cipatory politics. Acknowledging that evaluation is inevitably a po-litical undertaking, as with the educational process as a whole, acritical approach to educational evaluation entails purposeful sys-tematic engagement in realpolitik — and, at every opportunity, atthe center of things rather than on the margins. After all, the agen-das of the most powerful stakeholders are determined and advancedthrough strategic considerations of realpolitik. Fortunately, theseagendas, unavoidably characterized by contradictions that can bereadily identified from a critical perspective, are never completelyunassailable. There are always seams in the system, possibilities(however slim in times like these) for negotiation and meaningfulresistance, in which sensible strategies for emancipatory interestscan be explored and enacted. For example, the increasing publicityat this time about perceived indifference in Canada to the problemof poverty, relative to its UN rating as the country which affords itscitizens the highest quality of life in the world, presents a relevantopportunity to initiate an evaluative discourse that examines howcurriculum, educational programs, and policy formation addressesthe needs of children in poverty.

CONCLUSION

A critical approach to educational evaluation is still largely concernedwith the conventional practice of evaluation as intervention or me-diation in institutional settings. The scope of this concern extendsfrom individual assessments to encompass curriculum evaluation,institutional and community-based program evaluation, and policyanalysis. These evaluative contexts, steered predominantly by sys-tem imperatives, describe the terrain where the commitments of anemancipatory pedagogy can be relevantly engaged. In addition, acritical perspective serves to focus wider attention on how sociallearning processes (lifeworld values) are shaped by system impera-tives. (Educational evaluation can learn much from cultural studies


in this regard). At an institutional, system, level this critical under-standing can be advanced through critiques (meta-evaluations) ofcurrently implemented evaluation methods, models, processes, andpolicies.

Thus, the boundaries of a critical approach to evaluation are exten-sive and, given its emancipatory quest to foster human potentialthrough education, the staking out of preordained, neatly definedboundaries need not be viewed as a commendable aspiration. In thisregard, the philosopher Henri Bergson (1946) offers us a more real-istic understanding of what is entailed than the advocates of CBEand other deterministic objectives-based models of evaluation:

In life, a multitude of useless things are said, many su-perfluous gestures made, there are no sharply drawnsituations; nothing happens as simply or as completelyor as nicely as we should like; the scenes overlap; thingsneither begin nor end; there is no perfectly satisfyingending, nor absolutely decisive gesture, none of thosetelling words which give us pause; all effects are spoiled.Such is human life. (p. 249)

From this viewpoint, the continuing challenge for a critical perspec-tive on educational evaluation is to identify possibilities for, andimpediments to, the development of human competence as an eman-cipatory project. For, despite the seeming messiness of the practicalcircumstances of everyday life as depicted by Bergson, we are notwithout useful guidelines and signposts. While attempts to hold outcritical evaluation as a method are not helpful, a critical approachto educational evaluation can be, and should be, systematic. Anemancipatory project through education, and its relevant implica-tions for evaluation, are clearly discernible in the work of social theo-rists such as Habermas and emancipatory educators such as Freire.In the meantime, however, an overdetermined concern about im-portant issues of representation, and the inclusion of all potentiallyinterested stakeholders within the evaluative gaze, should not beelevated to the extent that it diminishes capacities for critical en-gagement in conventional evaluation initiatives. However flawed theprocess may be in regard to adequate representation and completeinclusion within the evaluative gaze (“nothing happens as simply oras completely or as nicely as we should like”), critical engagementin conventional evaluation initiatives holds out possibilities for anemancipatory politics of education.


A pressing challenge for a critically informed approach to educa-tional evaluation at this time concerns the future of our publiclyfunded education system. The critical evaluative discourse in edu-cation — focusing attention on what questions are posed and on howevaluations are designed and implemented — can draw relevantlyfrom a standpoint which endeavours to give reasons why continuingpublic support for education should not be eroded, from a commit-ment to the interests of the constituency identified in Freire’s peda-gogy, and from the aspirations held out by Hodgskin for the“education of a free people.”

REFERENCES

Bergson, H. (1946). The creative mind. New York: The Philosophical Li-brary.

Bloch. E. (1995). The principle of hope, vols. 1–3. Cambridge, MA: MITPress.

Bordieu, P., & Passeron, J. (1977). Reproduction in education, society, andculture. London: Sage.

Bowles, S., & Gintis, H. (1976). Schooling in capitalist America. London:Routledge.

Bruner, I., & Guzman, A. (1989). Participatory evaluation: A tool to assessprojects and empower people. In R. Conner & M. Hendricks (Eds.),International innovations in evaluation methodology (pp. 9–18). SanFrancisco: Jossey-Bass.

Collins, M. (1991). Adult education as vocation. New York: Routledge.

Collins, M. (1994). Putting ourselves into practice: New prospects for pro-gram planning and evaluation. In M. Brown (Ed.), A collection oforiginal essays on curriculum for the workplace (pp. 94–131). Victo-ria, Australia: Deakin University Press.

Collins, M. (1998). Critical crosscurrents in education. Malabar, FL: Krieger.

Cook, G. (1987). Educational justice for the campmen: Alfred Fitzpatrickand the foundation of Frontier College. In M. Welton (Ed.), Knowl-edge for the people. Toronto: OISE Press.


Cousins, J., & Earl, L. (1992). The case for participatory evaluation. Edu-cational Evaluation and Policy Analysis, 14, 397–418.

Dewey, J. (1960). The quest for certainty. New York: Putnam’s Sons.

Eisner, E. (1986). The art of educational evaluation: A personal view. Phila-delphia: Falmer.

Everitt, A. (1996). Developing critical evaluation. Evaluation: InternationalJournal of Theory, Research and Practice, 2(2), 173–188.

Fanon, F. (1988). The wretched of the earth. New York: Grove Press.

Farber, M. (1968). The foundations of phenomenology. New York: The StateUniversity Press.

Fetterman, D., Kaftarian, S., & Wandersman, A. (Eds.). (1996). Empower-ment evaluation. Thousand Oaks, CA: Sage.

Feyerabend, P. (1984). Against method. London: Verso.

Fischer, F. (1985). Critical evaluation of public policy: A methodologicalcase study. In J. Forester (Ed.), Critical theory and public life (pp.231–257). Cambridge, MA: MIT Press.

Freire, P. (1981). Pedagogy of the oppressed. New York: Continuum.

Freire, P. (1997). Pedagogy of hope: Reliving pedagogy of the oppressed.New York: Continuum.

Fukuyama, F. (1992). The end of history and the last man. New York: FreePress.

Garaway, G. (1995). Participatory evaluation. Studies in EducationalEvaluation, 21, 85–102.

Giroux, H. (1988). Teachers as intellectuals: Toward a critical pedagogy oflearning. Granby, MA: Bergin & Garvey.

Gramsci, A. (1971). Selections from the prison notebooks. New York: Inter-national.

Greene, J. (1988). Communication of results and utilization in participa-tory program evaluation. Evaluation and Program Planning, 11,341–351.


Guba, E., & Lincoln, Y. (1992). Effective evaluation. San Francisco: Jossey-Bass.

Habermas, J. (1984, 1987). The theory of communicative action, vols. 1 &2. Translated by T. McCarthy. Boston, MA: MIT Press.

Halévy, E. (1956). Thomas Hodgskin. London: Ernest Benn.

Hodgskin, T. (1963). Labour defended against the claims of capital. NewYork: August M. Kelley.

Holt, J. (1976). Instead of education. New York: Delta.

Illich, I. (1970). Deschooling society. New York: Harper.

James, W. (1971). Essays in radical empiricism and a pluralistic universe.New York: EP Dutton.

Kemmis, S. (1993). Foucault, Habermas and evaluation. Curriculum Stud-ies, 1(1), 35–54.

Lindeman, E. (1961). The meaning of adult education. Montreal: HarvestHall.

McLuhan, M. (1967). The medium is the message. New York: Bantam.

Patton, M. (1982). Practical evaluation. Beverly Hills, CA: Sage.

Richer, S. (1998). Explorations in participatory research: Towards equal-ity in the production of knowledge. 29th Annual Sorokin Lecture(monograph). Saskatoon: University of Saskatchewan.

Skinner, B. (1976). About behaviorism. New York: Vintage Books.

Stake, R. (1983). Program evaluation, especially responsive evaluation. InG. Madaus, M. Scriven, & D. Stufflebeam (Eds.), Evaluation models(pp. 287–310). Boston: Kluwer-Nijhoff.

Tyler, R. (1942). Basic principles of curriculum and instruction. Chicago:University of Chicago Press.

Tyler, R. (1949). General statement on evaluation. Journal of EducationalResearch, 35, 492–501.


United States Office of Education. (1978). Report of the USOE invitationalworkshop. Washington, DC: US Government Printing Office.

VanderPlaat, M. (1995). Beyond technique: Issues in evaluating for em-powerment. Evaluation, 1(1), 81–96.

VanderPlaat, M. (1997). Emancipatory politics, critical evaluation and gov-ernment policy. Canadian Journal of Program Evaluation, 12(2),143–162.

Welton, M. (Ed.) (1997). In defense of the lifeworld: Critical perspectives onadult learning. Albany, NY: SUNY Press.

CRITICAL PERSPECTIVES ON EDUCATIONAL … · approach to evaluation can be consistent ... concern...

Documents

Transcript of CRITICAL PERSPECTIVES ON EDUCATIONAL … · approach to evaluation can be consistent ... concern...