in

,

s, 11

30

ualadpromequathin

best solved. Consequently, critical thinking should not be just

Ennis, 1989, 1992; Paul, 1995). Creating instructional envi-

self-reported understandings of critical thinking.

tualize critical thinking beyond the dispositions and skills

Critical-thinking dispositions include truth-seeking, open-

Case, Coombs, & Daniels, 1999).Critical-thinking skills include taking into consideration

multiple perspectives, examining implications and conse-quences, resolving disagreements with reason and evidence,and re-evaluating a point of view in light of new information

* Corresponding author. Tel.: 35 799478030; fax: 35 722377950.E-mail address: cangeli@ucy.ac.cy (C. Angeli).

Learning and Instruction 19 (ronments for critical thinking raises a number of issues. Forexample, is it more appropriate to invest a significant portion ofeducational resources on teaching general critical-thinkingskills in isolation of subject matter or to contextualize instruc-tion through a knowledge base?

The aim of the present study was the examination of theextent to which the method of teaching general critical-thinking skills differentially affects (a) learners critical-thinking performance on an ill-defined issue, and (b) learners

mindedness, systematicity, analycity, maturity, inquisitiveness,and self-confidence (Facione & Facione, 1994). Mostresearchers also agree that dispositions toward critical thinkingplay an important role in critical-thinking performance(Facione, Facione, & Sanchez, 1994; McBride, Xiang, & Wit-tenburg, 2002; Perkins, Tishman, Ritchhart, Donis, & Andrade,2000), but research attention has mostly focused on the conceptof critical thinking as a set of thinking skills and how to developappropriate instructional processes for promoting them (Bailin,general education (Education Commission of the States, 2007;

another educational option, but rather an indispensable part of paradigm (Alston, 2001; Bailin, 2002; Papastephanou &

Angeli, 2007; Peters, 2005; Ten Dam & Volman, 2004).Keywords: Critical thinking; Ill-defined problems; Problem solving; Teaching approaches

1. Introduction

Critical-thinking skills are necessary for active citizenship inany pluralistic and democratic society, where citizens are dailyconfronted with tremendous amounts of information and ill-defined problems with real uncertainty as to how they can be

1.1. The concept of critical thinking

Critical thinking has mainly been conceptualized in terms ofdispositions and skills (Ennis, 1989; McPeck, 1990a; Paul,1995). Nonetheless, researchers in the literature have alsoacknowledged the existence of other perspectives that concep-Instructional effects on critical think

Charoula Angeli*

Department of Education, University of Cypru

Received 13 January 2008; revised

Abstract

Undergraduate students in dyads (N 72) were randomly and eqInfusion, and Immersion) and the control group. Students were initiallyinstruction, each dyads critical-thinking performance on an ill-definedof each dyad as covariate, indicated that the covariate and the teachingand the Immersion groups outperformed only the control group. Otherdifferent teaching groups exhibited diverse understandings of critical 2008 Elsevier Ltd. All rights reserved.0959-4752/$ - see front matter 2008 Elsevier Ltd. All rights reserved.doi:10.1016/j.learninstruc.2008.06.010king.g: Performance on ill-defined issues

Nicos Valanides

-13 Dramas street, Nicosia CY-1678, Cyprus

May 2008; accepted 16 June 2008

ly assigned to four groups, namely three teaching groups (General,ministered the California Critical Thinking Skills Test (CCTST). Afterblem was tested. A one-way ANCOVA, with the mean CCTST scorethod were significant. Post hoc comparisons showed that the Infusionntitative and qualitative analyses revealed that students assigned to the

2009) 322e334www.elsevier.com/locate/learninstruc

1997; Halpern, 1997; Kennedy, Fisher, & Ennis, 1991) arguedfined

problems for which critical thinking is indeed needed. Other

an(Paul, 1995). The present study focuses on the conceptuali-zation and teaching of critical thinking as a set of thinkingskills. According to Ennis (1989), critical thinking starts asa problem-solving process in a context of interacting with theworld and other people. Then, it continues as a reasoningprocess informed by background knowledge and previouslyacceptable conclusions, and it results in drawing a number ofinferences through induction, deduction, and value judging.Lastly, according to Ennis (1989), the critical-thinking processends in a decision about what to believe or do.

Similarly, Paul (1995) defined critical thinking as a set ofintegrated macro-logical skills, and McPeck (1990a, 1990b)characterized critical thinking as the ability to suspend judg-ment, or temporarily suspend judgment, until sufficientevidence is accumulated to establish the validity of a proposi-tion or action. The Delphi Report on critical thinking (Facione,1990a), which was a product of a Delphi research projectinvolving 46 experts in thinking, described critical thinking asa purposeful thinking process. During this process, a personforms a judgment of what to believe or do in a given context,and, in doing so, he or she uses a set of cognitive skills, suchas, analysis, interpretation, inference, explanation, evaluation,and self-regulation (Facione, 1990a).

In recent years, researchers have continued to mostly directtheir efforts toward thinking skills and how to find effectiveways for teaching them (Anderson, Howe, Soden, Halliday, &Low, 2001; Frijters, Ten Dam, & Rijlaarsdam, 2008; Halpern,2003; Kalman, 2002; Kuhn, 1999; Pithers & Soden, 2000;Tsui, 1999). The focal point of the most well-known debaterelating to the concept of critical thinking concentrates onwhether critical thinking should be viewed as generic acrossdifferent disciplines (Ennis, 1989; Paul, 1995) or discipline-specific (McPeck, 1981). Ennis (1989, 1992) argued thatknowing a field of study, understanding what constitutesa good reason in that field, and having a grasp of the fieldssemantic concepts do not sufficiently produce critical thinking.Instead, he concluded that general strategies are applicable tomany subjects, and argued that critical thinking is best taughtwith a content-free approach. Along the same line ofreasoning, Paul (1995: 372) also insisted that, the logics weuse, and which we are daily constructing and reconstructing,are far more mutable, less discrete, more general, more open-textured and multi-textured, more social, more dialectical, andeven more personaldand hence far less susceptible todomain-specific skills and concepts. We need to base ourmodel of the critical thinker, not on the domain-bound indi-vidual with subject-specific skills, but on the disciplinedgeneralist.

On the contrary, McPeck (1981: 19) argued against thenotion of critical thinking as general skills bound and statedthat, thinking, critical thinking, is always about some partic-ular thing or subject (let us call this thing X), and that ittherefore makes little or no sense to say, I teach thinking sim-pliciter, or I teach in general, but not about anything inparticular. According to McPeck (1981), teaching general

C. Angeli, N. Valanides / Learningskills in order to enhance someones critical thinking is useless,since there is no such thing as a general critical-thinking ability.researchers have examined the effects of specific instructionaltactics on critical thinking, such as, the role of cognitivedissonance (Browne & Freeman, 2000; Tsui, 1999), student-led seminars and role play (Dennick & Exley, 1998), dialogue(Frijters et al., 2008), academic controversy and cooperativelearning (Johnson & Johnson, 1993; Kaplan & Kies, 1994;Pithers & Soden, 2000). Garside (1996) also examined theextent to which group discussion can facilitate the develop-ment of critical-thinking skills more than traditional forms ofinstruction, such as lectures, and found no significant differ-ences between the two methods. In addition, Tynjalas (1998)research findings show that learning in a constructivist envi-ronment with an emphasis on examining different perspec-that critical thinking must be taught using authentic ill-deTherefore, if one is an effective critical thinker in one area, itdoes not mean that he or she will be an effective thinker in allother areas, because the skills and knowledge required for onearea are different from the skills and knowledge required foranother.

During the last years, this debate has faded away, sincemost researchers commonly accept the notion that somegeneral critical-thinking skills, which are widely applicable invarious contexts, exist, and that familiarity with a knowledgebase also plays an important role in thinking (Tsui, 1999; Voss,Perkins, & Segal, 1991). Thus, research focuses primarily onhow to develop effective instructional designs for teachinggeneral critical-thinking skills.

1.2. Critical-thinking skills and instructional approaches

Most studies on the teaching of critical thinking fall in oneof two categories, that is, the General approach with a focus onteaching general critical-thinking skills separately fromsubject matter, and the Infusion approach that calls for theteaching of general critical-thinking skills embedded insubject matter. An example of the General approach tothinking skills is the work of Whimbey and Lochhead (1986).According to this approach, thinking is a skill of know-howthat invokes general-purpose heuristics. Consequently, whatone needs to become an effective thinker is a set of generalheuristics that are likely to be effective in a variety of problemsituations, along with the meta-knowledge about situations inwhich specific heuristics are more appropriate. The Whimbeyand Lochhead (1986) approach advocates for students to beworking in pairs and thinking aloud, while they are trying tosolve problems. This way, students may learn something bylistening to what other people say and by exposing ones ownthought processes.

In the 1990s, the method of teaching critical thinking hasshifted from the General approach to the Infusion approach.This shift was based on the assumption that, through infusion,transfer to other domains can become possible (Prawat, 1991).For example, proponents of the Infusion approach (Brown,

323d Instruction 19 (2009) 322e334tives, free expression of ideas, and analysis from theperspective of students own experiences can enhance critical

made explicit during teaching with the Immersion approach. Inogue

where they are prompted to consider, analyze, and evaluate

potentially more effective than instruction with a focus onfrom

context-specific knowledge.

andifferent points of view.Obviously, there are many arguments both for and against

each one of the three approaches. For example, an advantageof the General approach is that it does not penalize studentswho do not have prior subject-matter knowledge. At the sametime, the approach does not guarantee that the generic skills ofcritical thinking will necessarily be transferred to differentcontent areas or out-of-school tasks (Resnick, 1987). TheInfusion approach integrates the teaching of explicitly statedgeneral critical-thinking skills with the teaching of a specificcontent domain and maintains an equal balance between thetwo. Advocates of the Infusion approach (Collins, Brown, &Newman, 1989) have different views about the most appro-priate starting point for teaching critical-thinking skills. Someargue that skills should be taught first, followed by a directapplication of the skills in a content domain, while othersargue that skills should be taught when the need arises (Pra-wat, 1991). Those who were skeptical about the Infusionapproach expressed a concern about diverting attention fromimportant curricular issues, and this concern about theimportance assigned to skills versus subject matter gave rise tothe Immersion approach. Ennis (1992), however, expressedconcern about the Immersion approach arguing that if generalcritical-thinking skills are not made explicit to students, thenstudents will not be able to draw generalizations, and thuscritical-thinking skills might not transfer from one domain toanother.

1.3. The present study

No research to our knowledge has set out to empiricallycompare the effects of the General, the Infusion, and theImmersion approaches on teaching critical-thinking skills. AsEnnis (1992) suggested more than a decade ago, and as TenDam and Volman (2004) recently stated, the literature oncritical thinking constitutes a rich theoretical body of knowl-edge about the conceptualization of critical thinking, but thefield is still missing a systematic body of research on theeffects of complete and detailed instructional designs aboutthe teaching of critical-thinking skills.

Most efforts related to the teaching of general cognitivethe Immersion approach, students are involved in dialthinking more easily than a traditional learning environmentwith an emphasis on grades. Recently, Frijters et al. (2008)found that within the domain of biology, a dialogic lessonseries, as compared to a non-dialogic, resulted in higher scoreson critical-thinking skills that were not gained at the expenseof subject-matter knowledge of biology.

In the 1990s, researchers also proposed a third approach toteaching critical thinking, namely, the Immersion approach(Prawat, 1991). In the Immersion approach, the most importantresource or tool in promoting thought is the role of ideasdnotthinking skills. Concomitantly, critical-thinking skills are not

324 C. Angeli, N. Valanides / Learningskills have been implemented in the form of courses isolatedfrom subject matter (Nickerson, Perkins, & Smith, 1985), with1.3.1. Research questionsdHypothesesThe present study set out to empirically investigate the

effects of the General, the Infusion, and the Immersionteaching approaches on learners critical-thinking perfor-mance, as compared to a control condition in which studentsreceived no instruction. In particular, the study was designedto seek answers to the following research questions: (a) Doesthe method of teaching general critical-thinking skills, that is,particular versions of the tactics that are used in the General,the Infusion, and the Immersion teaching approaches, and thecontrol group, differentially affect learners critical-thinkingperformance on an ill-defined issue? (b) Does the method ofteaching general critical-thinking skills differentially affectlearners self-reported understandings of critical thinking?

It was hypothesized that students in the Infusion andImmersion groups will outperform students in the General andthe control groups in terms of their critical-thinking perfor-mance about an ill-defined problem. Also, students in theInfusion group will outperform those in the Immersion group,and students in the General group will outperform those in thecontrol group (Hypothesis 1).

Furthermore, it was hypothesized that the Infusionapproach will be more effective than the General and theImmersion approaches as well as the control condition ininducing correct understandings of critical thinking. Similarly,it was hypothesized that the General and the Immersionapproaches will also be more effective than the controlcondition in inducing correct understandings of criticalthinking. Additionally, the Immersion approach will be moreeffective than the General approach and less effective than theInfusion approach in inducing correct understandings of crit-ical thinking (Hypothesis 2).

2. Method

2.1. Participants

A total of 160 undergraduates volunteered to participate inthe study. Of the 160 students, data from 16 of them were usedto pilot-test the research materials in order to clarify themeaning of certain expressions as well as to determine theduration of the research procedures. The remaining 144students participated in the study. Their mean age was 19.81years (SD 1.94). Of the 144 participants, 36 (25%) of themteaching general critical-thinking skills in isolationthe exception of a few studies that successfully managed toteach general thinking skills in contextualized ways (e.g.,Perkins, 1986; Valanides, 1990). It is believed that this latterdirection about the teaching of general thinking skills incontextualized ways is promising and might be the way tofollow to adequately prepare critical thinkers. Accordingly,contextualized critical-thinking skills instruction may be

d Instruction 19 (2009) 322e334were freshmen, 79 (54.86%) were sophomores, 19 (13.19%)were juniors, and 10 (6.94%) were seniors. Also, 100

(69.44%) of the participants were females and 44 (30.56%)were males. Lastly, 97 (67.36%) majored in education, and theremaining 47 (32.64%) in other areas, such as psychology,computer science, business, and political science. None of theparticipants had any previous experience with the teaching ofcritical-thinking skills.

2.2. Design

Each student participated in three sessions, namely, Sessions I,II, and III. The procedures for the three sessions are summarizedin Table 1. As shown in Table 1, the procedures for Sessions I andIII, but not for Session II, were the same for all participants.

Table 1

Design of the study and time (min) devoted to sessions (total 225 min).

Time

Session I (85 min)

1. Introduction to Session I 5

2. Demographics Questionnaire 15

3. California Critical Thinking Skills Test 45

4. Participants individually read and summarized the issue Are

American values shaped by the mass media?

20

Session II (80 min)

1. Introduction to Session II 5

2. Each student in each dyad reviewed his/her summary of the

issue Are American values shaped by the mass media?

10

3. Teaching method (each one 65 min)

General

Lecture 35

Each dyad discussed the issue and prepared a joint

outline for a paper on the issue

30

Infusion

Each dyad discussed the issue and started preparing a

joint outline for a paper on the issue

15

Students in each dyad reflected on their thinking 10

Short lecture 10

Dialogue with the researcher 15

Each dyad completed a joint outline for a paper on the issue 15

Immersion

Each dyad discussed the issue and began to prepare 15

65

4. All students

C. Angeli, N. Valanides / Learning anStudents were given the issue Should drugs be legalized?

to read and summarize at home

Session III (60 min)

1. Introduction to Session III 5

2. Each student in each dyad reviewed his/her summary of

the issue Should drugs be legalized?

10

3. Each dyad discussed the issue Should drugs be legalized? 30Each dyad discussed the issue and prepared a joint

outline for a paper on the issuea joint outline for a paper on the issue

Students in each dyad reflected on their thinking 10

Socratic questioning with the researcher 25

Each dyad completed a joint outline for a paper on the issue 15

Controland prepared a joint outline for a paper on the issue

4. Student evaluations 152.2.1. Session IFor Session I, participants were divided into six groups of

24 students each. The first author introduced students to thepurpose of the study. Students in all four groups were told:

Our interest focuses on your thinking about some issues.Specifically, we are interested in critical thinking; that is, inthe nature of critical thinking and how we can promote iteffectively. We feel that critical thinking is very importantin our life. We have seen so much thinking that is biased,distorted, unquestioned, and even down-right prejudiced. Yetthe quality of our life, and the growth of our society, dependsprecisely on the quality of our thinking. Shoddy thinking iscostly, both in terms of money and quality of life.

After that, participants individually filled out a question-naire for demographic information and took the CaliforniaCritical Thinking Skills Test (CCTST; Facione, 1990b).Students were subsequently given a one-page text about thedebate surrounding the issue Are American values shaped bythe mass media? The text presented the opposing views oftwo experts on the specific issue. The first expert arguedsuccinctly in 300 words that American values are shaped bythe mass media and accordingly he presented three argumentsto support his position. Similarly, the second expert argued,also in 300 words, that American values are not shaped by themass media and he also presented three arguments to supporthis position. Students were asked to individually read andsummarize the text, and turn in their summaries. The proce-dures for Session I lasted 85 min.

Following Session I, students were randomly assigned into72 dyads. The dyads were then randomly divided into threeteaching groups and a control group of 18 dyads each. Eachgroup was randomly assigned to a different teaching method(versions of the General, the Infusion, and the Immersionapproaches) or the control group. The decision to groupstudents in dyads was based on the rationale that thinking withothers, as opposed to thinking alone, about a controversial ill-defined issue would trigger more discussion and thus richerinformation (data) for analysis.

2.2.2. Session IIFor each of the 72 dyads, meetings for Sessions II and III

were scheduled on 2 consecutive days and at times convenientfor the participants. Thus, data collection for Sessions II andIII was done in parallel and was completed in 4 months.Sessions II and III were facilitated by the same researcher(first author) with only one dyad at a time. During Session II,the researcher reminded the students of the issue AreAmerican values shaped by the mass media? they read andsummarized in Session I, and each student was given back hisor her summary from Session I to review. Students summarieswere not graded by the researchers and no specific feedbackabout them was provided. Students were then informed that,during Session II, they would work together in their dyads todiscuss the same issue and produce an outline for a paper

325d Instruction 19 (2009) 322e334representing their joint position on the issue. Subsequently,each dyad, except those in the control group, was taught about

angeneral critical-thinking skills with one of the three teachingmethods, namely, versions of the General, the Infusion, or theImmersion approach.

As shown in Table 1, the duration of each teaching inter-vention was 65 min. Students across the four groups worked indyads and were given the same instructions of how to tacklethe same ill-defined issue. The instructions were given to themin writing so that students by themselves could refer to them,when they were feeling unsure about their task, or the facili-tator could ask the students to read the instructions again,when they were not following them duly. Indicatively, thewritten instructions in Session II stated:

I would like the two of you to work together and developa common point of view about the impact of the media onsocietys values. I want you to develop a two-page outlinefor a paper that presents your joint position. Remember, weare interested in seeing your best thinking about theseissues. It is very important for you to do your best inanalyzing these issues and coming to a position. Please besure to raise and generate reasons and arguments that youthink are important, and be sure to try to understand,evaluate, and resolve differences in points of view. Doallow yourself to enjoy the discussion and be open to theprospect of learning from one another.

The three teaching methods differed only in the approachthey adopted for teaching five general critical-thinking skills,namely: (a) analyze the problem, (b) generate solutions, (c)develop the reasoning for each solution, (d) decide which is thebest solution, and (e) use criteria to evaluate ones thinking.These five general critical-thinking skills were chosen, becausethey reflect the characterization of critical thinking, as it isdescribed in the Delphi definition (Facione, 1990a).

In the General group, a content-free teaching strategy wasused, because it attempted to teach critical-thinking skillsseparately from the presentation of any content. A 35-minlecture in videotaped format was used to explicitly deliver vialecture mode the five general critical-thinking skills. Duringthe 35-min videotaped lecture, tactics, such as presentation,examples, practice of the first three critical-thinking skillsbefore teaching the last two, and review, were used to teachparticipants the skills of a general critical-thinking process.There was no interaction between the researcher and thestudents regarding the content of the videotaped lesson, and nofeedback was given to the students about their critical-thinkingperformance. At the end of the 35-min lecture, the researcherinstructed the students in each dyad to work together for30 min and develop a common point of view on the issue AreAmerican values shaped by the mass media?

In the Infusion group, each dyad was initially given 15 minto start developing a joint position on the issue Are Americanvalues shaped by the mass media? Then, the researcherinterrupted the students, and asked them to reflect on theirjoint thinking process for about 10 min and to write down ona piece of paper their rules for good critical thinking. Next, the

326 C. Angeli, N. Valanides / Learningresearcher presented each dyad with a 10-min videotapedlecture about the five general critical-thinking skills. Theinstructor on the videotape for the Infusion group was thesame instructor as the one on the video for the General group.The 10-min videotaped lecture was a shorter version of the 35-min lecture on the five general critical-thinking skills. Themain difference between the two lectures was that only in the35-min lecture students were asked to practice the first threecritical-thinking skills with different practice examplesprovided by the instructor on the videotape, before learningabout the last two critical-thinking skills. After the 10-minlecture, the researcher engaged each dyad in a 15-min criticaldialogue of comparing and contrasting the skills presented tothem with those they had earlier written down. The purpose ofthis interaction was to involve students in a process ofreflection and evaluation of their thinking. Finally, each dyadwas allowed an additional 15 min to complete the outline.Thus, in the Infusion group, participants were encouraged tothink about critical-thinking skills within the presentation ofa controversial real-world issue. In contrast to the Generalgroup, the researcher in the Infusion group guided studentlearning by first asking students to write down their owncritical-thinking skills, and then to compare and contrast theircritical-thinking skills with those presented to them.

In the Immersion group, each dyad was also given 15 minat the beginning to think about the same issue Are Americanvalues shaped by the mass media? and start developing anoutline for a paper based on their joint position on the issue.Then, the researcher asked them to reflect on their jointthinking process for about 10 min and to write down ona piece of paper their rules for good critical thinking. There-upon, through questioning that lasted for about 25 min, theresearcher challenged participants to re-think their rules aboutgood critical thinking, and re-evaluate their reasoning and theway they developed their point of view. The goal of thediscussion was to have students clarify, analyze, and evaluatethoughts and perspectives, compare and contrast differentpoints of view, and stimulate their thinking. At the end, eachdyad took an additional 15 min to complete the outline. Thus,the five general critical-thinking skills were never madeexplicit to students, as it was the case with the other twoteaching methods, but students were left to infer them byresponding to relevant questions such as: Have you analyzedthe problem in depth?, What are your reasons for support-ing this point of view?, Are there different perspectives onthe issue? Hence, the Immersion teaching method wasa variation of the Infusion method, because participants wereagain guided to think about critical-thinking skills within thepresentation of a controversial real-world issue, but the fivecritical-thinking skills were not explicitly stated.

Students in the control group were first given the samewritten instructions, as those given to the students in theGeneral, Infusion, and Immersion groups, about how to tacklethe ill-defined issue Are American values shaped by the massmedia?, and were then engaged in problem solving withoutreceiving any instruction or feedback from the researcher.They were only told that they had 65 min to prepare an outline

d Instruction 19 (2009) 322e334for a paper that presented the dyads joint position on the issueAre American values shaped by the mass media?

At the end of Session II, each participant from the three

in the study, namely, (a) What is your understanding of

et al., 1994). But, given the concerns about variations in theo-inter-

C. Angeli, N. Valanides / Learning ancritical thinking after participating in Sessions II and III?and, (b) What were the difficulties that you encountered?Session III procedures were completed in 60 min.

A month after the intervention, four dyads from each of thethree teaching groups and the control group were randomlyselected (32 participants from 16 different dyads) and partic-ipated in a debriefing session for the purpose of sharing theirthoughts and reflections about the intervention with otherparticipants and the researcher. The session was videotapedand later transcribed.

2.3. Instruments

2.3.1. California critical thinking skills test (CCTST)The CCTST is a 34-item multiple choice test which targets

general critical-thinking skills essential in college education(Facione, 1990b; Facione & Facione, 1994). No discipline-specific college level content knowledge is presumed on theCCTST. The items are given without any technical vocabulary orcritical-thinking jargon, and they are set in contexts that arefamiliar to college students. The CCTST is available in twoequivalent forms: Form A, which was used in this study, andForm B. The CCTST reports six scores: an overall score on onescritical-thinking skills, and five scores on subscales which are (a)Analysis; (b) Evaluation; (c) Inference; (d)DeductiveReasoning;and (e) Inductive Reasoning. The internal consistency reliability(KudereRichardson 20) of the CCTST for Form A is KR-20 0.70.1 In terms of concurrent validity, the CCTST corre-lated with the scores of Grade Point Average (GPA; r 0.20),Scholastic Aptitude Test-verbal (SAT-verbal; r 0.55), andScholastic Aptitude Test-math (SAT-math; r 0.44) (Facioneteaching groups and the control group was given a new text toread and summarize at home before coming to Session III.Students were specifically instructed to read and summarizethe text alone and to refrain from discussing the issue withothers. The new one-page text presented the opposingperspectives of two experts on the issue Should drugs belegalized? As it was the case with the first issue AreAmerican values shaped by the mass media?, the first expertargued succinctly in 300 words that drugs should be legalizedand based his position on three arguments that he presentedand discussed. Accordingly, the second expert argued also in300 words that drugs should not be legalized, and also pre-sented and discussed three arguments to support his position.

2.2.3. Session IIIIn Session III, students in each dyad were instructed by the

researcher to work together, discuss the new issue Shoulddrugs be legalized?, and then produce an outline for a paperthat presented their joint position on the issue. Subsequently,students answered two questions relating to their participation1 According to Facione and Facione (1994), for an instrument of this kind,

a KR-20 between 0.65 and 0.75 is sufficient.2.3.2. Critical-thinking performanceStudents critical-thinking performance was measured with

a rubric that was constructed to assess the 72 outlines preparedby the dyads in Session III for the issue Should drugs belegalized? The rubric was constructed inductively using theconstant comparative analysis method, which constitutes thecore of qualitative analysis in the grounded-theory approachdeveloped by Glaser and Strauss (1967) (Strauss & Corbin,1990). The constant comparative method was carried out inthree phases: (a) comparison within a single outline, (b)comparison among outlines within the same group (i.e.,General, Infusion, Immersion, control), and (c) comparison ofoutlines from different groups. The analysis led to four criteriafor discriminating between the different levels of the scoringrubric. The four criteria used in the scoring regarded whetherstudents in each dyad (a) reasoned clearly within a point ofview, (b) discussed the issue from different perspectives, (c)identified pros and cons for each perspective, and (d)explained with reasons and evidence which perspective theyconsidered as the best. Consequently, the scoring rubric wasformed as shown in Appendix A.

The critical-thinking scoring rubric had five mutually exclu-sive levels, and participants scores ranged from 1 (low perfor-mance) to 5 (high performance). An independent rater wastrained on the scoring rubric and assessed 13 (72.22%) outlinesfrom each group, which were randomly selected by theresearchers, in order to determine the interrater reliability. Pear-sons correlation between the rating of the independent rater andthat of the researchers was calculated and was found satisfactory,r 0.85. The rater and researchers also discussed the observeddisagreements and easily resolved the existing differences.

2.3.3. Understanding of critical thinkingTwo independent raters performed a qualitative analysis

(Patton, 2002) of students responses to the two questions ofthe evaluation questionnaire, which was completed at the endof Session III. The qualitative analysis revealed four categoriesof student understandings. In particular, participants exhibitedprecise understandings of critical thinking, such as A criticalthinker evaluates different perspectives and A criticalthinker is systematically engaged in the thinking process.They also developed erroneous or imprecise understandings,such as A critical thinker compromises to reach a decisionand A critical thinker collaborates and listens to others. Theinterrater agreement was found to be 0.89.

3. Results

3.1. Effects of teaching methods on studentscritical-thinking performanceretical constructs, such correlations should be cautiouslypreted. In the present study, KR-20 was found to be 0.78.327d Instruction 19 (2009) 322e334Table 2 shows descriptive statistics (means andstandard deviations) of students CCTST scores and their

students in the Immersion group, as compared to those in theControl group, was Cohens d 0.99. This effect size,although smaller than the previous one, was also very highindicating that the critical-thinking performance of the averagedyad in the Immersion group was at 0.99 standard deviationsabove the mean critical-thinking performance in the controlgroup. Effect size statistics should be however interpreted

results of these comparisons showed that students assigned toluatentrol

group. Significant differences between the control group and

Table 2

Means and SD of CCTST scores and critical-thinking performance of

students dyads per teaching method.

Teaching method CCTST scores Critical-thinking performance

M SD Dyads (N) M SD Dyads (N )

General 15.89 5.56 18 3.11 1.45 18

Infusion 15.89 4.65 18 4.00 1.46 18

Immersion 15.94 4.45 18 3.83 1.42 18

328 C. Angeli, N. Valanides / Learning and Instruction 19 (2009) 322e334critical-thinking performance regarding the ill-defined issue(Should drugs be legalized?) for the three teaching groupsand the control group. The CCTST score of each dyad is theaverage value of the individual scores in each dyad.

A one-way analysis of covariance, with teaching method asthe independent variable, the average value of the individualscores in each dyad on the CCTST as the covariate, and crit-ical-thinking performance as the dependent variable, wasperformed. The covariate was found to be significant, F(1,67) 7.56, P< 0.01, partial h2 0.10, indicating that dyadswith better CCTST scores had better critical-thinking perfor-mance than dyads with lower CCTST scores. The results alsoindicated that, after partialling out the effects of the covariateon the dependent variable, the differences in critical-thinkingperformance among the four groups were statistically signifi-cant, F(3, 67) 4.30, P< 0.01, partial h2 0.16.

Post hoc comparisons, using the Bonferroni procedure,were subsequently performed and revealed that studentsassigned to the Infusion and Immersion groups significantlyoutperformed those assigned to the control group (P< 0.05).There was no other significant difference between theperformances of students assigned to the Infusion andImmersion groups or between the performances of studentsassigned to any other two groups.

The magnitude of the superior critical-thinking perfor-mance of students in the Infusion group in comparison withstudents performance in the control group was estimatedusing Cohens d effect size, which is the degree of meandifference between the first (Infusion) and second group(control) of students relative to, or divided by, the pooledstandard deviation (SD 1.42) (Cohen, 1988). The effect size(Cohens d 1.10) was very high indicating that the critical-thinking performance of the average dyad in the Infusion

Control 14.75 4.60 18 2.44 1.38 18

Total 15.62 4.76 72 3.35 1.53 72group was at 1.10 standard deviations above the mean critical-thinking performance in the control group. Similarly, themagnitude of the superior critical-thinking performance of

Table 3

Frequencies of students understandings of critical thinking.

Teac

Gen

A critical thinker evaluates different perspectives 18

A critical thinker is systematically engaged in the thinking process 22

A critical thinker compromises to reach a decision 9

A critical thinker collaborates and listens to others 9the General, Infusion, and Immersion groups were found,c2(1, N 25) 7.41, P< 0.01, c2(1, N 33) 20.20,P< 0.001, and c2(1, N 33) 20.20, P< 0.001, respectively.

hing method

eral Infusion Immersion Control Total

26 26 7 77

15 8 5 50the three teaching groups were more inclined to evadifferent perspectives than students assigned to the cocautiously taking into consideration possible departures fromnormality, especially when sample sizes are small (Feingold,1992; Wilcox, 1995), as it was the case in the present study.

3.2. Effects of teaching methods on studentscritical-thinking understanding

As shown in Table 3, the four categories of understandingcritical thinking were expressed in different frequencies acrossthe three teaching groups and the control group. In the threeteaching groups, it was explicitly stated or strongly impliedthat a critical thinker takes into consideration different pointsof view and systematically evaluates diverse ideas. From thisperspective, a critical thinker often has to collaborate andlisten to what others have to say for the purpose of becomingbetter informed. However, collaborating and listening to othersalone is not enough and it should not be understood as an actfor reaching a compromise; but, it should be understood as anact for becoming better informed, before thinking further anddeciding about something.

To determine the significance of the differences in the fourcategories of understanding critical thinking across the threeteaching groups and the control group c2 statistics were used.In all cases they were significant, for A critical thinkerevaluates different perspectives, c2(3, N 77) 27.10,P< 0.01; for A critical thinker is systematically engaged inthe thinking process, c2(3, N 50) 21.20, P< 0.01; for Acritical thinker compromises to reach a decision, c2(3,N 23) 9.68, P< 0.01, and for A critical thinker collab-orates and listens to others, c2(3, N 46) 17.76, P< 0.01.Subsequently, pairwise comparisons were carried out todetermine significant between-group differences in the fourcategories of student understandings of critical thinking. The0 7 7 23

11 5 21 46

Clearly, students in the General, the Infusion, and theImmersion groups correctly understood that a critical thinkerevaluates different perspectives; however, this was not the casewith the students of the control group.

Significant differences between the control group and thethree teaching groups were also observed for the category A

threences

among them in terms of this false impression, and they were

indicated that the existing differences amongst the threeteaching groups and the control group were nonsignificant andthus no further analyses were conducted.

3.4. Students reflections

329C. Angeli, N. Valanides / Learning and Instruction 19 (2009) 322e334less inclined to think about collaborating and listening toothers as a critical-thinking skill.

The results also showed significant differences between thecontrol group and the General or the Infusion group for thecategory A critical thinker is systematically engaged in thethinking process, c2(1, N 27) 17.13, P< 0.001, andc2(1, N 20) 6.92, P< 0.01, for the General and Infusiongroups, respectively. Similarly, a significant difference wasfound between the General group and the Immersion group forthe same category as above, c2(1, N 30) 11.20, P< 0.01.This finding is justified by the fact that only in the General andthe Infusion groups it was explicated that critical thinking isa systematic thinking process. Moreover, none of the studentsassigned to the Infusion approach developed the misunder-standing that a critical thinker should compromise to reacha decision, while several students from the other three groupsdid so.

3.3. Difficulties encountered

Regarding the difficulties students encountered duringSessions II and III, three main themes emerged, namely: (a)lack of evidence, knowledge, and information to back upa point of view; (b) difficulty in putting beliefs and biasesaside, and examining the other side of the issue; and (c)difficulty, at times, in communicating ideas and resolvingdifferences. Interrater agreement was found to be 0.92. Thesedifficulties were expressed with different frequencies acrossthe four groups as shown in Table 4. A c2 analysis, however,

Table 4

Observed frequencies of learners perceived difficulties.

Lack of evidence, knowledge, and information to back up a point of viewto others. In contrast, students assigned to the otherteaching groups did not exhibit any significant differecritical thinker collaborates and listens to others, c2(1,N 30) 8.23, P< 0.01, c2(1, N 32) 5.63, P< 0.05, andc2(1, N 26) 15.41, P< 0.001, for the General, Infusion,and Immersion groups, respectively. These significant differ-ences indicate that students in the control group were morelikely to think that a critical thinker collaborates and listens toothers than students assigned to the other three teachinggroups. Students in the control group, who received neitherinstruction nor guidance from the researcher, thought that thegoal of the intervention was to learn to collaborate and listenDifficulty in putting beliefs and biases aside, and examining the other side of the

Difficulty, at times, in communicating ideas and resolving differencesTeaching method

General Infusion Immersion Control Total

8 3 7 7 25As it was previously mentioned, 1 month after the inter-vention the researcher hosted a debriefing session in whichfour dyads from each of the four groups participated (32research participants from 16 different dyads), and expressedtheir thoughts and reflections about the intervention.

Students from the Immersion group strongly stated thatduring the intervention they felt frustrated at times, becausethe researcher was not giving them the correct answers oranswers to questions they were asking. As a student clearlystated:

The biggest frustration, I felt during the study, was notknowing if what we were doing was right. We had to comeup with the answers on our own (S15).

This initial puzzlement turned out to be however the biggestsatisfaction not only for student S15, but for almost everystudent, as indicated by the excerpts from the reflections ofthree of them. In these cases, students stated that the inter-vention was satisfying and proved to be useful for improvingtheir ability to write better papers and get higher grades intheir courses:

Even though this was my biggest frustration, it led to mybiggest satisfaction. It was satisfying to know at the endthat my partner and I came up with the answers on ourown.Even though, I didnt have complete mastery ofcritical thinking; I thought I had vastly improved from whatI used to know (S15).

The most valuable thing that I learned was that it isimportant to toughly think things through from all per-spectives.I believe that teaching by letting people figurethings out on their own is very good.You asked us simplequestions that triggered explosive thinking about the topicarea (S34).

The biggest satisfaction is that I am able to do papers a lotbetter now. The intervention helped me out a lot. To behonest, I have gotten better grades on my papers since theintervention. My grades have increased tremendously.I think I owe it all to the intervention (S52).

Students from the Infusion group compared the quality ofthe outlines they prepared in Session II with those theyissue 6 7 5 5 23

7 11 16 10 44

anprepared in Session III. They stated that after they watched thevideotape with the critical-thinking skills in Session II, theyrealized that their thinking exhibited in the outline about thefirst ill-defined issue reflected mostly biased thinking, sincethey never really examined the opposite point of view. Theycandidly admitted that they gained insights about thedynamics involved in the critical-thinking process. They alsosuggested that the specific experience would be helpful tothem throughout their lives, and that they would use the sametactics to teach their own students how to think critically.Some representative examples of these ideas are illustrated inthe following excerpts:

I was surprised to realize that I had not really been thinkingclearly about the critical-thinking process.I feel that I willbe using the critical-thinking process a lot more now thatI understand the dynamics involved. I also think that I mayuse this type of process as an example of how my futurestudents should really think when writing an argumentativeor persuasive paper. I learned more about myself frombeing a participant in this study (S77).

In this case, however, I learned many things that I believewill be helpful throughout my life. This interventionbrought me to the realization that I had a very vulnerableflaw in my critical-thinking skills. What I took away fromthe study was a valuable way to interpret two separate sideson an issue and to be able to use both in the evaluation ofmy final thought process (S121).

Furthermore, there were also students in the Infusion groupwho compared and contrasted their experience from partici-pating in this study with their experiences from other inter-ventions. Student S98 eloquently stated:

I wanted to touch upon how this experience was valuable forme. I volunteer for an organization called the VictimOffenderReconciliation Program. This program is an alternative to thepenal system that brings the victim and offender of a crimetogether so they canmake things right between them again. Asa mediator, I must meet with both individuals separately andlisten to each of their stories. This involves a great deal ofopen-mindedness which is a critical-thinking skill. Talk abouta lesson in critical thinking!.This was an important study. Isay this because I was able to take something away from it. Ihave participated in many psychology studies but have neverreally gotten anything from them. They just had me takea series of battery and IQ tests and tried to predict mypersonality type or something. Such interventions really donthelp the participant very much.

The same student went on to explain why the study wasdifferent from other studies she participated in, and explainedwhy she considered this study to be beneficial for her andother participants as well:

But this study was different - it taught you something aboutyourself. It made you realize things you may not have

330 C. Angeli, N. Valanides / Learningnoticed before about the way you think. This, in my mind,makes for a valuable and worthwhile intervention. Theresearcher will be able to use these results to actually helppeople. Perhaps the data she collects could be used toimplement different strategies in the classroom to helpstudents improve their critical-thinking skills (S98).

While students in the Immersion and Infusion groups werevery enthusiastic and very talkative about their participation inthe study, students in the General group were very skepticaland expressed disappointment about their overall performance.They also attributed their poor performance to their previouseducational experiences. This was clearly stated in the excerptfrom student S36:

I am really disappointed about my performance, but if Icould somehow explain why I performed so poorly I wouldsay this. One of the problems is that often we are taught atan early age never to address the other side of our argu-ments in papers. We are told that this only weakens ourstand and makes us look as if we are not sure of ourselvesor that we are undecided. I dont think that a paper shouldsimply be a thesis and then three facts supporting thisthesis. Yet unfortunately, too often, all teachers want isa simple thesis and three supporting facts.

The majority of the students indirectly stated that theGeneral approach did not provide enough help and scaffoldingfor critically examining whether drugs should be legalized,and that only after the debriefing session they realized whatthey needed to do. Indicative examples of these ideas are theexcerpts from students S57 and S88:

I guessed I did not realize that I needed to examine allpossible points of view. But, I see now how important thatwas. I wished I realized that earlier, so I could practice theskill in Session III. I also wished that the researcher wouldsay something when I was not really thinking critically aboutthe issue, but I do understand that she could not do that (S57).

Now, that I know how badly I performed in the interven-tion, I feel more disappointed about it. However, I am gladthat we are having this discussion today, since I am nowrealizing what I needed to do. Better late than never (S88).

Finally, the students from the control group stated that theythought the goal of the intervention was to learn to collaborateand listen to others. They were in most cases disappointedwith their performance and expressed complaints relating totheir educational experiences, and the lack of opportunitiesthroughout high school and college to develop critical-thinking skills that are lost amidst all the commotion in thelife of a typical student according to student S42:

Now that I know what the intervention was all about, I believethat if I had had a brief overview of what critical thinkingmeant, I might have been more successful in my task. I thinkthatmuch of the education throughout high school and collegeis lost due to the constant hustle and necessity to focus on thespecific tasks at hand. Critical-thinking skills are easy to lose

d Instruction 19 (2009) 322e334amidst all the commotion in the life of a typical student. I, forone, feel that I have lost the skill to think critically.

was differentiated depending upon the kinds of learningexperiences in which they were engaged. The findings

anpartially supported Hypothesis 1, while they supportedHypothesis 2 for the most part. Regarding Hypothesis 1,students assigned to the Infusion and Immersion groups hadsignificantly better critical-thinking performance on the ill-defined issue Should drugs be legalized? than studentsassigned to the control group. There were no other significantdifferences in terms of critical-thinking performance betweenstudents assigned to any other groups. Regarding Hypothesis2, students assigned to the three teaching groups reportedsignificantly better understandings of critical thinking thanstudents in the control group. Additionally, students in theInfusion group reported better understandings of criticalthinking than students in the Immersion group and in all othergroups. However, students in the Immersion group did notalways report better understandings of critical thinking thanstudents in the General group.

Thus, students were not automatically disposed to thinkcritically even for controversial issues that lend themselvesnaturally to critical examination of different perspectives. Onthe contrary, the evidence from the present study clearlysuggests that the kind of teaching method is important forsomeone to construct correct understandings of criticalthinking and learn how to think critically. Also the findingsStudents also consistently expressed disappointmentregarding their nave understandings of critical thinking, but theyalso requested from the researcher copies of the critical-thinkingteaching materials that were used in the study, so they could alsohave an opportunity to benefit from them. The researcherinformed the students that the teaching materials were availableand could be picked up during office hours. Two students (S3 andS131) verbatim statements are indicative of these ideas:

I am very frustrated that I was so over-confident in some-thing that I no longer practice as much as I should. I am sodisappointed that I assumed so much. I had assumed thateveryone shared the same point of view as I did, eventhough I had read the opinion of a professor, which statedquite the opposite. I assumed that I was an effective criticalthinker, even though I am considerable out of practice. I amjust very surprised at all the assumptions I had made andwithout even realizing what I was doing (S3).

I would really like to get a copy of the materials that youused to teach the skills of critical thinking. I heard fromsomebody that you will make them available so that allparticipants can have an opportunity to benefit from them. Iam hoping that I can get a copy of them (S131).

4. Discussion

The findings of the study clearly indicated that, afteradjusting for possible initial differences concerning theirgeneral thinking skills, students critical-thinking performance

C. Angeli, N. Valanides / Learningshow that we cannot rest assured that students will eventuallybecome critical thinkers without any deliberate instructiontargeting the teaching of critical-thinking skills. Moreover,students in the General, Infusion, and Immersion groups didnot report more difficulties than students in the control group,and students in the Infusion and Immersion groups seemedmore satisfied from their performance and learning thanstudents in the other two groups.

In explaining these findings, the differences among theteaching methods should be taken into careful consideration.The same directions were given to all dyads about what theywere supposed to do and what task they had to complete.Students in the dyads from all groups, including the controlgroup, also had to work together to prepare an outline for theissue Are American values shaped by the mass media? inSession II, and an outline for the issue Should drugs belegalized? in Session III. Students in the control group didnot have any kind of training, were not engaged in any kind ofdialogue with the researcher, and were simply engaged inproblem solving without any instruction, guidance or feed-back. Students assigned to the General group watched thevideotaped lecture and followed the researchers instructionsregarding the task they needed to complete, but they were alsonot engaged in any kind of dialogue with the researcher anddid not receive any kind of feedback from the researcher abouttheir critical-thinking performance. Students in the Infusionand Immersion groups did not have the same training, mainlybecause the skills were not stated explicitly in the Immersiongroup. But, in both cases, students were engaged in dialoguewith the researcher, and were guided to construct their owncritical-thinking skills and subsequently to confront theirthinking through reflection for the purpose of evaluating it.

The dialogues in the Infusion and Immersion groups wereparticularly important for inducing critical thinking. Theystarted out as inquiry aiming toward the answering ofa specific question, such as, What is your point of view? orWhy do you think that? Then, the researcher moved todialogues as guided instruction aiming toward directing thestudents to decide about something. For instance, theresearcher asked: Someone could argue that a critical thinkeris one who evaluates different perspectives before decidingabout an issue. In your outline though, you have not done this.Can you explain to me why? This was the question thatstimulated sharp thinking and spurred tremendous interestfrom the part of the students. Initially, most students said thatthey did not need to explain a perspective that someone elsemight have, because everyone is entitled to his or her point ofview, and people should respect that. At this point, the dia-logue took the form of a debate between the researcher and thestudents trying to decide about the question Should weexamine different points of view before deciding about anissue or not? While many students showed skepticism anda willingness to reexamine their beliefs, some others stayedfirm in their beliefs and showed no willingness to alter the waythey were thinking. This seems to have happened in both, theImmersion group where the five critical-thinking skills werenot explicitly spelled out, and the Infusion group where the

331d Instruction 19 (2009) 322e334five critical-thinking skills were explicitly and clearly spelledout.

anBased on these results, students critical thinking seems toimprove most in teaching environments where learning ismediated by someone who confronts students beliefs andalternative conceptions, encourages students to reflect on theirown thinking, creates cognitive dissonance or puzzlement, andchallenges and guides students thinking when they areactively involved in problem solving. These tactics seem toincrease students metacognitive awareness enabling them tothink and talk about their own thinking processes (Hartman,2001). Rogoff (1990) termed this type of social interactionguided participation. By guided participation, she meansactive involvement by learners in structured activities with theguidance, support, and challenge of companions. Accordingly,asking students to constantly draw distinctions not in theabstract but concretely, allowing them to doubt things as wellas to be engaged in debates, and making them aware of thefact that there are multiple ways of carving up the samedomain are key strategies for the teaching of critical thinking.Most importantly, creating social contexts for instruction doesnot mean throwing students in an activity and letting themlearn alone. It means guiding students learning by providinga support system tailored to their needs and graduallyprompting them to take control of their own learning.

The results also indicate that learning, which is situated ina meaningful context of challenging problem solving, scaf-folded by a more experienced other, is significantly moreeffective than unguided exploration. Interestingly, in thepresent study, it was the control group that was engaged inunguided exploration. In the literature, this type of instruc-tional approach is referred to as discovery learning (Klahr &Nigam, 2004), or inquiry learning (Windshitl, 2004). Despitethe fact that these approaches have been very popular, theresults of this study indicate that free exploration with noguidance from the teacher does not work. These findingssupport research by Mayer (2004), Klahr and Nigam (2004),and Kirschner, Sweller, and Clark (2006) who concluded thatteaching guidance is more effective than discovery andunstructured exploration.

The results also showed that students in the General group,who were taught about general critical-thinking skills in a de-contextualized way and with no support or feedback from theresearcher, performed relatively lower than those assigned tothe Infusion and the Immersion groups, but the differencesamong them were not significant in this study. Despite the factthat these findings do not confirm Hypothesis 1, which statedthat the Infusion and Immersion groups would outperform theGeneral group, future research investigations with largersample sizes may show significant differences between theGeneral and the Infusion and Immersion teaching approaches.In addition, the findings relating to the nonsignificant differ-ences between the General and the control groups corroborateGarsides (1996) results, which showed that lectures alone wasnot a more effective strategy for the development of critical-thinking skills than mere group discussions. Obviously, thelack of feedback in both the General and control groups

332 C. Angeli, N. Valanides / Learninginfluenced in a negative way the quality of students critical-thinking performance in those groups.The results of the present study also corroborate otherresearch findings (Moreno, 2004), indicating that in instruc-tional environments where students learn with minimal guid-ance, not only do they become lost and frustrated, but theyalso develop misunderstandings, or alternative understandings,that diverge from the widely accepted ones. Students in thecontrol group were, for example, more inclined to believe thata critical thinker collaborates and listens to others thanstudents in the three teaching groups. Also, students in theGeneral group were more inclined to believe that a criticalthinker is systematically engaged in the thinking process thanstudents in the Immersion group. Similarly, only the studentsassigned to the Infusion group did not express the idea that Acritical thinker compromises to reach a decision. The latterqualitative findings show that students in the Infusion groupwere more likely to construct correct understandings of criticalthinking than students in the Immersion group, as well as inthe other groups, despite the fact that no significant differenceswere found between students critical-thinking performance inthe Infusion and Immersion groups. However, more research isneeded to further investigate the differences between theInfusion and Immersion teaching approaches. Regardingstudents self-reported difficulties, no significant differenceswere found across the three teaching groups and the controlgroup.

An issue that needs to be followed up in future studiesconcerns the specific sequence of the instructional tactics to bepresented to learners. In the present study, the sequence oftactics was pre-specified and applied as such for both Infusionand Immersion. However, future studies may examine theeffects of different instructional sequences taking into consid-eration students prior knowledge, so that the questionregarding the most appropriate starting point in instruction forstudents to correctly understand the concept of critical thinking,as well as for learning how to think critically can be answered.

A second issue for future research relates to the possibletransfer and retention effects of the Infusion and Immersionapproaches. In the present study, no significant differenceswere found between the Infusion and Immersion groups,despite the fact that the five general critical-thinking skillswere stated explicitly only in the Infusion group and not in theImmersion group. This difference between the two instruc-tional approaches is important and it is worthy of furtherresearch, because it might be the case that if general critical-thinking skills are not made explicit to students, then studentscritical-thinking skills may transfer from one domain toanother in the short-run, but not in the long-run.

It is also important to interpret the evidence from thepresent study taking into consideration both its limitations interms of sample size and the short duration of the interven-tions. First, although there were 144 students who participatedin the study, the unit of analysis was the dyad and not theindividual participant. As a result, the sample size that wasused for detecting differences in instructional effects includeddata from the 72 dyads that were randomly formed and

d Instruction 19 (2009) 322e334randomly assigned to the three teaching groups and the controlgroup. Second, the duration of the instructional intervention

was limited. These limitations can always have implicationson the effects of the different instructional treatments and canmask to a certain extent any existing differences among them.

Other concerns also seem pertinent. For example, it cannot beinferred at this point in time whether the differences in critical-thinking performance, as they were reported in the present study,were enduring or temporary. Based on students commentsduring the debriefing session, learning with both the Infusion andImmersion approaches enabled them to use their critical-thinkingskills in contexts beyond the immediate instructional context ofthis study. However, despite of this promising evidence, moresystematic experimental studies are needed to further examinethe issue of transfer across different contents and contexts, andwhether the identified gains in critical-thinking performancepersist and become enduring habits of mind.

Lastly, the dyads in this study were randomly formed andalso randomly assigned to each approach, but no informationwas collected concerning the interactions and the dynamicsinvolved in each dyad. Recent developments relating to thesocially shared distributed cognition emphasize the impor-tance of examining the learner in interaction with others(Angeli, 2008; Salomon, 1992, 1993; Valanides & Angeli,2008). This type of analysis was not the focus of the presentinvestigation, but it should be considered in future studiesabout the teaching of critical thinking, because it is commonlyaccepted that much of human cognition is so varied and so

Appendix A

C. Angeli, N. Valanides / Learning anCritical-thinking performance scoring rubric.

Criteria Scoring

1. Reason clearly within a point of view. 5

2. Discuss the issue from different perspectives.

3. Identify pros and cons for each perspective.

4. Explain with reasons and evidence which

perspective they think is best.

1. Reason clearly within a point of view. 4

2. Discuss the issue from different perspectives.

3. Identify pros and cons for each perspective.

4. Do not explain with reasons and evidence

which perspective they think is best.

1. Reason clearly within a point of view. 3

2. Discuss the issue from different perspectives.

3. Do not identify pros and cons for each perspective.

4. Do not explain with reasons and evidence which

perspective they think is best.

1. Reason clearly within a point of view. 2

2. Do not discuss the issue from different perspectives.

3. Do not identify pros and cons for each perspective.

4. Do not explain with reasons and evidence which

perspective they think is best.

1. Do not reason clearly within a point of view. 1

2. Do not discuss the issue from different perspectives.

3. Do not identify pros and cons for each perspective.4. Do not explain with reasons and evidence which

perspective they think is best.sensitive to cultural context that we must also seek mecha-nisms by which people actively shape each others knowledgeand reasoning process (Resnick, 2004: 2).

References

Alston, K. (2001). Re/thinking critical thinking: the seductions of everyday

life. Studies in Philosophy and Eduction, 20, 27e40.

Anderson, T., Howe, C., Soden, R., Halliday, J., & Low, J. (2001). Peer

interaction and the learning of critical-thinking skills in further education

students. Instructional Science, 29, 1e32.Angeli, C. (2008). Distributed cognition: a framework for understanding the

role of technology in teaching and learning. Journal of Research on

Technology in Education, 40(3), 271e279.Bailin, S. (2002). Critical thinking and science education. Science & Educa-

tion, 11(4), 361e375.

Bailin, S., Case, R., Coombs, J. R., & Daniels, L. B. (1999). Conceptualizing

critical thinking. Journal of Curriculum Studies, 31, 285e302.Brown, A. (1997). Transforming schools into communities of thinking and

learning about serious matters. American Psychologist, 52, 399e413.

Browne, M. N., & Freeman, K. (2000). Distinguishing features of critical-

thinking classrooms. Teaching in Higher Education, 5, 301e309.Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd

ed).. Hillsdale, NJ: Erlbaum.

Collins, A., Brown, J. S., & Newman, S. E. (1989). Cognitive apprenticeship:

teaching the crafts of reading, writing, and mathematics. In L. B. Resnick

(Ed.), Knowing, learning, and instruction: Essays in honor of Robert

Glaser (pp. 453e494). Hillsdale, NJ: Erlbaum.

Dennick, R. G., & Exley, K. (1998). Teaching and learning in groups and

teams. Biochemical Education, 26, 111e115.

Education Commission of the States. (2007). Framework for reauthorizing the

No child Left Behind (NCLB) Act. Recommendations to improve and

strengthen the law. Denver, CO: Author.Ennis, R. H. (1989). Critical thinking and subject specificity: clarification and

needed research. Educational Researcher, 18, 4e10.

Ennis, R. H. (1992). The degree to which critical thinking is subject specific:

clarification and needed research. In S. P. Norris (Ed.), The generalizabilityof critical thinking: Multiple perspectives on an educational ideal (pp. 21e

37). New York: Teachers College.

Facione, N., Facione, P., & Sanchez, M. (1994). Critical-thinking disposition

as a measure of competent clinical judgment: the development of the

California Critical Thinking Disposition Inventory. The Journal of Nursing

Education, 33, 345e350.

Facione, P. A. (1990a). Critical thinking: A statement of expert consensus forpurposes of educational assessment and instruction. Millbrae, CA:

Academic.

Facione, P. A. (1990b). The California critical thinking skills test (Reports No.

1e4). East Lancing, MI: National Center for Research on TeacherLearning (ERIC Document Reproduction Service No. ED 327 549, ED

327 550, ED 327 584, and ED 327 566).

Facione, P. A., & Facione, N. C. (1994). The California critical thinking skillstest: Test manual. Millbrae, CA: Academic.

Feingold, A. (1992). Sex differences in variability in intellectual abilities: a new

look at an old controversy. Review of Educational Research, 62, 61e84.

Frijters, S., Ten Dam, G., & Rijlaarsdam, G. (2008). Effects of dialogic learning

on value-loaded critical thinking. Learning and Instruction, 18, 66e82.

Garside, C. (1996). Look whos talking: a comparison of lecture and group

discussion teaching strategies in developing critical-thinking skills.

Communication Education, 45, 212e227.Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory:

Strategies for qualitative research. Chicago: Aldine.

Halpern, D. F. (1997). Critical thinking across the curriculum. Mahwah, NJ:

Erlbaum.

Halpern, D. F. (2003). Thought and knowledge (4th ed.). Mahwah, NJ:

Erlbaum.

333d Instruction 19 (2009) 322e334Hartman, H. J. (2001). Metacognition in learning and instruction: Theory,research and practice. Dordrecht, The Netherlands: Kluwer.

Johnson, D. W., & Johnson, R. T. (1993). Creative and critical thinking.

Through academic controversy. American Behavioral Scientist, 37, 40e53.

Kalman, C. S. (2002). Developing critical thinking in undergraduate courses:

a philosophical approach. Science and Education, 11, 83e94.

Kaplan, E. J., & Kies, D. A. (1994). Strategies to increase critical thinking in

the undergraduate college classroom. College Student Journal, 28, 24e31.

Kennedy, M., Fisher, M. B., & Ennis, R. H. (1991). Critical thinking: literature

review and needed research. In L. Idol, & B. Fly Jones (Eds.), Educationalvalues and cognitive instruction: Implications for reform (pp. 11e40).

Hillsdale, NJ: Erlbaum.

Kirschner, P. A., Sweller, J., & Clark, R. E. (2006). Why minimal guidance

during instruction does not work: an analysis of the failure of construc-

tivist, discovery, problem-based, experiential, and inquiry-based teaching.

Educational Psychologist, 41(2), 75e86.

Klahr, D., & Nigam, M. (2004). The equivalence of learning paths in early

science instruction. Psychological Science, 15(10), 661e667.Kuhn, D. (1999). A developmental model of critical thinking. Educational

Researcher, 28, 16e25.

Mayer, R. E. (2004). Should there be a three-strikes rule against pure

discovery learning? The case for guided methods of instruction. American

Psychologist, 59, 14e19.

Peters, M. A. (2005). The new Prudentialism in education: actuarial rationality

and the entrepreneurial self. Educational Theory, 55, 122e137.

Pithers, R. T., & Soden, R. (2000). Critical thinking in education: a review.

Educational Research, 42, 237e249.

Prawat, R. S. (1991). The value of ideas: the Immersion approach to the

development of thinking. Educational Researcher, 20, 3e10.

Resnick, L. (1987). Education and learning to think. Washington, DC:

National Academy Press.

Resnick, L. (2004). Shared cognition: thinking as a social practice. In

L. Resnick, J. Levine, & S. Teasley (Eds.), Perspectives on socially shared

cognition (pp. 2e20). Washington, DC: American Psychological

Association..

Rogoff, B. (1990). Apprenticeship in thinking: Cognitive development in social

contexts. New York: Oxford University Press.

Salomon, G. (1992). New challenges for educational research: studying the

individual within learning environments. Scandinavian Journal of Educa-tional Research, 36(3), 167e182.

Salomon, G. (1993). Distributed cognitions: Psychological and educational

considerations. New York: Cambridge University Press.Strauss, A. L., & Corbin, J. (1990). Basics of qualitative research: Grounded

theory procedures and techniques. Newbury Park, CA: Sage.

334 C. Angeli, N. Valanides / Learning and Instruction 19 (2009) 322e334McBride, R. E., Xiang, P., & Wittenburg, D. (2002). Dispositions toward

critical thinking: the preservice teachers perspective. Teachers andTeaching, 8, 29e40.

McPeck, H. (1981). Critical thinking and education. New York: St. Martins

Press.

McPeck, H. (1990a). Critical thinking and subject specificity: a reply to Ennis.

Educational Researcher, 19, 10e12.

McPeck, H. (1990b). Teaching critical thinking: Dialogue and dialectic. New

York: Routledge.

Moreno, R. (2004). Decreasing cognitive load in novice students: effects of

explanatory versus corrective feedback in discovery-based multimedia.

Instructional Science, 32, 99e113.

Nickerson, R., Perkins, D. N., & Smith, E. (1985). The teaching of thinking.Hillsdale, NJ: Erlbaum.

Papastephanou, M., & Angeli, C. (2007). Critical thinking beyond skill.

Educational Philosophy and Theory, 39(6), 604e621.

Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.).Thousand Oaks, CA: Sage.

Paul, R. (1995). Critical thinking: How to prepare students for a rapidly

changing world. Santa Rosa, CA: Foundation for Critical Thinking.

Perkins, D. N. (1986). Knowledge as design. Hillsdale, NJ: Erlbaum.Perkins, D., Tishman, S., Ritchhart, R., Donis, K., & Andrade, A. (2000).

Intelligence in the wild: a dispositional view of intellectual traits. Educa-

tional Psychology Review, 12, 269e293.Ten Dam, G., & Volman, M. (2004). Critical thinking as a citizenship

competence: teaching strategies. Learning and Instruction, 14, 359e379.Tsui, L. (1999). Courses and instruction affecting critical thinking. Research in

Higher Education, 40, 185e200.

Tynjala, P. (1998). Traditional studying for examination versus constructivist

learning tasks: do learning outcomes differ? Studies in Higher Education,23, 173e189.

Valanides, N. (1990). Pragmatic versus syntactic approaches to deductive

reasoning. Doctoral Dissertation. University at Albany, State University of

New York. 1175A. (University Microfilms No. 90-24947).

Valanides, N., & Angeli, C. (2008). Distributed cognition in a sixth-grade

classroom: an attempt to overcome alternative conceptions about light

and color. Journal of Research on Technology in Education, 40(3),309e336.

Voss, J. F., Perkins, D. N., & Segal, J. W. (1991). Informal reasoning in

education. Mahwah, NJ: Erlbaum.

Whimbey, A., & Lochhead, L. J. (1986). Problem solving and comprehension.Hillsdale, NJ: Erlbaum.

Wilcox, R. R. (1995). ANOVA: a paradigm for low power and misleading

measures of effect sizes? Review of Educational Research, 65, 51e77.

Windshitl, M. (2004). Caught in the cycle of reproducing folk theories of

Inquiry: how pre-service teachers continue the discourse and practices

of an atheoretical scientific method. Journal of Research in Science

Teaching, 45(5), 481e512.

Instructional effects on critical thinking: Performance on ill-defined issuesIntroductionThe concept of critical thinkingCritical-thinking skills and instructional approachesThe present studyResearch questions-Hypotheses

MethodParticipantsDesignSession ISession IISession III

InstrumentsCalifornia critical thinking skills test (CCTST)Critical-thinking performanceUnderstanding of critical thinking

ResultsEffects of teaching methods on students critical-thinking performanceEffects of teaching methods on students critical-thinking understandingDifficulties encounteredStudents reflections

DiscussionAppendix AReferences