Background: Facilitation of reflective practice is critical forthe ongoing demands of health care practitioners. Reflec-tive thinking concepts, grounded in the work of Dewey andSchön, emphasize critical reflection to promote transforma-tion in beliefs and learning necessary for reflective practice.The Reflective Thinking Questionnaire (QRT) andRevised Study Process Questionnaire (RSPQ-2F) assessskill aspects of professional reasoning, with promise formeasuring changes over time. Objective: The purpose ofthis study was to examine the reliability and responsivenessand the model validity of reflective thinking andapproaches to learning measures for U.S. health professionsstudents enrolled in entry-level occupational (MOT) andphysical therapy (DPT) programs. Methods: This measure-ment study addressed reliability and responsiveness of twomeasures, the QRT and RSPQ-2F, for graduate health pro-fessionals. A convenience sample of 125 MOT and DPTstudents participated in the two-measure, test-retest inves-tigation, with electronic data collection. Results: Out-comes support the stability of the four-scale QRT (ICC0.63 to 0.82) and the two-scale RSPQ-2F (ICC 0.91 and0.87). Descriptive data supporting responsiveness are pre-sented. Conclusions: With noted limitations, the resultssupport the use of the QRT and RSPQ-2F measures toassess changes in reflective thinking and approaches tolearning. Measurement of these learning outcomes furthersour understanding and knowledge about instructionalstrategies, development of professional reasoning, and fos-tering of self-directed learning within MOT and DPT pro-grams. J Allied Health 2011; 40(3):128–136.

REFLECTIVE PRACTICE (RP) is a hallmark of both per-sonal and professional development for health profession-als.1,2 RP includes reflective thinking and self-directedlearning. These are essential outcomes to examine fordeveloping, graduate occupational (MOT) and physicaltherapy (DPT) students to assess measurable changes that

may occur within the academic program and translation forfuture practice. While there are many instructional strate-gies to facilitate reflection in health professions education,both in the classroom and clinic,3 reflective thinking (RT)remains a challenging concept to fully and pragmaticallyassess within academic programs and later during practice.RT is a construct grounded in the work of Dewey,1 Schön,2

and Mezirow,4 and others emphasizing the necessity of crit-ical reflection to promote transformation in beliefs andlearning for professional practice.

Reflection as a cognitive activity is described by Dewey1

as a systematic, rigorous, disciplined way of thinking thatneeds to happen in interaction with others. Deweydescribed the process of RT as moving the learner from oneexperience to another, with a deeper understanding of rela-tionships within connections, other experiences, and ideas.Dewey1 elucidates that reflection requires attitudes thatvalue both the personal and intellectual growth of oneselfand others. Dewey delineated fives stages of reflectivethought:

1. Perplexity, confusion, doubt 2. Attentive interpretation of the given elements3. Examination, exploration, and analysis to define and clarify

the problem4. Elaboration of the tentative hypothesis5. Testing the hypothesis by doing something overtly to bring

about anticipated results.1[p33]

RT focuses on the process of making judgments aboutwhat has happened, and Schön2 provided a Model ofReflective Practice from which to examine RT conceptu-ally. This model is based on how professionals think andemphasizes learning from experience. Schön contemplatedhow professionals in architecture, medicine, law, and otherprofessions readjust to the influences and changes withindaily practice. Schön described the following professionalpractices as hallmarks of RP: “knowing-in-action, recogniz-ing surprise, reflection-in-action, experimentation, andreflection-on-action.” 2[p119–140] Hence, RT is where learnersare assessing what they know, what they need to know, andhow they bridge the gap, while in the process of learning orin response to learning or after the fact. Schön’s RP modelis a conceptual framework that combines the artistry ofpractice with the science of medicine, where ambiguity andscience intersect.

Scholarship of teaching theorists and health professionsfaculty often measures reflective learning and practice out-comes through both qualitative and quantitative means,such as portfolios and reflective papers.3 Many of these

128

� Assessing Reflective Thinking and Approaches to LearningLouise Dunn, ScD, OTR/LGina Maria Musolino, PT, MSEd, EdD

Dr. Dunn is Pediatric Coordinator and Assistant Professor, Division ofOccupational Therapy, and Dr. Musolino is Associate Professor andDirector of Clinical Education, Department of Physical Therapy, Collegeof Health, The University of Utah, Salt Lake City, Utah. Dr. Musolino iscurrently Associate Professor and Director of Clinical Education, Schoolof Physical Therapy and Rehabilitation Sciences, College of Medicine,University of South Florida, Tampa, Florida.

RA1024—Received Sep 1, 2010; accepted Feb 21, 2011.

Address correspondence to: Dr. Louise Dunn, Division of OccupationalTherapy, College of Health, The University of Utah, 520 Wakara Way,Salt Lake City, UT 84108, USA. Tel 801-585-9356, fax 801-585-1001.louise.dunn@hsc.utah.edu.

assessment methods are time consuming and often specificto courses, as opposed to examining change across the cur-riculum or global learning outcome measures. In addition,portfolios and reflective papers may not be the most effi-cient and cost-effective ways to measure carryover of self-directed learning and RT as students become practitioners.RT is ever important for health professionals as practicecontinues to evolve at such a rapid pace, both scientificallyand technologically, that entry-level didactics are oftenquickly surpassed with contemporary evidence in practice.Currently, measures exist that assess critical thinking,5 RT,6

and learning approaches7,8; however, none provide tempo-ral stability and responsiveness estimates necessary formeasuring outcomes and changes over time.

RT is a complex construct to measure. Kember et al.6

developed The Questionnaire for Reflective Thinking(QRT) based on Schön2 and Mezirow’s4 work for this pur-pose. Researchers6 developed four dimensions for RT basedon the continuum proposed by Mezirow. These dimensionsincluded: habitual action, understanding, reflection, andcritical reflection.

• Habitual action corresponds with actions that are performedunconsciously.

• Understanding corresponds with student thinking that showscomprehension of concepts “without reflection upon its signif-icance in personal or practical situations.”6[p 384]

• Reflection represents actions where information is connectedwith experience.

• Critical reflection is when transformation of beliefs and meaningoccurs. It involves constructing personal meaning and usinghigher-level cognitive skills such as analysis, synthesis, andevaluation.9

Kember et al.6 determined that that the QRT had stronginternal consistency, with adequate to good construct valid-ity, for a sample of undergraduate occupational and physicaltherapy students in Hong Kong. Critical to measuringchange over time, stability and responsiveness measureswere not reported. To date, these RT measures have notbeen validated for graduate-level MOT and DPT healthprofessions students in the U.S.: hence, the purpose of thisinvestigation.

Transformational learning is central to Meizirow’s4 workand describes the learning process of “becoming criticallyaware of one’s own tacit assumptions and expectations ofthose of others and assessing their relevance for making aninterpretation.”4,10 Development of RT and transforma-tional learning stems from approaches to learning thatresult from interactions with and among students, instruc-tors, and peers and through engagement with the coursecontent as well as professional experiences.3,6,7,11,12

Another instrument that considers these transforma-tional aspects of RT is the Revised Study Process Ques-tionnaire7 (RSPQ-2F). The RSPQ-2F assesses students’motives for learning, along a continuum, ranging from sur-face motives for learning with a focus on memorization, todeeper motives for learning with a focus on seeking mean-

ing and relationships between the new and previousknowledge. Biggs and colleagues7 developed the RSPQ-2Fto assess how learning within courses promotes deeperlearning through transactions among students within thecontext of the course and assignments. The purpose forRSPQ-2F is not to measure students’ disposition for learn-ing, but to examine how instructional strategies andassignments promote deeper learning. Students will utilizean approach to learning that they believe fits with thedemands of the course.7,11 Biggs and colleagues7 validatedthe internal consistency and construct validity of theRSPQ-2F with undergraduate students in Hong Kong andAustralia; however, no validation studies have beenreported with entry-level graduate students in MOT andDPT programs in the U.S. Biggs et al. utilized the RSPQ-2F for descriptive purposes and did not complete furtherstudies related to test-retest reliability and responsivenessneeded for outcome assessments.

Understanding more about the ongoing development ofstudents’ RT and approaches to learning would assist fac-ulty in examining progression of these skills and potentiallyto further develop courses and instructional strategies topromote RP. Emphasis on RT development may influenceinstructional approaches, specifically those that involveactive participation, with opportunities for clinical reason-ing.6 Students’ understanding of their approaches to learn-ing and RT abilities may help them gain insights abouttheir professional development and acquire greater interestin self directed learning.

The purpose of this study was to validate the QRT6 andRSPQ-2F7 for U.S. health professions students enrolled inentry-level, graduate DPT and MOT programs. Four spe-cific research questions guided this investigation. Do thesemeasures, the QRT and RSPQ-2F, have:

1) adequate internal consistency or relatedness among items forgraduate health care students in the U.S.?

2) adequate temporal stability for assessing change over time?3) adequate responsiveness to measure changes in graduate stu-

dents’ RT and approaches to learning?, and 4) (considering the QRT only) expected conceptual direction

for scores on the QRT?

Measurement of these learning outcomes enable us to assessdevelopment of some aspects of professional reasoning andself-directed learning as essential components of RP forentry-level graduate students in DPT and MOT in the U.S.

Methods

DESIGN

To study the multifactorial aspects of RP for developingU.S. health professionals, a test-retest design was utilizedfor this measurement study to examine the reliability andresponsiveness of the QRT6 and RSPQ-2F7 as outcomemeasures. As a secondary purpose, we examined the con-

Journal of Allied Health, Fall 2011, Vol 40, No 3 129

ceptual direction for scores on the QRT to determine if oursample represented the continuum suggested by priorresearch. Utilizing two different instruments was relevantto examine the differing factors associated with RP. Thisstudy was conducted at a large Western U.S. doctoral/research university-extensive (Carnegie classification) inthe College of Health, DPT and MOT programs.

INSTRUMENTATION

For the purpose of this measurement study, two instrumentswere utilized: The Questionnaire for Reflective Thinking6

(QRT), and the Revised Study Process Questionnaire7

(RSPQ-2F). Each instrument is further described in termsof instrumentation for this assessment:

QRT

The QRT is a 16-item self-report questionnaire utilizing a 5-point Likert response scale, measuring four dimensions of RT:habitual action, understanding, reflection, and critical reflec-tion. Likert scale responses range from strongly disagree (1) tostrongly agree (5). Scoring is based on summed responses oneach of the four dimensions. The four dimension scores mayrange from 4 (strongly disagree) to 20 (strongly agree) in termsof agreement. The QRT has acceptable internal consistency(0.63 to 0.76 Cronbach’s alpha); construct validity is sup-ported through confirmatory analysis.13

For the QRT, Kember et al.6 established psychometricproperties with undergraduate health science majors and asmall representative sample of master’s level nursing stu-dents in Hong Kong. They did not assess change over timeand did not examine stability and responsiveness. The QRThas not been tested with U.S. health professions students interms of reliability and responsiveness, nor as an outcomemeasure of RT among MOT and DPT students specifically.For the purpose of this study, some of the wording on theQRT was adapted to conform to linguistic patterns in theU.S. (e.g., used the term grades instead of marks). Also, theformer studies utilized a paper/pencil format of the QRT,while this investigation for ease of utility used a web-based,online data collection format to gather instrument data.

RSPQ-2F

The RSPQ-2F is a 20-item, self-report questionnaire, alsowith a 5-point Likert response set, that measures surfaceand deep approaches to learning. Sample Likert scaleresponses range from descriptors: i.e., “this item is never orrarely true of me” to “this item is always true of me.” Scor-ing is based on summed responses for each subtest. Subtestscores may range from 10 to 50. Biggs et al.7 established psy-chometric properties with undergraduate health sciencemajors and a small sample of master’s level nursing studentsin Hong Kong and Australia. The RSPQ-2F has good inter-nal consistency (0.76 to 0.78 Cronbach’s alpha) and con-

struct validity; however, stability and responsiveness esti-mates are lacking in studies described to date.14,15

SUBJECTS

A convenience sample of 125 MOT and DPT graduate stu-dents participated in this measurement study using elec-tronic data collection processes. Study participants (seeTable 1 for demographic characteristics) represented all 3years of each MOT and DPT professional program. Partici-pants voluntarily consented to take part in this study, andthe sample does not represent every student in the pro-grams, yet it is representative of each class. Entry level isthe didactic and clinical/fieldwork preparation for thesehealth professions in which students are capable of entryinto the practice for the profession. Entry-level degrees varyby profession; currently, U.S. OT programs are a minimumof a master’s degree,16 and by 2020, all U.S. PT programs areto be at the DPT or clinical doctorate level for entry.17,18 Inthis study, 37 MOT students (30%) and 88 DPT students(70%) completed the two study measures.

In total, 124 graduate MOT and DPT students com-pleted both initial study measures. Return rates for the firstadministration of the measures was 68% across both profes-sions, and all 3 years in the program were represented. Forthe retest, a total of 56 graduate students (46%), consistingof 13 MOT (23%) and 43 DPT (77%) students, completedstudy measures for a second time. One questionnaire wasmissing more than 20% of the data and therefore wasexcluded from the data analysis.

Participants were recruited via email announcements(MOT and DPT), verbal class announcements, and postingson the web-based assignment board (DPT). Participantswere provided study information and informed consent pro-cedures prior to beginning the study measures. All studentswere offered the opportunity to have their name entered ina random drawing for a small gift card incentive followingcompletion of the questionnaires a second time (retest).Subject data were de-identified from the incentive pool.The researchers collected demographic information for theMOT and DPT student study participants, including year inthe program, degree program, and gender (Table 1).

STUDY PROCEDURES

This study was approved by the University of Utah Institu-tional Review Board. The researchers invited all students inthe MOT and DPT programs to participate in the study.The two measures, QRT and RSPQ-2F, were set up in anonline format. In order to examine temporal stability, stu-dents were again asked to complete the measurement ques-tionnaires a second time within a 2- to 3-week time framefollowing the initial completion of the questionnaires. Theresearchers sent reminder emails to all subjects and postedelectronic reminder announcements on the correspondingDPT website weekly for 3 weeks to promote recruitment.

130 DUNN, MUSOLINO, Assessing Reflective Thinking and Approaches to Learning

At the time of this study, no corresponding web coursesite link existed for OT students; MOT and DPT studentsboth utilized the same direct link to access the study meas-ures, either through email announcements (with the webpage link inserted) or a course website link (with the sameweb page link inserted). Students received two emails fol-lowing the initial completion of the measures to remindthem about the retest, with corresponding announcementregarding the incentive for a random drawing for a smallgift card if completing the retest. Informed consent proce-dures occurred with both the test and re-test procedures,prior to study measures. There was no further contact if stu-dents did not respond to the email reminders or announce-ments. The research study website was closed following a 6-week period.

DATA ANALYSIS

With the exception of one student, 90% of the sample com-pleted all items on both measures. Approximately 5% ofthe sample missed one or two items on the QRT. Relativemean item values (RMV) were used to replace missing data.One student had missing data for more than 20% of theitems on the RSPQ-2F, and therefore these subtest scoreswere excluded from the RSPQ-2F analyses.

Descriptive analyses were run to determine means andstandard deviations (SD) for both the QRT and the RSPQ-2F. Data analyses included examining four processes:

1) the interrelatedness among items on each measure; 2) the temporal stability of both measures over 2 to 3 weeks; 3) responsiveness or variance in summed scores and individual

item distribution for the four QRT dimensions and the twoRSPQ-2F subtests; and

4) validation of the conceptual continuum for the QRT.

Data for these analyses involved summed totals from Likertscales, and therefore use of parametric statistics was consid-ered acceptable.19

Interrelatedness among items by dimensions on theQRT and by subtests on the RSPQ-2F was examined withCronbach’s alpha. Alphas between 0.60 and 0.80 were setas an acceptable level when examining internal consistencywith conceptual measures, as described by Schmitt.20 Tem-poral stability (test-retest) for the four dimensions of theQRT and the two subtests of the RSPQ-2F was assessedusing intra-class correlation coefficient (ICC) model(3,1).21 An ICC between 0.70 and 0.80 was set as anacceptable level.19

Responsiveness or variance was assessed through a his-togram plot, illustrating the score distributions for the QRTand for the RSPQ-2F. Individual item distributions werecomputed for the QRT and the RSPQ-2F using frequencyand percentages of responses for each item. t-Tests wereused to examine differences in performance on the fourscales of the QRT in order to support the conceptual con-tinuum for RT proposed by Kember et al.6 t-Tests were alsoused to compare performance on the two subscales of theRSPQ-2F.

Results

The purpose of this study was to validate the QRT6 andRSPQ-2F7 for U.S. health professions students enrolled inentry-level, graduate MOT and DPT programs. Descriptivedata are presented to provide an overview of scores forgraduate health care students on the QRT and RSPQ-2F.Next, reliability and responsiveness data are presented, asthese are the primary purposes of this study. Overall, out-comes support the stability and responsiveness of the QRT(ICCs ranged from 0.63 to 0.82) and the RSPQ-2F (ICCsfrom 0.87 to 0.91). Specific results for each query follow.

DESCRIPTIVE DATA

Summed scores for the QRT ranged from 4 to 18 for habit-ual action, 11 to 20 for understanding, 10 to 20 for reflec-tion, and 7 to 20 for critical reflection. Overall mean scoreson the QRT were higher for understanding (mean 18.44,SD 1.79) and reflection (mean 17.28, SD 1.94), lower forcritical reflection (mean 14.65, SD 3.11), and much lowerfor habitual action (mean 10.72, SD 2.82).

Summed scores on the RSPQ-2F for deep approachesranged from 13 to 44, and from 22 to 29 for surfaceapproaches. Overall mean scores on the RSPQ-2F werehigher for surface learning approaches (mean 39.71, SD5.18) and lower for deep learning approaches (mean 29.65,SD 5.68).

Next, we share results of the four major queries posed forpurposes of this study.

RESEARCH QUESTION 1: Is internal consistency adequateon the QRT and the RSPQ-2F for measuring RT andapproaches to learning among health professional students in theU.S.?

For the QRT, internal consistency, using Cronbach’salpha, ranged from a low of 0.55 on reflection to 0.67 on

Journal of Allied Health, Fall 2011, Vol 40, No 3 131

TABLE 1. Demographic Characteristics of Sample (n = 125)

OT PTYear in Program No. (%) Sex Age Range (yrs) No. (% Class) No. (% Class)

1st year 49 (39.2) 19 M / 30 F 21–43 14 (52) 35 (83)2nd year 44 (35.2) 18 M / 26 F 22–28 14 (78) 30 (75)3rd year 32 (25.6) 12 M / 20 F 24–27 9 (39) 23 (58)

critical reflection. Reflection (alpha = 0.55) and habitualaction (alpha = 0.57) results were below the acceptablerange (Table 2).

For the RSPQ-2F, internal consistency scores, usingCronbach’s alpha, are presented in terms of its two-factorstructure: deep approaches to learning (alpha = 0.91) andsurface approaches to learning (alpha = 0.87). Both resultsfall within the significance level established for this study(Table 2).

RESEARCH QUESTION 2: Do the QRT and the RSPQ-2Fmeasures have adequate temporal stability for assessing changesin RT and approaches to learning over time?

For the QRT, three of the four dimensions had accept-able stability (habitual action, reflection, and criticalreflection). The understanding dimension was below thesignificance level set for this study. For the RSPQ-2F, boththe surface approaches and the deep approaches hadacceptable stability and met the significance levels set forthis study (Table 2).

RESEARCH QUESTION 3: Do the QRT subtests and theRSPQ-2F subtests have adequate responsiveness to measurechanges in graduate students’ approaches to learning and RT?

For the QRT, the distribution of scores, illustrated byhistograms in Figure 1, displays adequate distribution ofscores for habitual action, reflection, and critical reflection.The understanding dimension, however, demonstrates anegatively skewed distribution, which persisted even withlogarithmic conversion of scores. Ranges for the summedscores on the QRT are presented in Table 2. The distribu-tion of responses for individual items on the QRT shows anadequate distribution of scores for three of the four dimen-sions (see Table 3). The understanding dimension shows aclustering of scores at the higher end.

Next, in an examination of the responsiveness of theRSPQ-2F, the distribution of scores (Fig. 1) show adequatedistribution for surface approaches and deep approaches tolearning. Table 2 shows score ranges on the RSPQ-2F. Indi-vidual item scores were distributed across the ratings onboth of these subtests.

RESEARCH QUESTION 4: Do scores on the QRT show theexpected conceptual direction?

For the QRT, overall mean scores (Table 2) werehigher for understanding (mean 18.44, SD 1.79) andreflection (mean 17.28, SD 1.94), yet lower for criticalreflection (mean 14.65, SD 3.11) and much lower forhabitual sction (mean 10.72, SD 5.68). Understandingwas reported significantly more often than habitualaction [t(124) = 22.36, p = 0.00], reflection [t(124) =5.39, p = 0.00], and critical reflection [t(124) = 12.97, p= 0.00]. Reflection was reported significantly more oftenthan critical reflection [t(124) = 8.47, p = 0.00] or habit-ual action [t(124) = 21.03, p = 0.00]. Critical reflectionwas reported significantly more often than habitualaction [t(124) = 9.92, p = 0.00].

Discussion

Findings from this study support the stability of both theQRT and RSPQ-2F measures for examining changes inboth RT and approaches to learning, respectively, for grad-uate MOT and DPT students in the U.S. Internal consis-tency is at an acceptable level for the RSPQ-2F and theQRT understanding and critical reflection dimensions, butbelow acceptable levels for the habitual and reflectiondimensions. However, both measures offer utility for exam-ining changes in RT and approaches to learning for entry-level DPT and MOT students.

132 DUNN, MUSOLINO, Assessing Reflective Thinking and Approaches to Learning

TABLE 2. Means, Internal Consistency, and Test-Retest Data

Mean Summed Alpha AlphaSample Range Score (SD) ICC (Current Study) (Prior Studies)

RSPQ-2FDeep approach 124 13–44 29.65 (5.68) 0.91 0.75 0.64

(Biggs7)

Surface approach 124 22–49 39.71 (5.18) 0.87 0.75 0.64(Biggs7)

QRTHabitual Action 125 4–18 10.72 (2.82) 0.79 0.57 0.76

(Leung13, Kember6)

Understanding 125 11–20 18.44(1.79) 0.63 0.66 0.76 (Leung13, Kember6)

Reflection 125 10–20 17.28 (1.94) 0.82 0.55 0.63(Leung13, Kember6)

Critical reflection 125 7–20 14.65 (3.11) 0.73 0.67 0.68(Leung13, Kember6)

Journal of Allied Health, Fall 2011, Vol 40, No 3 133

FIGURE 1. Histograms of normative curves for QRT and RSPQ-2F.

Findings from this study also support stability of theQRT dimensions, yet with somewhat less stability noted forthe understanding dimension. Lack of variance on some ofthe dimensions may not adequately differentiate amongentry-level graduate MOT and DPT students in the U.S.Specifically, a ceiling effect was found on both the under-standing and reflection dimensions. These dimensionslacked adequate distribution of responses across individualitems. Clearly, RT, due to multifactorial influences, is amuch more difficult construct to define and measure.

The QRT outcomes should be interpreted cautiously, astwo of the dimensions did not meet the significance levelsset for internal consistency. Presence of a ceiling effect sug-gests that additional items might be needed, especially forthe reflective and understanding dimensions. Internal con-sistency for the reflection dimension is moderate and maynot be robust enough to capture all changes when the QRTis used with entry-level U.S. MOT and DPT students.

Examination of differences among the four QRT dimen-sions shows that MOT and DPT participants’ profilesreflect the hierarchy of RT skills.6,13 Specifically, the QRTmeasures the predicted hierarchy among the dimensions,with MOT and DPT students reporting greater use ofunderstanding and RT in their respective programs. Thesignificantly lower report of habitual action outcome scoressupport less reliance on memorization and greater use ofthinking and application of information to course contentand to experiences. The significantly lower reports of criti-cal reflection skills are consistent with findings from

Kember et al.6 These findings support the need for moreapplied experience in fieldwork/clinical education and sub-sequent practice to transform beliefs and meanings aboutpractice. This finding is consistent with trajectories fordevelopment of professional reasoning.12

On the RSPQ-2F, adequate distributions of summedscores for the deep approaches and surface approaches tolearning subtests would detect differences among students.Individual items show an adequate distribution that alsosupports utility for measuring variations in theseapproaches to learning. Internal consistency was strong,further supporting the interrelatedness of items on each ofthe subtests for entry-level MOT and DPT participants.

Summed scores on the RSPQ-2F show that the graduatestudents in this study had higher mean scores for surfaceapproaches than for deeper approaches to learning. As pro-posed by Biggs and colleagues,7 these findings suggest thatgraduate health profession students use both approaches asthey continue to learn and apply knowledge across theirprofessional curricula. Changes in deeper approaches tolearning may be best examined using a pre/post-test format,either by course or by year in program.22 Hayes et al.22

noted significant changes in deeper approaches to learningwith DPT students, with noted limitations in stability withthe Revised Approaches to Studying Inventory.

In summary, both the QRT and the RSPQ-2F offer com-plementary methods for examining changes in RP neededfor practice and to stay current in health care professions.The scales for the QRT and RSPQ-2F were constructed in

134 DUNN, MUSOLINO, Assessing Reflective Thinking and Approaches to Learning

TABLE 3. Item Distribution for Questionnaire on Reflective Thinking (QRT)

Likert Scale Responses†__________________________________________________________________________________________2 3 4

1 Disagree with Use only if definite Agree with 5Strongly disagree reservation answer not possible reservation Strongly agree Dimension Question ________________ ________________ ________________ ________________ ________________

with Question Number n % n % n % n % n %

RT1 Habitual Action 9 7.2 22 17.6 5 4.0 57 45.6 32 25.6RT5 Habitual Action 8 6.4 33 26.4 11 8.8 60 48.0 13 10.4RT9 Habitual Action 54 43.2 52 41.5 6 4.8 13 10.4RT13 Habitual Action 43 34.4 60 48.0 8 6.4 13 10.4 1 0.8RT2 Understanding 34 27.2 91 72.8*RT6 Understanding 1 0.8 3 2.4 24 19.2 97 77.6*RT10 Understanding 5 4.0 1 0.8 40 32.0 79 63.2*RT14 Understanding 1 .8 5 4.0 3 2.4 46 36.8 70 56.0*RT3 Reflection 8 6.4 5 4.0 56 44.8 56 44.8RT7 Reflection 6 4.8 7 5.6 62 49.6 50 40.0RT11 Reflection 3 2.4 7 5.6 49 39.2 66 52.8*RT15 Reflection 4 3.2 4 3.2 64 51.2* 53 42.4RT4 Critical Reflection 1 .8 16 12.8 3 2.4 58 46.4 47 37.6RT8 Critical Reflection 5 4.0 43 34.4 11 8.8 52 41.6 14 11.2RT12 Critical Reflection 2 1.6 23 18.4 8 6.4 62 49.6 30 24.0RT16 Critical Reflection 4 3.2 24 19.2 14 11.2 59 47.2 24 19.2

† n = total student responses for rating (total n = 125), % = percentage of sample. Empty spaces show responses that did not occur withinthe item.* Items where >50% of participants reported this rating.

relation to Bloom’s Cognitive Scale.9,13 The unique featureof RT is the emphasis on relating concepts with otherknowledge and experiences, thus attaching a personalmeaning to the concept.13 Critical reflection then involvestransformation learning, such as may occur when studentsdevelop more sophisticated perspectives about practicemodels and an ability to evaluate and adopt newer infor-mation as they continue in practice, as early career profes-sionals, with a foundation in RP. With some noted limita-tions, results support the potential use of the QRT to assesschanges in RT as curricular outcomes for health professions.The RSPQ-2F provides a way for individual instructors toassess the extent to which the design of their courses pro-motes deeper learning needed for the development of RT.

IMPLICATIONS FOR SCHOLARSHIP OF TEACHING AND

LEARNING

Measurement of RP includes RT, motivation to stay current,to appraise new evidence, and to revise practice interven-tions in accord with contemporary research.23 Studies exam-ining effectiveness of educational programs in developing RTand approaches to learning are lacking. Studies examiningcompetency of health care professionals with years of experi-ence and their continued RP are also lacking23; however, sev-eral researchers have addressed aspects of RP in their works,for both novice and expert practitioners, especially withinthe profession of physical therapy.3,24–30 Clearly, RT is a crit-ical component within RP2 that extends beyond analysis toa personal investment in staying current with changes inresearch and interventions. Both experienced and novicepractitioners should never be fully satisfied,31 always asking“how can this be improved upon?,” challenging ourselves toRT in both practice and continuing competence.

DPT and MOT educational program goals emphasizepreparation of health professionals who are skilled with pro-fessional reasoning and fostering lifelong, self-directedlearning.16–18 Professional reasoning and motivation are req-uisites to becoming expert practitioners.12,24–29 Cultivationof these skills and attitudes involves higher-order thinkingskills, commitment, and RT, and therefore the integration ofexperience with knowledge.9,11,25–29 Perhaps continuedinvestigations, with both qualitative and quantitative meas-ures, will reveal further the learning outcomes associatedwith the development of a professional capable of RP? Sup-port for and barriers24 to RP warrant consideration whenexamining these abilities, as RP remains a multifaceted skillfor developing and practicing health professionals.3

Future investigations entail revising the QRT by addingitems to strengthen its internal consistency and relevanceto graduate-level MOT and DPT students. Next, we pro-pose to use the QRT and RSPQ-2F to examine changes inRT and approaches to learning that are integral to clinicalreasoning and evidence-based practice for our students andtheir patients/clients. Correlations with similar measures,such as the Revised Approaches to Studying Inventory, and

the relationship with criterion-related measures of reflec-tive learning would further validate use of these measureswith graduate-level health professional students.

LIMITATIONS

Limitations in this study included the use of a conveniencesample from one large Western U.S. university. The samplesize was insufficient to perform exploratory confirmatoryfactor analyses.

Conclusions

With noted limitations, these results support the use of theQRT and RSPQ-2F measures to assess changes in RT andapproaches to learning. The QRT and RSPQ-2F provideinsight into the examination of changes in aspects of pro-fessional reasoning and higher-order thinking processes forgraduate health professions students as they progressthrough MOT and DPT programs. Use of quantitativemeasures affords academic faculty opportunities to examinechanges within a semester, across the curriculum, and, later,in practice. However, both qualitative and quantitativeconsiderations must be considered to fully understandaspects influencing RP. Examining learning outcomes willassist faculty to obtain feedback about instructional strate-gies that may promote active learning within the curricu-lum and to obtain information about further developmentof RT and self-directed learning as practitioners.

The authors thank Dr. Kathleen Rockefeller, PT, ScD, MPH, Universityof South Florida, School of Physical Therapy & Rehabilitation Sciences,for her statistical consultations.

REFERENCES

1. Dewey J. How We Think: A Restatement of the Relation of ReflectiveThinking to the Educative Process. Boston: D C Heath; 1933.

2. Schön DA. Educating the Reflective Practitioner. San Francisco: Jossey-Bass; 1987.

3. Musolino GM, Mostrom E. Reflection and the scholarship of teach-ing, learning, and assessment. J Phys Ther Educ 2005; 19(3):52–66.

4. Mezirow J. On critical reflection. Adult Educ Q 1998; 48(3):185–198.

5. Lederer JM. Disposition toward critical thinking among occupa-tional therapy students. Am J Occup Ther 2007; 61(5):519–526.

6. Kember D, Yeung E, Leung DYP, et al. Development of a question-naire to measure the level of reflective thinking. Assess Eval HigherEduc 2000; 25:381–395.

7. Biggs J, Kember D, Leung DYP. The revised two-factor Study ProcessQuestionnaire: R-SPQ-2F. Br J Educ Psychol 2001; 71(1):133.

8. Entwistle N, Tait H, McCune V. Patterns of response to anapproaches to studying inventory across contrasting groups and con-texts. Eur J Psychol Educ 2000; 15(1):33–48.

9. Anderson LW, Krathwhol P. A Taxonomy for Learning, Teaching, andAssessing: A Revision of Bloom’s Taxonomy of Educational Objectives.New York: Longman; 2001.

10. Mezirow J. Learning as Transformation: Critical Perspectives on aTheory in Progress. The Jossey-Bass Higher and Adult EducationSeries. San Francisco: Jossey Bass; 2000.

11. Peltier JW, Hay A, Drago W. The reflective learning continuum:reflecting on reflection. J Market Educ 2005; 27:250–263.

Journal of Allied Health, Fall 2011, Vol 40, No 3 135

12. Schell BAB, Schell JW. Clinical and Professional Reasoning in Occupa-tional Therapy. Philadelphia: Lippincott, Williams & Wilkins; 2007.

13. Leung DYP, Kember D. The relationship between approaches to learn-ing and reflection upon practice. Educ Psychol 2003; 23(1):61–71.

14. Tabachnick B, Fidell LS, Fidell L. Using Multivariate Statistics, 4th ed.Boston: Allyn & Bacon; 2000.

15. Harwell MR, Gatti GG. Rescaling ordinal data to interval data ineducational research. Rev Educ Res 2001; 71:105–131.

16. Accreditation Council for Occupational Therapy Education. Historyof AOTA Accreditation. Bethesda, MD: American OccupationalTherapy Assoc.; 2010. Available from: http://www.aota.org/Educate/Accredit/Overview/38124.aspx. Accessed Dec. 17, 2010.

17. American Physical Therapy Association. Criteria for Accreditationof Education Programs for the Preparation of Physical Therapists.CAPTE Accreditation Handbook. Alexandria, VA: APTA; May 6,2011. Available from: http://www.capteonline.org/AccreditationHandbook/. Accessed July 1, 2011.

18. American Physical Therapy Association. Vision 2020 [webpage].Alexandria, VA: APTA; Mar 2011. Available from: http://www.apta.org/AM/Template.cfm?Section=Vision_20201&Template=/TaggedPage/TaggedPageDisplay.cfm&TPLID=285&ContentID= 32061. Accessed July 1, 2011.

19. Portney LG, Watkins MP. Foundations of Clinical Research: Applica-tions to Practice, 3rd ed. Upper Saddle River, NJ: Prentice HallHealth; 2009.

20. Schmitt N. Uses and abuses of coefficient alpha. Psych Assess 1996;8:350–353.

21. DeVellis RF. Scale Development: Theory and Applications. NewburyPark: Sage Publications; 1991.

22. Hayes KW, Sanders B, Healey WE. Students’ study approches in anew curriculum. J Phys Ther Educ 2010; 24(3):27–34.

23. Mann K, Gordon J, MacLeod A. Reflection and reflective practicein health professions education: a systematic review. Adv Health SciEduc 2009; 14(4):595–621.

24. Musolino, GM. Fostering reflective practice: self-assessment abilitiesof physical therapy students and entry-level graduates. J Allied Health2006; 35(1):30–42.

25. Jensen GM, Gwyer J, Hack LM, Shepard KF: Expertise in PhysicalTherapy Practice, 2nd ed. St. Louis: Saunders; 2006.

26. Jensen GM, Gwyer J, Shepard KF, Hack LM. Expert practice in phys-ical therapy. Phys Ther 2000; 80(1):28–43.

27. Jensen GM, Shepard KF, Hack LM. The novice versus the experi-enced clinician: insights into the work of the physical therapist. PhysTher 1990; 70:314–323.

28. Jensen GM, Shepard KF, Gwyer J, Hack LM. Attribute dimensionsthat distinguish master and novice physical therapy clinicians inorthopedic settings. Phys Ther 1992; 72:711–722.

29. Shepard KF, Hack LM, Gwyer J, Jensen GM. Grounded theoryapproach to describing the phenomenon of expert practice in physi-cal therapy. Qua Health Res 1999; 9:746–758.

30. Benner P. From Novice to Expert: Excellence and Power in ClinicalNursing Practice. Menlo Park, CA: Addison-Wesley; 1982.

31. Craik, RL. Never satisfied. J Phys Ther 2005; 85(1):1224–1287.

136 DUNN, MUSOLINO, Assessing Reflective Thinking and Approaches to Learning

Reproduced with permission of the copyright owner. Further reproduction prohibited without permission.