Running Head: QUANTITATIVE OBSERVATION Head: QUANTITATIVE OBSERVATION ... This paper describes...

Running Head: QUANTITATIVE OBSERVATION

In press at the Journal of College Student Development

An Inside View: The Utility of Quantitative Observation in Understanding

College Educational Experiences

Corbin M. Campbell

Teachers College, Columbia University

525 W. 120th St, Box 101

New York, New York 10027

[email protected]

212-531-5182

Acknowledgements: This research is supported by a National Academy of Education/Spencer

Foundation fellowship. Appreciation also to Marisol Jimenez, research assistant for the CEQ research

project, who developed the training protocols detailed in this paper.

mailto:[email protected]

QUANTITATIVE OBSERVATION 2


Abstract

This paper describes quantitative observation as a method for understanding college educational

experiences. Quantitative observation has been used widely in several fields and in K-12

education, but has had limited application to research in higher education and student affairs to

date. The paper describes the central tenets of quantitative observation, using an example

protocol, the College Educational Quality (CEQ) study to illustrate its potential application to

higher education and student affairs research. Quantitative observation allows researchers to

witness the educational process as it unfolds, and does so in a systematic way that enables

understanding patterns across time, groups, and settings.



An Inside View: The Utility of Quantitative Observation in Understanding College

Educational Experiences

The learning outcomes assessment movement has given rise to several new ways to consider

student outcomes, including self-reported learning outcomes, standardized tests, and analysis of

student work (Ewell, 2008; Kuh, 2001; Rhodes, 2012). In this context, the field of higher education

and student affairs continues to play an important role in developing an understanding of the

student experiences in educational practices that bring forth these outcomes (Calhoun, 1996; Inkelas

& Weisman, 2003; Kuh, 2009). The field has developed rich theory and applications to practice that

point to the complex dynamics among student’s individual, social, and historical contexts, their lived

experiences, and their educational environments that shape student outcomes (e.g. Gurin, Dey,

Hurtado, & Gurin, 2002; Jones, 2009; Renn, 2003; Torres, Jones, & Renn, 2009). The purpose of

this paper is to suggest the utility of quantitative observation as a method of data collection for

understanding college educational experiences as situated in specific student and educational

contexts.

Due to the use of expert raters, quantitative observation has the ability to understand

patterns in rich, theoretically-driven constructs, which can be applied to understanding student experiences

both in and out of the classroom. For example, quantitative observation could be used to

investigate the proportion of students who show dissonance during inter-group dialogue

discussions, the prevalence of heteronormative values displayed during by-stander intervention

programs, or the group dynamics in orientation conversations. Secondly, quantitative observation is

well-suited to understanding educational practices given that this method records data in real-time. For

example, a quantitative observational protocol could compare two alcohol education programs by

witnessing how the programs support students in progressing through stages of change.

While the use of quantitative observation is not common in higher education and student

affairs research, there are a few studies that explored this approach, largely focusing on in-class



experiences. In the early 1980s, Ellner and Barnes (1983) described the use of structured

observation for understanding college teaching. More recently certain studies have used a

quantitative observational protocol to study the complex dynamics among students, faculty, and

subject-matter in higher education (Campbell, 2015; Hora & Ferrare, 2014). In the field of

educational psychology and also k-12 research, classroom observation methods have been used

extensively to describe educational practices and to research inequities (Alexander & Winne, 2006;

Anderson & Burns, 1989; Hilberg, Waxman, & Tharp, 2004; Kane, Kerr, & Pianta, 2014).

Although it is not possible to describe all aspects of quantitative observation due to the

variation in this data collection method1, this paper will: 1) describe the central tenets of quantitative

observation and resources for implementation in research on college educational experiences; and 2)

illustrate the potential utility of this approach by discussing an example quantitative observational

protocol—the College Educational Quality project, an observational study of college teaching and

academic rigor in 587 courses across nine colleges and universities in the United States.

Quantitative Observation, Defined

Observational research in education, broadly construed, is using the five senses in a

systematic way to understand a social phenomenon (Angrosino, 2007; 2012). Observation can be

clinical (conducted in a lab setting with manipulation by the researcher) or naturalistic (conducted in

the setting of the social phenomenon as it naturally occurs); un-obtrusive (participants do not know

observation is taking place), reactive (the observer is known to the participants, but does not

participate), or participatory (observers actively participate in the phenomenon); qualitative (open-

ended; themes emerge from data) or quantitative (closed-ended; systematically coded according to

pre-defined theory) (Alexander & Winne, 2006; Angrosino, 2012). This paper focuses on

naturalistic, reactive, quantitative observational protocol due to its potential use in higher education

and student affairs research for understanding trends in educational experiences. I draw from the

1 I refer readers to Anderson & Burns (1989) and Angrosino (2007) for additional information about the method.



field of educational psychology, which understands quantitative observation as “a great range of

tools and techniques, both for generating the basic corpus of raw data and for processing those data

to develop quantitative scores or induce generalizations” (Alexander & Winne, 2006, p. 755).

Quantitative Observation Applied to Higher Education and Student Affairs

Stallings and Mohlman (1988) described several elements common across quantitative

observational protocols. I apply these categorizations to a higher education and student affairs

context in Table 1.

[INSERT TABLE 1 ABOUT HERE]

The College Educational Quality Study—Brief Introduction

To discuss the potential utility of quantitative observation as a method for understanding

college educational experiences, I describe the College Educational Quality study as an example.

The purpose of the College Educational Quality (CEQ) study was to explore a new

conceptualization and a new methodological protocol that allows for deeper insight into college

educational experiences in coursework. CEQ examines the course-based educational practices that

take shape between students and faculty, between students and course content, and among students.

Across these practices, CEQ considers two facets of educational quality: academic rigor and college

teaching. I focus on the cognitive complexity facet of the academic rigor framework in this paper as

an example for an observational protocol. The first phase of the CEQ project took place in spring

semester of 2013 at two research institutions. The second phase of the CEQ project took place in

fall semester of 2014, including seven additional institutional sites: five were liberal arts colleges and

two were regional public institutions. Approximately 350 courses were sampled at each institution,

stratified by class size, discipline, and faculty category. Among faculty with sampled courses, 34%

agreed to participate. Two observers visited one class mid-semester of 587 courses.



CEQ Study Observational Protocol

Below I describe how the Stallings and Mohlman’s (1988) elements of quantitative

observation apply to the CEQ study.

Conceptual basis for observation. Stallings and Mohlman (1988) describe three elements

of quantitative observation that relate to the conceptual framing of the study: the purpose, the

observational focus, and the operational definitions. The purpose of the CEQ observational study was

to understand college educational quality by viewing academic rigor and college teaching as they take

shape between students and faculty, between course content and students, and among students.

Given this purpose, the focus was on the educational practices in context. Across these practices,

academic rigor was operationalized into two constructs according to the conceptual framework, one of

which was the level of cognitive complexity (the focus of this paper). CEQ defined the level of

cognitive complexity in the coursework by the revised Bloom’s Taxonomy, which suggests that

academic work can require students to engage, cognitively, in six increasingly complex levels:

remember, understand, apply, analyze, evaluate, and create (Anderson & Krathwohl, 2001; Bloom,

Engelhart, Furst, Hill, & Krathwohl, 1956; Braxton & Nordvall, 1985). This conceptualization

guided the training of observers and the rubric development.

Observer training. According to Stallings and Mohlman (1988), the training procedures for

observers is a key aspect of structured observation that allows for greater conceptual depth. Observer

training for the CEQ study was approximately 30 hours, and included three elements: observation

procedures, knowledge of the conceptual frameworks, and tuning observer ratings using the rubrics.

The training on procedures addressed the logistics of observing (what to expect before, during, and

after observing) and observer behavior. The training on the conceptual frameworks focused on

whether the observers fully understood the ways in which academic rigor and college teaching were

defined in the study. The tuning training helped observers to map their understanding of the

conceptual frameworks onto rubric scores. To illustrate, I provide excerpts from the training



protocols in Table 2 below. Observers were required to pass a test on each of the three elements.

The tests culminated in observers scoring an entire class session of a college course and passing an

inter-rater certification, comparing observer’s scores against the master ratings from the PI.


Observation procedures. Stallings and Mohlman (1988) delineated four aspects of

structured observation that specify procedures: the setting, observational schedule, unit of time, and

method for recording the data. For the CEQ project, based on the purpose and conceptualization

of the study, the setting was classrooms. To understand trends in educational practices, the team

observed a large number of courses (50-100) per institution—and the courses were representative as

much as possible by discipline, class size, and faculty category. The CEQ project sought the

“average” class experience, so the schedule was one-week site visits in mid-semester (avoiding exams

or vacations). Two in-person observers2 rated the entirety of one class session for each course (unit

of time). Observers were matched to rate courses in their own disciplinary background whenever

possible. The observers rated real time—for example, observers would record the highest level of

cognitive complexity in the lecture or class discussions, no matter when that took place.

Methods to process and analyze data. According to Stallings and Mohlman (1988), there

are several ways to process the rich observational data into codes or scores for analysis (e.g.

checklists, evaluative scales, interactive coding schemes). Given that the purpose of the CEQ

project was to measure college educational quality in the aggregate and to provide comparative

information, the research team created an evaluative rubric following our conceptual frameworks. I

provide example rubric items, for the cognitive complexity construct in Figure 1, below. While

these items may not be easily understood upon first reading, the training protocols assisted

observers in knowing how to rate reliably and validly according to the conceptual frameworks.

2 There were some classes where, due to scheduling conflicts, only one observer was present.



Raters were trained to code what behaviors were “class activities” versus “class discussion” and how

to distinguish among the levels of Blooms revised taxonomy, such as “analyze” and “evaluate.”

One aspect of quantitative observation that differentiates this method from qualitative

observation is the ability to generate descriptive, comparative, and predictive statistics. Results from

the CEQ study revealed the proportion of courses that achieved higher order cognitive levels

(analyze, evaluate, create). Results were also used to compare these practices across institutional

contexts and to examine which course characteristics predicted a higher level of cognitive

complexity (author, 2015).

Reliability and Validity. Given the quantitative nature of observation scores, the method

lends to statistical validation procedures. Inter-rater reliability of CEQ data was calculated using a

one-way, absolute, average-measure, mixed-effects intra-class correlation (ICC) calculation (i.e. one-

way mixed-effects ANOVA; Hallgren, 2012). The ICC across all rubric items was .705,

demonstrating that inter-rater reliability was good (Cicchetti’s (1994) cutoff values .6-.74 = good).

The inter-rater reliability of the CEQ study demonstrates that theoretically rich constructs can be

rated with strong agreement among expert raters in higher education and student affairs.

The CEQ observational study included five scales, one of which was the cognitive

complexity construct. The research team conducted Confirmatory Factor Analyses (CFA) using

MPlus to determine the construct validity and the relationships among the five constructs. Model fit

indices indicated excellent fit of a 5 factor intercorrelated model (RMSEA=.049, CI [.041, .057],

CFI=.965, TLI=.956, SRMR=.047). The reliability of constructs was high (Coefficient-H: .809 -

.970). Readers can obtain a full description of the construct validation process and results by

contacting Author (author email).

Limitations of Quantitative Observation

While quantitative observation may show promise for research on college educational

experiences, there are several limitations that should be considered. Perhaps the most highly cited



limitation is that quantitative observation requires more time and funding than some other

quantitative methodologies, such as survey research (Anderson & Burns, 1989; Hilberg, Waxman, &

Tharp, 2004). Although the researcher burden and cost may be higher than in survey method, it has

been successful on large scales in k-12 education (Kane, Kerr, & Pianta, 2014) and has begun to

receive funding in higher education (Hora & Ferrare, 2014). The CEQ project made a broad-scale

observational protocol more cost-effective by using volunteer observers (60+ graduate students in

higher education and student affairs programs). Further, the colleges involved in the CEQ study

were willing to pay in-kind in order to participate, by housing observers (making the cost

manageable for both the research team and participating institutions). A second limitation is

reactivity, subjects of observation may react differently and alter their behavior when observed

(Jacob,Tennenbaum, & Krahn). For example, students may be more conscious of their behaviors if

an observer is present. There are certain practices that can ameliorate reactivity: being as

unobtrusive as possible, guaranteeing confidentiality, and focusing on aggregate rather than

individualized results (Anderson & Burns, 1989; Jacob, Tennenbaum, & Krahn, 1987).

Finally, the validity of the observations will only be as good as the framework in the study.

In essence, the validity of quantitative observational data is only as clear as the glasses of the

researcher—the data paint a picture of the classrooms as the researcher sees it (assuming good

training protocols). While this is a limitation, this same aspect of quantitative observation means

that the method can closely follow a conceptual framework. If a study does use a strong

conceptualization and has good training procedures for raters, the data captured may be able to

more accurately report patterns of complex phenomenon in the classroom (Hilberg, Waxman, &

Tharp, 2004). In particular, specifying behaviors that are highly agreed upon by both practitioners

and scholars (e.g. higher education researchers and teaching faculty) as easily codable, such as in a

structured, interactive rubric coding process, may allay some of these validity critiques (Stallings &

Mohlmann, 1988).



Discussion

There are four features of quantitative observation that highlight its potential utility in higher

education and student affairs (Table 3). Although quantitative observation has had limited use in the

body of scholarship on higher education and student affairs to date, the time is ripe for the broader

introduction of this method. In a time when assessment of student outcomes is at the forefront of

higher education and student affairs as a field, quantitative observation (as a supplement to the

existing survey research) may contribute to a deeper understanding of what educational contexts

shape student outcomes. This paper used the CEQ study as an example of how quantitative

observation may apply in higher education research. Using observers who were experts in

understanding Bloom’s taxonomy revealed new insights about the level of cognitive complexity in

the classroom. Yet, this study was limited to in-class observations. Quantitative observation has,

largely, not been applied to examinations of the co-curriculum, to student developmental processes,

or to student affairs administration.


As the field continues to progress in understanding the complex and situated nature of

educational practices and student identities, quantitative observation may provide a useful additional

methodological tool for the higher education and student affairs researcher. The use of expert

raters may be more valid for coding certain theoretically-rich and contextualized constructs when

compared to a student’s self-report in a survey (Hilberg, Waxman, & Tharp, 2004). For example,

while students may report in a survey how often they interacted with racially diverse others (Gurin,

Dey, Hurtado, & Gurin, 2002), quantitative observation using observers who are expert in inter-

group contact theory (Allport, 1954; Pettigrew, 1988) might reveal whether student experiences with

these diverse others met the “common goals” condition for optimal inter-group contact.



Additionally, the quantitative nature of the scores garnered through structured observation lends to

demonstrating patterns of results across contexts. These descriptive and comparative statistics

would not be possible with qualitative observation (although qualitative observation has other

advantages, such as providing rich descriptions of educational processes). In these ways,

quantitative observation may offer a new window for asking and answering different questions that

shed light on patterns of complex educational practices in the field of higher education and student

affairs.

References

Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing:

A revision of Bloom's taxonomy of educational objectives. NY: Addison-Wesley Longman.

Alexander, P. A., & Winne, P. H. (Eds.). (2006). Handbook of educational psychology. Psychology Press.

Anderson, L. W. & Burns, R. B. (1989). Research in Classrooms. The study of teachers, teaching, and

instruction. Elmsford, NY: Pergamon Press.

Angrosino, M. V. (2007). Naturalistic Observation. Walnut Creek, CA: Left Coast Press.

Angrosino, M. V. (2012). Observation-based research. J. Arthur (Ed.). Research Methods and

Methodologies in Education, (pp. 165-170). Thousand Oaks, CA: Sage.

Astin, A. W. (1993). What matters in college? Four critical years revisited. CA: Jossey-Bass.

Bloom, B. S., Englehart, M. B., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy

of educational objectives, the classification of educational goals – Handbook I: Cognitive domain. New

York, NY: McKay.

Braxton, J. M., & Nordvall, R. C. (1985). Selective liberal arts colleges: Higher quality as

well as higher prestige? The Journal of Higher Education, 538-554.

Calhoun, J. C. (1996). The student learning imperative: Implications for student affairs. Journal of

College Student Development, 37(2), 188-122.



Campbell, C. M. (2015). Serving A Different Master: Assessing College Educational Quality

for the Public. Higher Education: Handbook of Theory and Research, 30, 525-579.

Cicchetti D.V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and

standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290.

Gurin, P., Dey, E., Hurtado, S., & Gurin, G. (2002). Diversity and higher education: Theory and

impact on educational outcomes. Harvard Educational Review, 72(3), 330-367.

Ewell, P. T. (2008). Assessment and accountability in America today: Background and context. New

Directions for Institutional Research, 7-17.

Ellner, C. L., & Barnes, C. P. (1983). Studies of college teaching. Lexington, MA: Lexington Books.

Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and

tutorial. Tutor Quantitative Methods Psychology, 8(1), 23-34.

Hilberg, R. S., Waxman, H. C., & Tharp, R. G. (2004). Purposes and Perspectives on Classroom

Obvservational Research. In H. C. Waxman, R. G. Tharp, & R. S. Hilberg, (Eds),

Observational Research in U. S. Classrooms: New Approaches for Understanding Cultural and Linguistic

Diversity. (pp. 1-20). West Nyack, NY, USA: Cambridge University Press.

Hora, M.T. & Ferrare, J. J. (2014). Remeasuring postsecondary teaching: How singular

categories of instruction obscure the multiple dimensions of classroom practice. Journal

of College Science Teaching, 43(3), 36-41.

Inkelas, K. K., & Weisman, J. L. (2003). Different by design: An examination of student outcomes

among participants in three types of living-learning programs. Journal of College Student

Development, 44(3), 335-368.

Jacob, T., Tenenbaum, D. L., & Krahn, G. (1987). Factors Influencing the Reliability and Validity of

Observation Data. In. T. Jacob. (Ed.). Family Interaction and Psychopathology. NY: Springer.

Jones, S. R. (2009). Constructing identities at the intersections: An autoethnographic exploration of

multiple dimensions of identity. Journal of College Student Development, 50(3), 287-304.



Kane, T. J., Kerr, K. A., & Pianta, R. C. (2014). Designing teacher evaluation systems. CA: Jossey-Bass.

Kuh, G. D. (2009). What student affairs professionals need to know about student engagement.

Journal of College Student Development, 50(6), 683-706.

Kuh, G. D. (2001). Assessing what really matters to student learning inside the national survey of

student engagement. Change: The Magazine of Higher Learning, 33(3), 10-17.

Pettigrew, T. F. (1998). Intergroup contact theory. Annual review of psychology, 49(1), 65-85.

Renn, K. A. (2003). Understanding the identities of mixed-race college students through a

developmental ecology lens. Journal of College Student Development, 44(3), 383-403.

Rhodes, T. L. (2012). Show me the Learning: Value, accreditation, and the quality of the degree.

Planning for Higher Education, 40(3), 36-42.

Stallings, J. A., & Mohlman, G. G. (1988). Classroom observation techniques. In J. P. Keeves

(Ed.), Educational research, methodology, and measurement: An international handbook. (pp. 469-474).

Elmsford, NY: Pergamon Press.

Torres, V., Jones, S. R., & Renn, K. A. (2009). Identity development theories in student affairs:

Origins, current status, and new approaches. Journal of College Student Development, 50(6),

577-596.

Waxman, H. C., Tharp, R. G., & Hilberg, R. S., Eds (2004). Observational Research in U. S.

Classrooms: New Approaches for Understanding Cultural and Linguistic Diversity. West

Nyack, NY, USA: Cambridge University Press.



Table 1

Main Elements of Quantitative Observational Method (Stallings & Mohlman, 1988)

Element Application to Higher Education and Student Affairs Research

Purpose for the Observation

Observation is ideally suited to understand student, faculty, and administrator behaviors and educational processes as they unfold in their naturalistic setting (Hilberg, Waxman, & Tharp, 2004). For example, it can be used to examine inequities in campus climate or the facilitation of leadership development programs. Observation is less suited to understanding perceptions and outcomes.

Specific Observational Focus

Given that educational processes and behaviors could be viewed in infinite ways, a conceptual framework can delineate, specifically, what aspects of the observational setting are to be examined (Anderson & Burns, 1989). For example, the observation could focus on interactions among students, facilitation skills among staff, group dynamics, or the climate within the setting.

Operational Definitions

Because quantitative observation requires trained observers to code what they see into specific categories of interest to the research questions, there should be very specific operational definition of all observed behaviors. These operational definitions can be derived from a conceptual framework that undergirds the research and are further specified in a rubric or other coding scheme. For example, an examination of racial climate in the residence hall might use a conceptual framework of sense of belonging and then operationalize how “belonging” would be coded into the rubric during observation.

Training Procedures for Observers

Given that expert observers are a linchpin of good data collection in quantitative observation, the training for observers is key to valid and reliable data (Anderson & Burns, 1989). These training procedures usually detail, in depth, the procedures for observation (e.g. time period, when to observe, what to observe, how to observe) and the conceptual frameworks for the study (to ensure inter-rater reliability). For example, observers of study habits in a student union could be trained to pay attention to the amount of collaboration in study groups.

A Setting Quantitative observation is an applied method where data collection occurs in a naturalistic setting (Stallings & Mohlman, 1988). Therefore, a quantitative observational study will specify where, specifically, the observations will take place—e.g. classrooms, residence halls, libraries, student unions.

An observation schedule

The observation schedule is the amount of time the observation will take and how many observations will take place during the specified time period. For example, will the observation be a week-long site visit? A day in fall and spring semester? A 3 hour course or program?

A Unit of Time The unit of time refers to how the observation data are recorded within one observation session. For example, a study of service learning could use the time sample method—recording what happens every 2 minutes during a reflection discussion, or the real time method, recording all critical behaviors that happened during that reflection discussion regardless of when those behaviors happened (Stallings & Mohman, 1988).

A Method to Record the Data

Will observations be video-taped, audio-recorded with supplemental notes from an in-person observer, will an in-person observer record the data in paper and pencil, computer, online software to capture the data? For example, a study of social justice emphasis in learning communities might be captured via video, but, perhaps, an in-person observer would be more appropriate in a study of racial climate and norms.

Methods to Process and Analyze Data

According to Stallings & Mohlman (1988), there are three primary ways to code the data in quantitative observation: 1. Checklists—e.g. at certain time intervals record which pedagogical technique faculty use; 2. Ratings or evaluative scales—e.g. how welcoming was the resident advisor on scale of 1-5?; 3. Interactive coding schemes or “systematic observation”—uses a simplistic rubric to capture easily identifiable behaviors that do not require the observer to make strong inferences or judgments (Galton, 1988; Hilberg, Waxman, & Tharp, 2004)—e.g. could be used to capture the flow of interactions among student group meetings.

Reliability According to Alexander & Winne (2006), there are three forms of reliability to consider in a quantitative observational protocol: inter-rater reliability (do observers’ ratings match each other?); sampling reliability (would sampling other classes produce different results?); and coding reliability (are the codes reliably distilled in the same manner each time from observed behaviors?).



Table 2

Excerpts from CEQ training scripts.

Training Component

Example Language from Training Script

Procedures Training

DURING THE OBSERVATIONS Please arrive 5-10 minutes before the beginning of the class to introduce yourself to the professor. Please do the following: □ Try to sit in the back, middle of the class so you can observe what is happening in the different parts of the room….However, also scan the room periodically to see whether the patterns of behavior or actions you see around are reflected in more distant areas of the room.

Knowledge of Frameworks Training

In 1956, Benjamin Bloom, along with a group of other educators, devised a classification system of the learning objectives that educators could expect of, or set for, students in a teaching context. The classification system consisted of six categories addressing the cognitive, affective, and psychomotor domains of learning….Bloom et al (1956) argued that the different categories represented a continuum from simple to increasingly more complex thinking, and that to address objectives at the upper levels of the continuum, teachers needed to have addressed skills and abilities at the lower levels of the continuum…In 2001, cognitive psychologists, testing and assessment experts, as well as other educators revised the taxonomy. The revised taxonomy consists of six categories: Remember, Understand, Apply, Analyze, Evaluate, and Create. …

Tuning Ratings Training

BLOOM’S TAXONOMY: THE ANALYZE LEVEL When students are asked to think at the level of analysis, they are asked to engage in differentiating, distinguishing, deconstructing, and comparing and contrasting. In essence, they are asked to compare attributes of diverse concepts, processes, or events, without rendering an opinion or judgment… For example, an assignment that calls for analysis could ask students to analyze an article in an academic journal. To complete such an assignment, students might be asked to indicate the article’s thesis, the parts of the article that support the thesis, the underlying assumptions of the author’s thesis, the implications of the author’s argument, and the possible implications of the author’s thesis….Tip: To differentiate between apply and analyze remember that applying calls for the use of one’s knowledge in new situations. Analysis calls for the skills to examine and break information into parts. The end goal of applying is using, while the end goal of analyzing is comparing and deconstructing.

Table 3

Utility of Using Quantitative Observation for Research in Higher Education and Student Affairs

Element Utility for Higher Education and Student Affairs Research

Real-time data collection

One of the most prominent features of quantitative observation is its ability to view educational experiences in process: “data are collected on classroom practices as they unfold” (Alexander & Winne, 2006, p. 755). Data collection does not suffer from possible recall biases, which has been noted as a limitation of survey responses (Porter, 2011; 2013).

An Applied Method

Quantitative observation allows researchers to study educational processes in naturalistic settings (Hilberg, Waxman, & Tharp, 2004). In this way, quantitative observation could be used to understand the application of educational interventions, change processes, and the genuine interactions among students.

Allows for Conceptual Training

Because quantitative observation uses trained observers to code data, quantitative observational studies can use more complex conceptual frameworks to examine educational processes (Anderson & Burns, 1989). Quantitative observation allows for “tuning” rubric ratings specifically to the conceptual framework intended in the study.

Understanding Trends in Subtler Educational Processes

Because quantitative observation allows for viewing and collecting data on subtler processes and is a more proximal data collection method than many other quantitative data collection methods, it may allow for more detailed and precise evidence about broader trends in college educational experiences (Anderson & Burns, 1989; Good, 1988).



Figure 1. Excerpt from Cognitive Complexity items in CEQ observation rubric

Remember: Recognize, recall, repeat back (State back the same information; Recall information) Understand: Exemplify, classify, explain, summarize (State back the same information in a different way or with additional examples) Apply: Execute, implement (Apply the same concept to a new setting, to the field, or to a real world problem) Analyze: Differentiate, distinguish, deconstruct, compare and contrast (Discuss and compare attributes of the idea—without forming an opinion) Evaluate: Critique, test, judge (Discuss strengths and limitations AND form an opinion about an idea, substantiate one’s opinion about the idea) Create: Generate, produce, construct, hypothesize (Take this idea and make something new from it—more than applying the same concept, actually creating a new concept different from the learned concept)

Please rate both the average level attained during the lesson (mark A) and highest level attained by the end of the lesson (mark H).

With regard to the class’s subject matter….

RE

ME

MB

ER

UN

DE

RS

TA

ND

AP

PL

Y

AN

AL

YZ

E

EV

AL

UA

TE

CR

EA

TE

NA (i.e. the class did

not contain lecture,

handouts, activities, questions,

discussions)

The instructor’s lecture reflected what level of cognitive processing?

The level of the handouts or other visual material in class reflected what level of cognitive processing?

The class activities required students to . . .

The questions asked by the instructor required students to…

The class discussions demonstrated students’ ability to. . .

The questions asked by students demonstrated students’ ability to…

Running Head: QUANTITATIVE OBSERVATION Head: QUANTITATIVE OBSERVATION ... This paper describes...

Documents

Transcript of Running Head: QUANTITATIVE OBSERVATION Head: QUANTITATIVE OBSERVATION ... This paper describes...