Why Do we (Really) Report all this Data?

Responding to Data Creep and Data Mania in Higher Education

Stephen Chemsak, Ed.D. Director, Institutional Research and Analysis, SUNY Cortland

AIRPO 2015 Conference

When I first started doing IPEDS reporting, I wondered why almost no one at the College seemed to care.

One of the main purposes of higher ed. data is to provide information or help make decisions, right?

Yet we notice subjective, limiting, or even irrational aspects of data.

• Data have specific definitions that restrict the scope of what they describe.• Data definitions constantly are being

refined and redefined.• The same data are collected over and

over again even if this does not seem to provide more information.• Sometimes it seems like no one uses the

data produced.

GoalTo give you tools to think

critically about data in higher education and

suggest ways to respond.

My thinking turned to the fields of sociology and philosophy…

Different theoristscan help us understand

why.

John Meyer,

(1935-)Sociologist

Thomas Kuhn (1922-66),

Philosopher of Science

Michel Foucault, Social Theorist/Philosopher (1926-84)

“Everyone’s Doing it!”Meyer and Institutional Theory:

Institutions strive for Legitimacy. Who wants to be different?

“Rational myths” – A myth is a story without basis in fact. Something rational is based on reason or logic.

Isomorphism: “being like the other guy”+ Coercive: pressures such as mandates/rules(e.g. accreditation). + Mimetic: copying other “successful” examples (they collect the data, I should too!).+ Normative: positive cultural practices/standards (that’s the way we did it last year!).

Normative, Mimetic, and Coercive Pressure: Harvard’s Graduation rate every year for the past 10 years.

An example of normative and coercive pressure:

Enrollment monitoring reports every Monday, Wednesday, and Friday for 6 weeks that each take approximately 3 hours each to produce over the course of many years.

Thomas Kuhn: Scientific ParadigmsThere are multiple ways to do science. Ultimately, science is grounded in judgements.

In contrast to Kuhn, two ideologies are prevalent in higher education research today:

Scientism: Science is objective. Science is the main or best way to understand reality.

Positivism: Scientific knowledge is the only kind of knowledge.

“Whenever you get two people interpreting the same data in different ways, that’s metaphysics.”

-Thomas Kuhn

Matching Metric to Goals or Outcomes: Examples of Non-scientific, Subjective, Judgement-based Moves

Graduation Rates

College Educational

Benefits?

Measure of being “the best” – e.g. U.S.

News and World Report?

Measure of College Accountability?

Do graduation rates reflect institutions or students?

0 0.2 0.4 0.6 0.8 1 1.20

100

200

300

400

500

600

700

satmt25Linear (satmt25)

Axis Title

Axis Title

0 500 1000 1500 2000 2500 3000 35000

0.2

0.4

0.6

0.8

1

1.2

grad rateLinear (grad rate)

Axis Title

Axis Title

SAT Scores and Graduation Rate

Pell Grants and Graduation Rate

Governmentality

“The ensemble formed by the institutions, procedures, analyses and reflections, the calculations and tactics that allow the exercise of this very specific albeit complex form of power, which has as its target population, as its principal form of knowledge political economy, and as its essential technical means apparatuses of security.”

-Michel Foucault

Foucault Continued

• Know your population.• State-istics (derived from

term “state)• Govern – mentality• An epistemology of suspicion.

“Discipline and Punish” Colleges: The Abuse of Educational Statistics

Test scores used as accountability measures for teacher education programs.

Campus retention rates used to reward campuses.

Use of student survey data to evaluate senior administration.

Subjective, limiting, or even irrational aspects of data

Definition or question wording. Data source. Matching or linking of metrics: to goals, “success”, or “achievement” is subjective. In psychometrics, we may think of this as face validity or content validity. Unit of analysis (why the institution?) Peers or points of comparison. Backside interpretation.

Cycle of Data in Higher Ed.

Cultural and

social ideologie

s

“Evidence”

Metrics will

provide evidence

But, metrics

not objective: Conflict!

More, or redefined, metrics

More Broadly…

Human desire to understand reality

Science will build confidence in knowledge

Institutionalization of scientism

Government promotes tools of science to understand education

(even if tools have serious shortcomings).

So, what to do about it?1. In conversations with policy makers,

senior administrators, and data users, stress data’s subjective components.

2. Ask questions about how data will be used. Push back against data’s irrational aspects.

3. Be meticulously correct in calculations. Many aspects of data are subjective, but not all of them!

4. Just because we did it that way in the past doesn’t mean we should do it the same way in the future. (Don’t follow the lemmings!) . 5. Students are learning. Education is happening. Whether we measure it or not. Cultivate trust .