Student Affairs Assessment Committee Training

Assessment Fellows Training

Brian Clark, Stan DuraAssessment Fellows Meeting

4/17/24

Brian Clark, Stan Dura 11/15/13

Training Agenda

• Research, Assessment, & Program Evaluation• Foundational Concepts

– Measurement is imprecise– “Things and the stuff about them”– Variables, variance, and modeling

• Q & A

Ruler 1 Ruler 2 Ruler 3 Ruler 4 Ruler 5 Ruler 6 Ruler 7 Ruler 8 Ruler 9 Ruler 10 Laser Ruler

11.85

11.9

11.95

12

12.05

Chart Title

Series1

Measurement is imprecise



11.85

11.9

11.95

12

12.05

Chart Title

Series1

Surveys are especially vulnerableWhat

“construct” are you measuring?

Survey and question design

and content

Responses




and content

Responses

Is it really measuring it?Does it measure it the same each time?




and content

Responses


Any errors in the questions or scales?Any bias in the questions asked or not asked?




and content

Responses



Response set– -Are they representative?-How complete are they?-Is there non-response bias?




and content

Responses



Response set– -Are they representative?-How complete are they?-Is there non-response bias?

Over 80 cognitive and

memory biases

Respondents prone to error• Response is a cognitive act. 4 Stages:

• Comprehension• Retrieval • Judgment• Response

All kinds of errors can happen in these stages

Respondents prone to errorCognitive Stages Definition Vulnerabilities

Stage 1 Comprehension Understanding the question as author intended

Unknown or misused words, ambiguity, complexity, length

Stage 2 Retrieval Search of memory for relevant information

Memory bias, recall error, fatigue

Stage 3 JudgmentConsiders information

retrieved, makes “guestimations” and decides

Social or political bias, “fuzzy logic,” error in “guesstimation”,

personal sensitivity

Stage 4 Response Provides the information Human error, incomplete response, wrong format

Respondents prone to error• Change bias – Remembers as more difficult• Context effect – out of context more difficult• Consistency bias – thinking past attitudes same

as current• Anchoring – relying too heavily on one piece of

info in order to make an estimation• Availability heuristic – overestimation due to

recency or emotional strength of memory

Respondents prone to error• Confirmation bias – tendency to search for and

remember info confirming one’s preconceptions• Framing effect – drawing different conclusions

or interpretations base on how info is presented• Optimism bias – tendency to overestimate

likelihood of positive outcomes• Social response bias – tendency to underreport

socially undesirable behaviors and vice/versa.

Lots more!!!!

Proof is in the Polls• Ever seen this?

• Down in the polls by 6 points• Down in the polls by 3 points• Ahead in the polls by 1 point• Landslide victory with 60% of the votes!

Keep these limitations in mind when drawing inferences from survey data.

Research, Assessment, & Program Evaluation

Research• Systematic collection of data• Generalize to larger populations• Based on Research question• Rigorous methodology• Used to test hypotheses• Contributes to the development

of theory and models

Assessment• Systematic collection of data• Generalize to larger population• Based on Assessment question• Borrows rigor when practical• Used to evaluate effectiveness• Contributes to judgments of the

quality of programs or activities

Program Evaluation:• Act of using data (assessment, operational,

etc.) related to outcomes, inputs, and processes of a program

• To assign judgment on its effectiveness

Research, Assessment, & Program Evaluation

Assessment (and statistics) is about describing:

• Things• Stuff pertaining to those things• The relationship between things & stuff

Foundational Concepts“Things and Stuff”

What is a “thing”?• An entity of some kind• An idea• A quality or characteristic


What is “stuff”?• Material out of which something is made• Essential substance or elements• Essence


Things are the objects we assess, and

Stuff characterizes those things


Let’s use more concrete terms:

• Entity – something that exists separate from its parts

• Property – a characteristic, trait or attribute and other stuff that describes something


Types of entities:• Organisms (including humans, trees, etc.)• Physical objects (rocks, buildings, etc.)• Actions and events (running, stroke, etc.)• Cognitive phenomena (ideas, emotions, etc.)• Organizations (governments, boards, etc.)• Scientific entities (waves, motivation, etc.)• Math entities (numbers, functions, vectors, etc.)


Different Properties:– Height– Age– Perceptions– Weight– Density– GPA– Level of engagement– Magnitude– Levels and expressions of anger, fear, anxiety


Notice that some properties could be examined as entities themselves– Relaxation– Motivation– Intelligence– Satisfaction– Health


What entities are we concerned with?


What properties of those entities concern us?


Entities and properties– Humans

• Height, satisfaction, health, perceptions, emotions, beliefs, etc.

– Students (whether human or not )

• Age, GPA, knowledge, skills, engagement, class standing, etc.

– Physical Objects• Weight, chemical composition, density, etc.

– Forces• Magnitude, direction, etc.


Most critical thing about properties -

they vary– Height varies– Age varies– Perceptions vary– Weight varies– Density varies– GPA varies– Level of engagement varies– Levels and expressions of anger, fear, anxiety vary


These are the first two foundational concepts:• Entity• PropertyThe others are:• Variable• Prediction and Control• Relationship• Statistical techniques


Properties are represented by variables - – Height is represented by inches, meters, or relational terms (bigger)– Age is represented by months, days, years or relational terms (older)– Weight is represented by kilograms, pounds or relational terms– Density is represented by pound per square inch, etc.– Grades can be represented by symbols (A+), numbers (92), etc.– Engagement can be represented by number of activities attended,

etc.


Some variables are straightforward-

– Height is often measured in Inches– Weight is often measured in Kilograms– Age is often measured in years


Some variables are not-

– Satisfaction is often measured in… ???– Anger is often measured in… ???– Engagement is often measured in… ???


Thus variables are just representations

of the properties


That describe an entity to some degree


That describe an entity to some degree

That may be precise or not


Broad goal of assessment/research: To understand the population

Two underlying fundamental goals:– Prediction and control of variables– Understanding of entities & their properties

Which is more important?

Foundational ConceptsPrediction and Control

Let’s pretend we are looking at student performance of a complex task:

Let’s look at these statements:

– Haste makes waste– Practice makes perfect

How would we assess these?


What are our variables?

– What variables might represent haste?– What variables might represent waste?– What variables might represent practice?– What variables might represent perfection?

What is our model?


What is our model?

– More haste = more waste?• Some of the waste is the result of haste?• Haste predicts waste?

– Simply looking at means, which is what we often do:• Those that were hastier answered fewer questions

correctly than those who took more time.


Waste

Haste


Waste

Haste

• The red area represents the degree to which Waste is due to Haste.• The green area is the degree to which Waste is influenced by other factors


What is our model?

– More practice = better performance?• Some of the performance is the result of practice?• Practice predicts performance?

– Simply looking at means, which is what we often do:• Those who practiced more scored higher than those who

practiced less.


Performance

• What is the red area?• What is the green area?

Practice


Is it really that simple?

– What if hasty practice results in worse performance?

– How do we know the true impact of practice, if we don’t remove the impact of haste?


So in order to accurately understand entities and properties, we must be able to predict and control.

– Prediction and control are more important – accurate understanding depends on it

– We must understand relationships to do so


Back to those statements

– Haste makes waste– Practice makes perfect

Both are engrained in our culture and both are about relationships

Establishing relationships between objects is how we naturally understand the world


Let’s operationalize this?

– Haste = time on task (in minutes)– Practice = # of repetitions– Performance = Evaluation score

• (waste = lost points or distance from a perfect score)


Our Model

• Performance = Practice + Haste

Or more specifically…• Evaluation Score = # Repetitions + Minutes on Task


Each variable contributes to the variance in evaluation scores

scores

# repetitions

minutes on task


scores extent to which observed scores vary across all individuals

We need to control for the impact of Haste on Performance in order to understand the impact of Practice.



scores

# repetitions

extent to which # of repetitions varies

across all individuals

extent to which minutes on task varies

across all individuals


scores

# repetitions

minutes on task


scores

# repetitions

minutes on task

variance in scores that we can’t explain (individual

differences, fatigue, human error, etc.)


scores

# repetitions

minutes on task

Variance attributed to the # of repetitions


scores

# repetitions

minutes on task

This is the degree to which “Practice Makes Perfect”

scores

# repetitions

minutes on task


This is the variance explained by

minutes on task


scores

# repetitions

minutes on task

In other words, the degree to which

“Haste makes waste”


scores

# repetitions

minutes on task

SHARED variance between minutes on task

and # repetitions


scores

# repetitions

minutes on task

variance in Scores attributed to the combination of

minutes on task AND # repetitions


scores

# repetitions

minutes on task

variance in scores uniquely attributed

to # repetitions


scores

# repetitions

minutes on task

Put another way: variance in scores

attributed to # repetitions AFTER

variance attributed to

minutes on task is controlled

Brian Clark

changed "The variance in scores attributed to Practice (repetitions) – AFTER the variance attributed to Haste is controlled for." to "variance in scores attributed to # repetitions AFTER variance attributed to minutes on task is controlled"

Our Model

– Performance = Practice + Haste


Performance

PracticeHaste


If we did not Control for Haste, we would think all of this was

the result of PracticePerformance

Practice


When really, this was the impact.

Performance

Practice


So…• Recognizing the relationships between

variables, and • controlling for them • is critical to understanding the impact

of any one of them.


Why is this important in assessment?• Consider you are studying the impact of stress on student

persistence?• You ask students to describe their stress level at one point in

time, and you see that:– 80% of those who reported high stress dropped out, while only– 25% of those who reported low stress did.

• You devote 70% of a staff person’s time to doing stress relief programming, totaling appx. $60,000


What variables are there in stress and persistence?

Stress• Family support and relationships• Peer support and relationships• Coping skills• Academic work load• Financial Support• Employment work load• Work environment

Persistence• Academic advising• Social connectedness• Engagement• Faculty / Staff connections• Institutional Bureaucracy• Meaningful learning experiences• Support Services


Stress• Family support and relationships• Peer support and relationships• Coping skills• Academic work load• Financial Support• Employment work load• Work environment

Persistence• Academic advising• Social connectedness• Engagement• Faculty / Staff connections• Institutional Bureaucracy• Meaningful learning experiences• Support Services

• What should the staff member focus their efforts on? • Is it possible that general Stress Relief may not be a very influential variable? • Would the $60k be well spent?


When we conduct quantitative or qualitative analyses, we are constructing models

• That model can be very, very simple• Mean – this is how the average person did…• Mode – The most frequent score was X• Standard Deviation – roughly 68% scored between these two

points• Range – All students scored between X and Y.

• Descriptive stats = the weakest and least informative models


When we conduct quantitative or qualitative analyses, we are constructing models

• Other models can be complex– For every 1 repetition of Practice, when Haste is controlled, Evaluation

Scores increased by 2.4 points.– For every hour engaged in co-curricular activities, when controlling for HS

GPA, Class standing, and Major, retention likelihood increases by .45%

– More complex stats = more informative and more accurate models


So when we assess, we need to Think of our Model

• What are the entities and properties we’re examining?• What are the variables involved? Operationalize them.• What is their relationship; how do they interact?• How simple or complex/accurate a model do I need?• How can we predict and control for these variables?


Questions?Concerns?

Snide Comments?

HomeworkOffice-work

• Use your notes and slidedeck to create a study guide

• Email to me by next Friday

Student Affairs Assessment Committee Training

Education

Transcript of Student Affairs Assessment Committee Training