Assessment Fellows Training
Brian Clark, Stan DuraAssessment Fellows Meeting
4/17/24
Brian Clark, Stan Dura 11/15/13
Training Agenda
• Research, Assessment, & Program Evaluation• Foundational Concepts
– Measurement is imprecise– “Things and the stuff about them”– Variables, variance, and modeling
• Q & A
Ruler 1 Ruler 2 Ruler 3 Ruler 4 Ruler 5 Ruler 6 Ruler 7 Ruler 8 Ruler 9 Ruler 10 Laser Ruler
11.85
11.9
11.95
12
12.05
Chart Title
Series1
Measurement is imprecise
Ruler 1 Ruler 2 Ruler 3 Ruler 4 Ruler 5 Ruler 6 Ruler 7 Ruler 8 Ruler 9 Ruler 10 Laser Ruler
11.85
11.9
11.95
12
12.05
Chart Title
Series1
Measurement is imprecise
Ruler 1 Ruler 2 Ruler 3 Ruler 4 Ruler 5 Ruler 6 Ruler 7 Ruler 8 Ruler 9 Ruler 10 Laser Ruler
11.85
11.9
11.95
12
12.05
Chart Title
Series1
Measurement is imprecise
Ruler 1 Ruler 2 Ruler 3 Ruler 4 Ruler 5 Ruler 6 Ruler 7 Ruler 8 Ruler 9 Ruler 10 Laser Ruler
11.85
11.9
11.95
12
12.05
Chart Title
Series1
Measurement is imprecise
Measurement is imprecise
Ruler 1 Ruler 2 Ruler 3 Ruler 4 Ruler 5 Ruler 6 Ruler 7 Ruler 8 Ruler 9 Ruler 10 Laser Ruler
11.85
11.9
11.95
12
12.05
Chart Title
Series1
Ruler 1 Ruler 2 Ruler 3 Ruler 4 Ruler 5 Ruler 6 Ruler 7 Ruler 8 Ruler 9 Ruler 10 Laser Ruler
11.85
11.9
11.95
12
12.05
Chart Title
Series1
Measurement is imprecise
Ruler 1 Ruler 2 Ruler 3 Ruler 4 Ruler 5 Ruler 6 Ruler 7 Ruler 8 Ruler 9 Ruler 10 Laser Ruler
11.85
11.9
11.95
12
12.05
Chart Title
Series1
Measurement is imprecise
Surveys are especially vulnerableWhat
“construct” are you measuring?
Survey and question design
and content
Responses
Surveys are especially vulnerableWhat
“construct” are you measuring?
Survey and question design
and content
Responses
Is it really measuring it?Does it measure it the same each time?
Surveys are especially vulnerableWhat
“construct” are you measuring?
Survey and question design
and content
Responses
Is it really measuring it?Does it measure it the same each time?
Any errors in the questions or scales?Any bias in the questions asked or not asked?
Surveys are especially vulnerableWhat
“construct” are you measuring?
Survey and question design
and content
Responses
Is it really measuring it?Does it measure it the same each time?
Any errors in the questions or scales?Any bias in the questions asked or not asked?
Response set– -Are they representative?-How complete are they?-Is there non-response bias?
Surveys are especially vulnerableWhat
“construct” are you measuring?
Survey and question design
and content
Responses
Is it really measuring it?Does it measure it the same each time?
Any errors in the questions or scales?Any bias in the questions asked or not asked?
Response set– -Are they representative?-How complete are they?-Is there non-response bias?
Over 80 cognitive and
memory biases
Respondents prone to error• Response is a cognitive act. 4 Stages:
• Comprehension• Retrieval • Judgment• Response
All kinds of errors can happen in these stages
Respondents prone to errorCognitive Stages Definition Vulnerabilities
Stage 1 Comprehension Understanding the question as author intended
Unknown or misused words, ambiguity, complexity, length
Stage 2 Retrieval Search of memory for relevant information
Memory bias, recall error, fatigue
Stage 3 JudgmentConsiders information
retrieved, makes “guestimations” and decides
Social or political bias, “fuzzy logic,” error in “guesstimation”,
personal sensitivity
Stage 4 Response Provides the information Human error, incomplete response, wrong format
Respondents prone to error• Change bias – Remembers as more difficult• Context effect – out of context more difficult• Consistency bias – thinking past attitudes same
as current• Anchoring – relying too heavily on one piece of
info in order to make an estimation• Availability heuristic – overestimation due to
recency or emotional strength of memory
Respondents prone to error• Confirmation bias – tendency to search for and
remember info confirming one’s preconceptions• Framing effect – drawing different conclusions
or interpretations base on how info is presented• Optimism bias – tendency to overestimate
likelihood of positive outcomes• Social response bias – tendency to underreport
socially undesirable behaviors and vice/versa.
Lots more!!!!
Proof is in the Polls• Ever seen this?
• Down in the polls by 6 points• Down in the polls by 3 points• Ahead in the polls by 1 point• Landslide victory with 60% of the votes!
Keep these limitations in mind when drawing inferences from survey data.
Research, Assessment, & Program Evaluation
Research• Systematic collection of data• Generalize to larger populations• Based on Research question• Rigorous methodology• Used to test hypotheses• Contributes to the development
of theory and models
Assessment• Systematic collection of data• Generalize to larger population• Based on Assessment question• Borrows rigor when practical• Used to evaluate effectiveness• Contributes to judgments of the
quality of programs or activities
Program Evaluation:• Act of using data (assessment, operational,
etc.) related to outcomes, inputs, and processes of a program
• To assign judgment on its effectiveness
Research, Assessment, & Program Evaluation
Assessment (and statistics) is about describing:
• Things• Stuff pertaining to those things• The relationship between things & stuff
Foundational Concepts“Things and Stuff”
What is a “thing”?• An entity of some kind• An idea• A quality or characteristic
Foundational Concepts“Things and Stuff”
What is “stuff”?• Material out of which something is made• Essential substance or elements• Essence
Foundational Concepts“Things and Stuff”
Things are the objects we assess, and
Stuff characterizes those things
Foundational Concepts“Things and Stuff”
Let’s use more concrete terms:
• Entity – something that exists separate from its parts
• Property – a characteristic, trait or attribute and other stuff that describes something
Foundational Concepts“Things and Stuff”
Types of entities:• Organisms (including humans, trees, etc.)• Physical objects (rocks, buildings, etc.)• Actions and events (running, stroke, etc.)• Cognitive phenomena (ideas, emotions, etc.)• Organizations (governments, boards, etc.)• Scientific entities (waves, motivation, etc.)• Math entities (numbers, functions, vectors, etc.)
Foundational Concepts“Things and Stuff”
Different Properties:– Height– Age– Perceptions– Weight– Density– GPA– Level of engagement– Magnitude– Levels and expressions of anger, fear, anxiety
Foundational Concepts“Things and Stuff”
Notice that some properties could be examined as entities themselves– Relaxation– Motivation– Intelligence– Satisfaction– Health
Foundational Concepts“Things and Stuff”
Entities and properties– Humans
• Height, satisfaction, health, perceptions, emotions, beliefs, etc.
– Students (whether human or not )
• Age, GPA, knowledge, skills, engagement, class standing, etc.
– Physical Objects• Weight, chemical composition, density, etc.
– Forces• Magnitude, direction, etc.
Foundational Concepts“Things and Stuff”
Most critical thing about properties -
they vary– Height varies– Age varies– Perceptions vary– Weight varies– Density varies– GPA varies– Level of engagement varies– Levels and expressions of anger, fear, anxiety vary
Foundational Concepts“Things and Stuff”
These are the first two foundational concepts:• Entity• PropertyThe others are:• Variable• Prediction and Control• Relationship• Statistical techniques
Foundational Concepts“Things and Stuff”
Properties are represented by variables - – Height is represented by inches, meters, or relational terms (bigger)– Age is represented by months, days, years or relational terms (older)– Weight is represented by kilograms, pounds or relational terms– Density is represented by pound per square inch, etc.– Grades can be represented by symbols (A+), numbers (92), etc.– Engagement can be represented by number of activities attended,
etc.
Foundational Concepts“Things and Stuff”
Some variables are straightforward-
– Height is often measured in Inches– Weight is often measured in Kilograms– Age is often measured in years
Foundational Concepts“Things and Stuff”
Some variables are not-
– Satisfaction is often measured in… ???– Anger is often measured in… ???– Engagement is often measured in… ???
Foundational Concepts“Things and Stuff”
That describe an entity to some degree
That may be precise or not
Foundational Concepts“Things and Stuff”
Broad goal of assessment/research: To understand the population
Two underlying fundamental goals:– Prediction and control of variables– Understanding of entities & their properties
Which is more important?
Foundational ConceptsPrediction and Control
Let’s pretend we are looking at student performance of a complex task:
Let’s look at these statements:
– Haste makes waste– Practice makes perfect
How would we assess these?
Foundational ConceptsPrediction and Control
What are our variables?
– What variables might represent haste?– What variables might represent waste?– What variables might represent practice?– What variables might represent perfection?
What is our model?
Foundational ConceptsPrediction and Control
What is our model?
– More haste = more waste?• Some of the waste is the result of haste?• Haste predicts waste?
– Simply looking at means, which is what we often do:• Those that were hastier answered fewer questions
correctly than those who took more time.
Foundational ConceptsPrediction and Control
Waste
Haste
• The red area represents the degree to which Waste is due to Haste.• The green area is the degree to which Waste is influenced by other factors
Foundational ConceptsPrediction and Control
What is our model?
– More practice = better performance?• Some of the performance is the result of practice?• Practice predicts performance?
– Simply looking at means, which is what we often do:• Those who practiced more scored higher than those who
practiced less.
Foundational ConceptsPrediction and Control
Performance
• What is the red area?• What is the green area?
Practice
Foundational ConceptsPrediction and Control
Is it really that simple?
– What if hasty practice results in worse performance?
– How do we know the true impact of practice, if we don’t remove the impact of haste?
Foundational ConceptsPrediction and Control
So in order to accurately understand entities and properties, we must be able to predict and control.
– Prediction and control are more important – accurate understanding depends on it
– We must understand relationships to do so
Foundational ConceptsPrediction and Control
Back to those statements
– Haste makes waste– Practice makes perfect
Both are engrained in our culture and both are about relationships
Establishing relationships between objects is how we naturally understand the world
Foundational ConceptsPrediction and Control
Let’s operationalize this?
– Haste = time on task (in minutes)– Practice = # of repetitions– Performance = Evaluation score
• (waste = lost points or distance from a perfect score)
Foundational ConceptsPrediction and Control
Our Model
• Performance = Practice + Haste
Or more specifically…• Evaluation Score = # Repetitions + Minutes on Task
Foundational ConceptsPrediction and Control
Each variable contributes to the variance in evaluation scores
scores
# repetitions
minutes on task
Foundational ConceptsPrediction and Control
scores extent to which observed scores vary across all individuals
We need to control for the impact of Haste on Performance in order to understand the impact of Practice.
Foundational ConceptsPrediction and Control
Foundational ConceptsPrediction and Control
scores
# repetitions
extent to which # of repetitions varies
across all individuals
extent to which minutes on task varies
across all individuals
Foundational ConceptsPrediction and Control
scores
# repetitions
minutes on task
Foundational ConceptsPrediction and Control
scores
# repetitions
minutes on task
variance in scores that we can’t explain (individual
differences, fatigue, human error, etc.)
Foundational ConceptsPrediction and Control
scores
# repetitions
minutes on task
Variance attributed to the # of repetitions
Foundational ConceptsPrediction and Control
scores
# repetitions
minutes on task
This is the degree to which “Practice Makes Perfect”
scores
# repetitions
minutes on task
Foundational ConceptsPrediction and Control
This is the variance explained by
minutes on task
Foundational ConceptsPrediction and Control
scores
# repetitions
minutes on task
In other words, the degree to which
“Haste makes waste”
Foundational ConceptsPrediction and Control
scores
# repetitions
minutes on task
SHARED variance between minutes on task
and # repetitions
Foundational ConceptsPrediction and Control
scores
# repetitions
minutes on task
variance in Scores attributed to the combination of
minutes on task AND # repetitions
Foundational ConceptsPrediction and Control
scores
# repetitions
minutes on task
variance in scores uniquely attributed
to # repetitions
Foundational ConceptsPrediction and Control
scores
# repetitions
minutes on task
Put another way: variance in scores
attributed to # repetitions AFTER
variance attributed to
minutes on task is controlled
If we did not Control for Haste, we would think all of this was
the result of PracticePerformance
Practice
Foundational ConceptsPrediction and Control
So…• Recognizing the relationships between
variables, and • controlling for them • is critical to understanding the impact
of any one of them.
Foundational ConceptsPrediction and Control
Why is this important in assessment?• Consider you are studying the impact of stress on student
persistence?• You ask students to describe their stress level at one point in
time, and you see that:– 80% of those who reported high stress dropped out, while only– 25% of those who reported low stress did.
• You devote 70% of a staff person’s time to doing stress relief programming, totaling appx. $60,000
Foundational ConceptsPrediction and Control
What variables are there in stress and persistence?
Stress• Family support and relationships• Peer support and relationships• Coping skills• Academic work load• Financial Support• Employment work load• Work environment
Persistence• Academic advising• Social connectedness• Engagement• Faculty / Staff connections• Institutional Bureaucracy• Meaningful learning experiences• Support Services
Foundational ConceptsPrediction and Control
Stress• Family support and relationships• Peer support and relationships• Coping skills• Academic work load• Financial Support• Employment work load• Work environment
Persistence• Academic advising• Social connectedness• Engagement• Faculty / Staff connections• Institutional Bureaucracy• Meaningful learning experiences• Support Services
• What should the staff member focus their efforts on? • Is it possible that general Stress Relief may not be a very influential variable? • Would the $60k be well spent?
Foundational ConceptsPrediction and Control
When we conduct quantitative or qualitative analyses, we are constructing models
• That model can be very, very simple• Mean – this is how the average person did…• Mode – The most frequent score was X• Standard Deviation – roughly 68% scored between these two
points• Range – All students scored between X and Y.
• Descriptive stats = the weakest and least informative models
Foundational ConceptsPrediction and Control
When we conduct quantitative or qualitative analyses, we are constructing models
• Other models can be complex– For every 1 repetition of Practice, when Haste is controlled, Evaluation
Scores increased by 2.4 points.– For every hour engaged in co-curricular activities, when controlling for HS
GPA, Class standing, and Major, retention likelihood increases by .45%
– More complex stats = more informative and more accurate models
Foundational ConceptsPrediction and Control
So when we assess, we need to Think of our Model
• What are the entities and properties we’re examining?• What are the variables involved? Operationalize them.• What is their relationship; how do they interact?• How simple or complex/accurate a model do I need?• How can we predict and control for these variables?
Foundational ConceptsPrediction and Control
Top Related