Using Action Research to Ensure Relevance and Excellence
description
Transcript of Using Action Research to Ensure Relevance and Excellence
Using Action Research to Ensure Relevance and Excellence
Dan Friedman, Ph.D.Director, University 101 Programs
University of South Carolina
Student Success Symposium
Does it work?
Academic achievement
– GPA and hours earned
Retention persistence to second year
Utilization of campus resourcesPaul Fidler
A movement created
87% of institutions with a FYS
Padgett & Keup (2011). 2009 National Survey of First-Year Seminars.
High-Impact Educational Practices (AACU)
My Focus for Today
Using assessment/action research to demonstrate the value of our programs, and to continually improve what we do by understanding why our programs work and for whom.
FAITH-BASED?
“Estimates of college quality are essentially "faith-based," insofar as we have little direct evidence of how any given school contributes to students' learning.” RICHARD HERSCH (2005). What does college teach?
ATLANTIC MONTHLY.
Assessment Cycle
3) Interpret Evidence
4) Implement Change
2) Gather Evidence
1) Identify Outcomes
Maki, P. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus Publishing.
Easy Stuff!
Friedman, D. (2012). Assessing the first-year seminar.
Two Types of Assessment
1) Summative – used to make a judgment about the efficacy of a program
2) Formative – used to provide feedback in order to foster improvement.
The Prescription
Relevance (doing the right things)
Excellence (doing things right)
EEnvironments
(College)
I O
Astin’s Value-Added
I – E – O Model
Astin, A. (1991)
“outputs must always be evaluated in terms of inputs”
Outcomes Inputs (Students)
EEnvironments
I O Inputs Outcomes
Common Mistakes
Just looking at inputs
15
EEnvironments
I O Inputs Outcomes
Common Mistakes
Just looking at environment
16
EEnvironments
I O Inputs Outcomes
Common Mistakes
Just looking at outcomes
17
EEnvironments
I O Inputs Outcomes
Common Mistakes
E-O Only (No Control for Inputs)
18
EEnvironments
(College)
I O
Astin’s Value-Added
I – E – O Model
Astin, A. (1991)
“outputs must always be evaluated in terms of inputs”
Outcomes Inputs (Students)
Disaggregating the Inputs
What does this tell us?
2002 2003 2004 2005 2006 2007 2008 2009 2010 201180
81
82
83
84
85
86
87
88
89
90
84.585.2
83.8
86.2
87.488.1
87.6
86.3 86.3
88
83.5
84.3
81.3
84.6
85.4
84.384.6
85.1
86.8
84.4
1-Year Retention
U101Non-U101
Year
Perc
enta
ge
What does this tell us?
PGPA Quintile UNIV 101 (n) Non-101 (n) P-Value
5 (High) 93.8% (n=497) 94.6% (n=297) NS
4 90.5% (n=652) 84.4% (n=154) .02
3 88.3% (n=656) 86.5 % (n=163) NS
2 87.4% (n=657) 79.1% (n=148) .008
1 (Low) 83.1% (n=628) 70.8% (n=182) .001
All 88.0% (n=3483) 84.4% (n=1086) .002
Positive Impact on Graduation
U101 (all) Non-101 (all) U101 (lowest ability)
Non-101 (lowest ability)
25%30%35%40%45%50%55%60%65%70%75% 71.1%
64.9%
56.9%
36.9%
5-Year Graduation Rates(2007 Cohort)
Need to Disaggregate
Disaggregate data by input variables Predicted GPA
SAT/ACT High School grades
Race/Ethnicity Family Income (Pell eligible) First Generation
Outcome(O)
College(E)
Student
(I)
High ImpactPractic
e
How to do it
WELL
Disaggregating Environmental Variables
Which factors predict persistence?
Used FYI data set & included variables from student data file (persistence and GPA) 2,014 responses (72% response rate)
A series of logistic regressions were conducted Controlled for gender, race, and high school grades
A standard deviation increase in Sense of Belonging & Acceptance increased the odds of persisting into the second year by 38% (p < .001), holding all other variables constant.
Assessing Educational Methods
Compare methods to determine if one approach is better than another
Continual Improvement
Identifying and replicating best practices
Structural Variables
FACTCRA
P
Fact or Crap?
Student Affairs professionals had higher ratings on overall course effectiveness than other instructors.
No statistically significant differences were found on any of the fifteen FYI factors or course evaluation factors for Division of Student Affairs employees versus non-division employees.
CRAP
Fact or Crap?
Sections that met 3 days a week (MWF) had significantly higher course effectiveness ratings than sections that met twice a week.
CRAP
Mean (n)
p
2 Days Per Week
4.14 (2153)
.286
3 Days Per Week
4.19 (668)
Overall Course Satisfaction by Days Per WeekData from fall 2011 Course Evaluations
Fact or Crap?
Sections with a Peer Leader had significantly higher course effectiveness ratings than sections without a Peer Leader.
FACT
Overall Course Satisfaction by Peer Leader StatusData from fall 2011 Course Evaluations
Mean (n)
p
Peer Leader 4.22 (1585)
.004
No Peer Leader 4.05 (530)
“Can’t fatten pig without weighing it”
Need to use assessment data to drive continual improvement
Contact Information
University 101 Programs 1728 College Street
Columbia, South Carolina 29208(803) 777-6029www.sc.edu/univ101