Measuring the Impact of Learning Outcomes
description
Transcript of Measuring the Impact of Learning Outcomes
Keith Wurtz, Senior Research Analyst, Chaffey College
Giovanni Sosa, Research Analyst, Chaffey College
Jim Fillpot, Director of Institutional Research, Chaffey College
An Integrated Model for Measuring the Impact of Course, Program, and Institutional Learning Outcomes
Strengthening Student Success ConferenceOctober 9th, 2009
Measuring the Impact of Learning Outcomes
“Good evidence, then, is obviously related to the questions the college has investigated and it can be replicated, making it reliable. Good evidence is representative of what is, not just an isolated case, and it is information upon which an institution can take action to improve. It is, in short, relevant, verifiable, representative, and actionable.”
Guide to Evaluating InstitutionsACCJC, August 2008
Measuring the Impact of Learning Outcomes
How do we collectively examine disparate learning outcomes assessment efforts? (representativeness and replicability)
How do we provide evidence about these disparate efforts? (verifiable)
How can we use evidence to improve upon learning outcomes practices? Upon district practices? (actionable)
Measuring the Impact of Learning Outcomes
Assessment often stands alone; relationships are often not explored between:
Courses From Different Programs Diverse Programs (within Instruction and Between Instruction and Student Services) Course/Program Learning Outcomes and Their Contribution to Institutional Learning Outcomes Types of Learning Outcomes (SLOs & AUOs) Differing Methods of Assessment Learning Outcomes and College Goals
Measuring the Impact of Learning Outcomes: Statistical Analysis
How do we statistically assess impact? Tests of statistical significance are not
necessary Greatly influenced by sample size (mean =
10.0; 10.1) Do not speak to the magnitude of the
difference Not well understood – even by ‘experts’
Effect Size as measure of practical significance Unstandardized Standardized (d, r)
Cohen’s conventions: d = .20 – small; .50 – moderate; .80 – large
Statistical Significance N = 30
Question Item Statistically
Significant? Q1 - FAFSA Deadline No Q2 - Online FAFSA Application No Q3 - Pin Number No Q4 - Application Window No Q5 - Verification Process No Q6 - Checking Award Status No Q7 - Receipt of Funds No Q8 - Enrollment Requirements No Q9 - Progress Requirements No Overall (Avg.) No
N = 30 Question Item Number Correct
Pre-Test Number Correct
Post-Test Statistically Significant?
Q1 19 24 No Q2 12 16 No Q3 26 30 No Q4 7 10 No Q5 13 21 No Q6 5 13 No Q7 10 15 No Q8 6 16 No Q9 3 15 No Overall (Avg.) 7.0 9.0 No
Effect Size (unstandardized)
N = 30
Question Item Number Correct Pre-Test
Number Correct Post-Test
Statistically Significant? Effect Size (d)
Q1 19 24 No .37 Q2 12 16 No .27 Q3 26 30 No .75 Q4 7 10 No .22 Q5 13 21 No .55 Q6 5 13 No .60 Q7 10 15 No .34 Q8 6 16 No .71 Q9 3 15 No .93 Overall (Avg.) 5.0 7.0 No .61
Effect Size (standardized)
Meta-Analysis
Combines research results across all identified studies Combines results using Effect Size (i.e.,
d) - generates an Average d
Average d may vary across study characteristics (i.e., Moderators): Number of Assessment Cycles Type of Assessment Instrument
Moderators(Grouping Variables)
Term – term data collection occurred Academic Year (AcadYr) – Academic year
(SU, FA, SP) data collection occurred Learning Outcome (LearnOutcome)
AUO – Administrative Unit Outcome SLO – Student Learning Outome
Researcher – person who conducted research
Program – department Course
Effect Size by Academic Year
Number of Studies 1 9 15 44 48
Effect Size by Learning Outcome Type (AUO or SLO)
Number of Studies 2 115
Effect Size by Program
Number of Studies
3 1 14 1 2 11 10 5 11 59
Effect Size by Course
Number of Studies 1 21 12 12 14 3 1 4 2 4 2
Moderators (Continued)(Grouping Variables - Measure)
Criterion – How was the Criteria for Success set by the program? Population – A measure obtained from the population (e.g.: critical thinking, Nursing Licensing
Exam, etc.) Pre-assessment score – The pre-assessment score in a pre-post assessment Program – The program set a specific criterion (e.g.: PE, Communication Studies, etc.)
PrePost – The assessment was a post-assessment or pre-post Standardized – Measure used to assess outcome was standardized or unstandardized Instrument
Rubric True/False Multiple choice Matching Likert Scale Anchored Scale
Unduplicated – used to identify number of students impacted by learning outcomes processes. From 2004-005 to 2008-2009 6,714 students have been impacted by learning outcomes.
Assessor Self-Assessment (e.g.: student self-assessed) External Evaluation (e.g.: student evaluated by teacher)
Effect Size by Criterion Type
Number of Studies 1 31 85
Effect Size by Pre-Post or Post-Assessment
Number of Studies 85 32
Effect Size by whether the Instrument was Standardized or Unstandardized
Number of Studies 10 107
Effect Size by Measurement Type
Number of Studies 7 22 2 8 75 3
Effect Size by whether Student Self-Assessed Learning or Learning was Assessed by External Evaluator (e.g.: teacher)
Number of Studies 82 35
Moderators (Continued)(Grouping Variables - Planning)
EndsPolicy1 (1) Instructional and Student Services, (2) Comprehensive Education
Program, (3) Collaborative Partnerships, (4) Continuous Improvement, (5) Learning Outcomes, (6) Core Competencies
EndsPolicy2 (1) Foundation Skills Courses, (2) Occupational Programs, (3) Transfer
Level Programs, (4) Outreach, (5) Student Services, (6) Library and Learning Support Services
EndsPolicy3 (1) College Facilities, (2) Technology, (3) Human Resources
EndsPolicy4 (1) Balanced Budget, (2) Achievement of Planned Enrollment Growth,
(3) External Funding, (4) Bond Reserve EndsPolicy5
(1 Qualified Personnel and Professional Development, (2) Commitment to Diversity, (3) Employment Agreement, (4) Professional Ethics
Effect Size by Ends Policy 1: Learning Centered College
Number of Studies 1 7 23 86
Effect Size by Ends Policy 2: Institutional Effectiveness
Number of Studies 1 92 22 1
Moderators (Continued)(Grouping Variables – Learning Outcomes)
KSA – Knowledge, Skill, or Ability (e.g.: critical thinking, speech, confidence, self-efficacy, etc.)
KSACateg = KSA Categorized by Core Competency and Instruction1. Communication – Students will demonstrate effective communication and
comprehension skills.2. Critical Thinking – Students will demonstrate critical thinking skills in problem
solving across the disciplines and in daily life3. Community/Global Awareness and Responsibility – Students will demonstrate
knowledge of significant social, cultural, environmental and aesthetic perspectives
4. Personal, Academic, and Care Development – Students will assess their own knowledge, skills and abilities; set personal, educational, and career goals; work independently and in group settings; identify lifestyle choices that promote self reliance, financial literacy and physical, mental and social health
5. Instruction – Students will demonstrate KSA specific to course subject (e.g.: theory identification, research methods, etc.)
Cycles – Number of cycles program has assessed outcome Level
Course Program Institution
Effect Size by Knowledge, Skill, or Ability
Number of Studies 65 9 12 20 11
Effect Size by Number of Times Program has Assessed Outcome
Number of Studies 74 34 8 1
Effect Size by Course and Program Level Learning Outcomes
Number of Studies 77 40
Limitations
Database only includes results from outcomes evaluated by the Office of Institutional Research
How do we connect information to student success (e.g.: goals, course success, transfer, etc.)?
Resources
Microsoft Excel Spreadsheets that calculate effect size statistics: http://www.stat-help.com/spreadsheets.html
Effect Size Calculator with Confidence Intervals: http://www.cemcentre.org/renderpage.asp?linkID=30325017