A Mixed-Method Evaluation of a Series of Strategies Designed to Improve Student Success in...
-
Upload
jonah-maxwell -
Category
Documents
-
view
215 -
download
2
Transcript of A Mixed-Method Evaluation of a Series of Strategies Designed to Improve Student Success in...
A Mixed-Method Evaluation of a Series of Strategies Designed to Improve Student Success in Mathematics
and English Developmental Education Courses at Essex County College
Dr. Leigh Bello de Castro
Essex County College
Dr. Charyl Yarbrough
The Heldrich Center for Workforce Development at Rutgers
The Center for Academic Foundations
ECC Incoming Freshman• Accuplacer and Companion Tests• Results
• 85% - Math 086• 10% - Math 092• 5% - Math 100 or higher
ECC Center for Academic FoundationsObjectives
• Established in 2007• Target Group: students scoring between
20 -29 Accuplacer• 25% pass rate in Math 086 (Target Group)• Develop Learning Communities for students in need of
Math and English developmental/transitional courses. • Provide an array of support services.
• Counseling• Early Intervention• Career Counseling• Intensive Tutoring
• Implement new teaching strategies into the curriculum.
ECC Developmental Sequence
Mth 100
Mth 092
AFM 083 Mth 086
Eng 101
Eng 092
AFE 083 Eng 085
*** AFM and AFE Divisional Midterms and Finals
ECC Center for Academic Foundations Staff• Full time Coordinator• Full time Teacher Advisor Math• Full time Teacher Advisor English• Adjunct Math and English Faculty• Supplemental Instruction Leaders• Resource Specialists• Math and English Faculty Advisors
ECC Center for Academic Foundations
• Implement new teaching strategies into the curriculum.• Customized Book• ALEKS• My Skills Lab• Early Intervention• Supplemental Instructional Leaders
• Block Scheduling• Math, English and Computer Course (CSS 101)• Tutoring with SI Leader after class• 2 class periods of lecture, 1 class period in Lab with ALEKS.
• Stand Alone Math Sections
ESSEX COUNTY COLLEGEINSTITUTIONAL RESEARCH
BASIC SKILL TRACKING - CAF and RegularSpring 2009, Spring 2010 and Fall 2010 Cohorts
Criteria
Cohort Original Cohort Return Following Sem. Success* Rate in Percent W Return Following Sem. Success* Rate in Percent W
Size/ N Who took MTH 092 MTH 092 MTH 092 took ENG 096 ENG 096 ENG 096
Passed Course
FALL 2010 N % N % N % N % %
original/ passed original/ passed original/ passed
AFM 083 609/ 335 270 44.3%/ 80.6% 104 38.5% 25.9%
MTH 086 2202/ 1146 896 40.6%/ 78.2% 417 46.5% 19.0%
AFE 083 444/ 278 233 52.5%/ 83.8% 156 67.0% 11.6%
ENG 085 957/ 708 578 60.4%/ 81.6% 395 68.3% 11.1%
SPRING 2010
AFM 083 470/ 220 99 21.1%/ 45.0% 12 12.1% 41.4%
MTH 086 1528/ 670 348 22.8%/ 51.9% 120 34.5% 24.1%
AFE 083 305/ 159 73 23.9%/ 45.9% 34 46.6% 23.3%
ENG 085 426/ 237 92 25.6%/ 38.8% 54 58.7% 9.8%
SPRING 2009
AFM 083 302/ 91 42 13.9%/ 42.6% 9 21.4% 19.0%
MTH 086 1400/ 577 278 19.9%/ 48.1% 104 37.4% 15.5%
AFE 083 183/ 112 42 23%/ 37.5% 19 45.2% 23.8%
ENG 085 321/ 200 73 22.7%/ 36.5% 45 61.6% 11.0%
*Success rate = A,B+,B,C+,C/All Grades The numbers in RED refer to the number or percent of students who successfully completed their first remedial course and would be allowed to take the next level course.For Example: For Fall 2010 AFM 083 - originally there were 609 students registered for this course. Of these, 335 passed the course and would qualify to takeThe next level mathematics course. In fact, 270, or 44.3 % of the original cohort took MTH 096. This increases to 80.6% of those qualified to take MTH 092.
Of the 270 who took MTH 092, 104 or 38.5% successfully completed MTH 092 and 25.9% withdrew.
Center for Academic Foundations Math 092
• Students were not succeeding in next course• Spring 2011• PBI Funds for CAF 09 level
• Block scheduling• Intrusive Support • Resource Specialist• Mandatory Tutoring• Supplemental Instructional Leaders
• MTH 092 and ENG 096 • Curriculum• Books• Instructional Software
• Implemented the original CAF model• Additional Staffing• Evaluation
Purpose of Evaluation• Describe ECC’s PBI Formula Grant Activities
• Generate actionable information on the implementation of the PBI Formula Grant initiatives
• Assess the overall effectiveness of ECC’s PBI Formula Grant initiatives in achieving positive outcomes
Key Research Questions
1. Do the ECC project services have the intended positive effects on student outcomes (e.g., higher percentage of students passing mathematics and English courses)?
2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses?
3. In what ways can the services be modified to maximize their effectiveness?
Evaluation Design • Consisted of formative and summative components
• Utilized qualitative and quantitative methodologies
• Evaluators from multiple academic disciplines participated in the evaluation.
• Submit interim reports
• Focused on understanding the culture and politics of the school as part of the evaluation process
• Considered the views of all stakeholders
Qualitative Data Collection(Spring 2011)
Method Time Frame Number of Participants/Sessions
Course Observations 2-3 times per course per semester
sixteen observations
Instructor Interviews Pre-Post Semester Five interviews
Student Focus Groups
Post Semester Four groups consisting of 5-8 students
Self-Efficacy and Subject Matter Confidence Surveys
Pre-Post Semester N = 42-67
Competing Priorities and Barriers Scales
Pre Semester
Quantitative Data Collection
Administrative Data
• The Heldrich Center collects class level data from the program administrator.
Program Description
Profile of Students• 121 students in five developmental math sections
• 109 students in five developmental English sections
• Most students were comfortable with use of computers at baseline
• 67% of students worked 20 hours per week or less
• Moderate levels of confidence at the beginning of the semester
RQ 1. Do the ECC project services have the intended positive effects on student outcomes?
Success rate is the number of students who passed the course divided by the number of students enrolled.
Spring 2011
Math092 ENG0960.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
Average College Wide Success RateCAF Program Success Rate GoalCAF Overall Success rate (withdrawals included)
RQ 1. Do the ECC project services have the intended positive effects on student outcomes?
Table 3. MTH-090 Grade Distribution and Success and Withdrawal Rates, by Section
Section
Total Enrolle
d Withdrawals
Total Completing
Course
Success Rate Withdrawals
IncludedWithdrawal
RateAF1 30 0 30 96.7% 0.0%AF2 26 4 22 53.8% 15.4%AF4 21 3 18 76.2% 14.3%AF5 29 4 25 75.9% 13.8%AF6 15 5 10 13.3% 33.3%
Table 4. ENG-090 Grade Distribution and Success and Withdrawal Rates, by Section
Section
Total Enrolle
d Withdrawals
Total Completing
Course
Success Rate Withdrawals
IncludedWithdrawal
RateAF1 24 1 23 87.5% 4.2%AF2 25 8 17 48.0% 32.0%AF4 23 2 21 73.9% 8.7%AF5 21 0 21 81.0% 0.0%AF6 16 2 14 62.5% 12.5%
Spring 2011The charts below represent the grade distribution for CAF and the Division of Mathematics and Physics. Data reflects a 68% success rate for those students taking the Math course in comparison to their counterparts taking the traditional MTH 092 whose success rate is 40%. In the ENG 096, the CAF courses had 69% success rate as compared to 60% in the traditional ENG 096 courses.
MTH 092Sect. Total A's B+ B C+ C D F I W A - C A - F All Pass* SuccessAF 1 30 4 7 2 5 11 0 1 0 0 29 30 30 97% 97%AF 2 26 4 1 3 4 2 0 8 0 4 14 22 26 64% 54%AF 4 22 2 4 4 0 6 0 2 0 4 16 18 22 89% 73%AF 5 29 6 5 5 1 5 0 3 0 4 22 25 29 88% 76%AF 6 15 0 0 1 0 1 0 8 0 5 2 10 15 20% 13%
TOTAL 122 16 17 15 10 25 0 22 0 17 83 105 122 79% 68%
ENG 096
AF 1 25 3 2 5 4 7 0 2 0 2 21 23 25 91% 84%AF 2 26 0 2 2 4 4 0 5 0 9 12 17 26 71% 46%AF 4 23 1 3 3 0 10 0 4 0 2 17 21 23 81% 74%AF 5 21 2 4 4 1 6 0 4 0 0 17 21 21 81% 81%AF 6 16 0 7 2 0 1 0 4 0 2 10 14 16 71% 63%
TOTAL 111 6 18 16 9 28 0 19 0 15 77 96 111 80% 69%
ALLMTH 092 1944 170 129 140 136 232 80 634 16 407 807 1521 1944 53% 42%
CAF 122 16 17 15 10 25 0 22 0 17 83 105 122 79% 68%less CAF 1822 154 112 125 126 207 80 612 16 390 724 1416 1822 51% 40%
ALLENG 096 1504 168 152 212 141 244 15 363 18 191 917 1295 1504 71% 61%
CAF 111 6 18 16 9 28 0 19 0 15 77 96 111 80% 69%less CAF 1393 162 134 196 132 216 15 344 18 176 840 1199 1393 70% 60%
* Pass rate = A,B+,B,C+,C/A,B+, B, C+, C, D, and F* Success Rate = A, B+, B, C+, C/ all grades including W and I.
RQ 2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses?
• Scheduling
The CAF program successfully grouped the components of the courses to support block scheduling.
• Each CAF course required the same group of students to meet in three different session types (i.e., classroom, tutorial, and lab).
• Students in CAF Math were not necessarily in CAF English with the same group of students
RQ2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses?
Supplemental Instructors
• Typically advanced ECC students or recent graduates attending four year college
• Varied in experience and skill
• Established positive rapport with students
• Administrators matched inexperienced supplemental instructors with lead teachers
RQ2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses?
Inclusion of technology:
• Students and instructors reported that the math lectures, computer programs, and tutorials aligned rather seamlessly
in the MTH-100 course.
• Students interpreted the grammatical issues presented by the web-based tutorial program and the supplemental instructors (in tutorial sessions) as distinct and discrete topics, unrelated to writing and reading comprehension topics presented by the course instructor.
• Students were more motivated to complete computer-aided assignments in math than in English
RQ 2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses?
• Program Culture
• The CAF program built a trusting environment in which students felt comfortable seeking help from instructors and supplemental instructors.
Table 8. CAF ENG-096 Student Survey Responses Related to General Perceptions of School Support
Very Little Some Quite a Bit Very MuchEncouraging you to spend significant amounts of time studying
3.2% 12.7% 46.0% 38.1%
Providing the support you need to help you succeed at this college
3.2% 11.1% 38.1% 47.6%
Encouraging contact among students from different economic, social, and racial or ethnic backgrounds
3.2% 25.8% 25.8% 45.2%
Helping you cope with your non-academic responsibilities (work, family, etc.)
6.5% 21.0% 33.9% 38.7%
Providing the support you need to thrive socially
3.2% 11.3% 37.1% 48.4%
Providing the financial support you need to afford your education
4.8% 11.1% 33.3% 50.8%
Using computers in academic work 4.8% 11.3% 24.2% 59.7%
English - How much does this college emphasize each of the following? Please select the most appropriate answer.
Table 9. CAF MTH-092 Student Survey Responses Related to General Perceptions of School Support
Very Little Some Quite a Bit Very MuchEncouraging you to spend significant amounts of time studying
4.1% 16.3% 34.7% 44.9%
Providing the support you need to help you succeed at this college
2.1% 16.7% 37.5% 43.8%
Encouraging contact among students from different economic, social, and racial or ethnic backgrounds
6.1% 14.3% 38.8% 40.8%
Helping you cope with your non-academic responsibilities (work, family, etc.)
8.2% 16.3% 36.7% 38.8%
Providing the support you need to thrive socially
4.2% 18.8% 35.4% 41.7%
Providing the financial support you need to afford your education
0.0% 18.4% 30.6% 51.0%
Using computers in academic work 2.0% 16.3% 32.7% 49.0%
Math - How much does this college emphasize each of the following? Please select the most appropriate answer.
RQ 2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses?
Program Culture
• The students did not develop the sense of community that learning communities are designed to foster and that have been shown to have positive effects on student achievement.
• CAF instructors did not provide substantive guidance to students related to collaborative learning.
Table 10. CAF Math and English Pre-Post Confidence and Self-Efficacy Mean
Pre-test sample size
Pre-test mean
Post-test sample size
Post-test mean
Math Confidence 67 3.18 46 3.41English Confidence 71 3.51 59 3.55
Math Self-Efficacy 75 7.58 55 7.74English Self-Efficacy 63 7.76 42 7.7
RQ 2. Which of the project services have worked well and which have not been as effective in helping students perform better in subject courses?
General Teaching Approach
• Across math and English courses, the instructors implemented procedural teaching techniques.
• The teaching methods did not encourage students to engage proactively in the “process” of learning.
RQ3. In what ways can the services be modified to maximize their effectiveness?
• Administrators should provide professional development training in “scaffolding”
• Instructors should emphasize analysis and critical thinking
• Instructors should ensure that all components of a course are clearly connected
• Instructors should lead activities that foster interaction and collaboration between students
• Ensure that instructors are punctual and present at each class session
Fall 2011 – Success Rate Update
Success rate is the number of students who passed the course divided by the number of students enrolled.
Math092 ENG0960.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
90.0%
Average College Wide Success RateCAF Program Success Rate GoalCAF Overall Success rate Spring 2011 (with-drawals included)CAF Overall Success rate Fall 2011 (withdrawals included)
Fall 2011 Comparison Data
course NO_A NO_B+ NO_B NO_C+ NO_C NO_D NO_F NO_INO_W A-C A-F A-W Previous Enrolled Success Rate
AFE083 45 43 85 46 96 5 109 2 74 315 429 505 431 505 62.4%ENG085 114 103 202 108 132 1 230 2 58 659 890 950 950 957 69.4%AFM083 105 54 82 39 95 13 148 33 118 375 536 687 603 694 54.6%MTH086 250 136 213 122 217 57 701 19 342 938 1696 2057 2057 2164 45.6%ENG096AF 20 27 17 17 19 0 21 0 17 100 121 138 138 138 72.5%ENG096- 155 123 194 134 189 9 323 4 168 795 1127 1299 1299 1307 61.2%MTH092AF 18 4 12 8 15 1 31 2 24 57 89 115 115 115 49.6%MTH092- 106 48 98 71 134 67 504 19 343 457 1028 1390 1390 1403 32.9%
Spring 2012 – Evaluation UpdateMethod Time Frame
Administrative Data – Success Rates and Withdrawal Rates
Post Semester
Course Observations Pre-Post Semester
Instructor Interviews Pre-Post Semester
Supplemental Instructor Interviews Post Semester
Student Focus Groups Post Semester
Self-Efficacy and Subject Matter Confidence Surveys
Pre-Post Semester
Competing Priorities and Barriers Scales
Pre Semester