Adult obesity final presentation- shanna miller-computerapp-week11-team
Developmental Education Assessment, Placement, and Progression Thomas Bailey Based on Research by...
-
Upload
hayley-hogate -
Category
Documents
-
view
218 -
download
0
Transcript of Developmental Education Assessment, Placement, and Progression Thomas Bailey Based on Research by...
Developmental EducationAssessment, Placement, and Progression
Thomas Bailey
Based on Research by
Katherine Hughes, Shanna Jaggars, Judith Scott-Clayton
National Context
• For many (most?) entering CC students, assessment center is one of first places they will visit
• For the majority of students sitting for these exams, the result is placement into developmental education
• Yet research has not consistently found that this process actually improves student outcomes
CCRC Literature Review(Hughes & Scott-Clayton)
• Examined three questions:1. Is there consensus regarding the proper
purpose and role of assessment in CCs?2. Are the most commonly used assessments
valid for their intended purpose?3. Are there alternative models of assessment
that may improve outcomes for underprepared students?
• CUNY study brings new data to bear on similar set of questions
No Consensus on Meaning of College Ready
• Many assessments
• Many cut off scores
• Many policies with respect to – Mandatory Testing– Mandatory Placement
Figure 3: Educational Outcome by Math CPT Score and Estimated Discontinuity
0.2
.4.6
.8
-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Math Cutoff
Estimated Discontinuity = -0.014(0.012)
Passing First College-Level Course
0.2
.4
-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Math Cutoff
Estimated Discontinuity = -0.006(0.006)
2 yr Degree Completion
20
25
30
35
40
45
50
-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Math Cutoff
Estimated Discontinuity = 3.590(0.657)
Total Credits Earned
0.2
.4.6
.8
-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Math Cutoff
Estimated Discontinuity = 0.020(0.012)
Fall-to-Fall Retention
0.2
.4
-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Math Cutoff
Estimated Discontinuity = -0.001(0.006)
Transfer to 4 yr
20
25
30
35
40
45
50
-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Math Cutoff
Estimated Discontinuity = 0.233(0.649)
Total College-Level Credits Earned
Are Dev Ed Assessments Valid?
• CUNY uses COMPASS math & reading tests (published by ACT, Inc.; one of two most common assessments)
• There are lots of different ways to think about validity:– Construct validity: does the test measure what you think it does?– Predictive validity: does the test predict some measure of later success?– Argument-based approach to validity: “It is the interpretation of test
scores required by proposed uses that are evaluated, not the test itself” (Standards for Educational and Psychological Testing)
• Focus here is on predictive validity– This is a necessary, but not sufficient component of overall validity of
the test– “[U]ltimately, it is the responsibility of the users of a test to evaluate this
evidence to ensure the test is appropriate for the purpose(s) for which it is being used” (College Board, 2003, p. A-62)
– Broadest analysis of validity eventually requires a program evaluation: when students are assigned to some treatment on the basis of a score, do better outcomes result?
Predictive Validity Analysis
• Research questions:– How well do placement test scores predict “success”
in the relevant gatekeeper course?– How well do other measures (such as high school
performance) predict success, either instead of or in addition to placement test scores?
– How many students are “correctly placed” using current placement test cutoffs to divide students, versus assigning all students to the same level?
What is “Gatekeeper Success”?
• “Gatekeeper” course: first college-level course
• We look at three measures:– Completed course with B or higher– Completed course with C or higher– Passed course (D- or higher)
• These measures of success are all conditional upon actually enrolling in a gatekeeper course
Research Method Overview• Focus on first-time 2004-2007 entrants at two-year
colleges only, who have CAS and placement test data• First, estimate statistical relationships between
placement test scores (and/or other predictors) and gatekeeper success– Restrict sample to students who took gatekeeper without taking
developmental coursework (“estimation sample”)– Then, regress gatekeeper success on placement test scores
(and/or other predictors) to estimate relationships– Examine two summary measures: R-squareds and correlation
coefficients• Second, use logistic regression to predict which students
are likely to be “correctly placed” using different placement criteria
Methodological Concerns
• Restriction of range– R-squareds, correlations are measured only for those
who were placed directly into gatekeeper course– In general this tends to depress r-squareds and
correlations
• Extrapolation– For placement accuracy analysis, we must use
relationships estimated on about 25% of the data to predict likelihood of “success” for the other 75%
– So we must hope that the other 75% aren’t that different (not totally implausible)
Table 1 (R-squareds)
Proportion of Variation in Gatekeeper Outcomes That Can Be ExplainedBy Alternative Sets of Predictor Variables ("R-Squareds")
Test Scores, HSPlacement Test High School Subj. Test Scores and GPA/Units, LocalHS,
Scores Only GPA/Units Only HS GPA/Units YearsSinceHS
MathEarned B or higher in GK 0.121 0.094 0.160 0.184Earned C or higher in GK 0.069 0.070 0.104 0.122Passed GK (D- or higher) 0.040 0.054 0.070 0.080
EnglishEarned B or higher in GK 0.022 0.029 0.048 0.098Earned C or higher in GK 0.009 0.028 0.035 0.066Passed GK (D- or higher) 0.004 0.024 0.027 0.053
Table 1 (R-squareds)
Proportion of Variation in Gatekeeper Outcomes That Can Be ExplainedBy Alternative Sets of Predictor Variables ("R-Squareds")
Test Scores, HSPlacement Test High School Subj. Test Scores and GPA/Units, LocalHS,
Scores Only GPA/Units Only HS GPA/Units YearsSinceHS
MathEarned B or higher in GK 0.121 0.094 0.160 0.184Earned C or higher in GK 0.069 0.070 0.104 0.122Passed GK (D- or higher) 0.040 0.054 0.070 0.080
EnglishEarned B or higher in GK 0.022 0.029 0.048 0.098Earned C or higher in GK 0.009 0.028 0.035 0.066Passed GK (D- or higher) 0.004 0.024 0.027 0.053
Table 1 (Correlations)
Correlation Coefficients Between Gatekeeper OutcomesAnd Alternative Sets of Predictor Variables
Test Scores, HSPlacement Test High School Subj. Test Scores and GPA/Units, LocalHS,
Scores Only GPA/Units Only HS GPA/Units YearsSinceHS
MathEarned B or higher in GK 0.349 0.306 0.400 0.429Earned C or higher in GK 0.263 0.265 0.322 0.349Passed GK (D- or higher) 0.199 0.233 0.265 0.282
EnglishEarned B or higher in GK 0.148 0.169 0.218 0.314Earned C or higher in GK 0.093 0.167 0.188 0.257Passed GK (D- or higher) 0.063 0.154 0.165 0.229
Placement Accuracy Rates
• We know who will be placed in dev ed or gatekeeper based on test scores
• We can estimate whether or not given individual is predicted to succeed based on test scores
• Can then assign each person to one of four cells• Placement accuracy rate is sum of bottom left/upper right cells• Can also compare this to accuracy rates without using test at all
Predicted to Succeed in GK
Not Predicted to Succeed in GK
Placed in Dev Ed “False negative”
(Type II error)
Accurately Placed
Placed in GK Accurately Placed “False positive”
(Type I error)
Figure 1Probability of Gatekeeper Success,
By Math Part 2 Scores
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 91 93 95 97 99
Math Part 2 Score
Per
cen
t M
eeti
ng
Su
cces
s C
rite
rio
n
Earned B or Better
False positives
Accurately placed
Acc. placed
False neg.
Figure 1
Probability of Gatekeeper Success, By Math Part 2 Scores
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 91 93 95 97 99
Math Part 2 Score
Per
cen
t M
eeti
ng
Su
cces
s C
rite
rio
n
Passed GK Earned C or Better Earned B or Better
Figure 2 (By Writing Score)
Probability of Gatekeeper Success, By Writing Placement Scores
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2 3 4 5 6 7 8 9 10 11 12
Writing Placement Scores
Per
cen
t M
eeti
ng
Su
cces
s C
rite
rio
n
Passed GK Earned C or Better Earned B or Better
Table 2Predicted Placement Accuracy Rates Using Placement Test Scores,
Versus Placing All Students in College Level or Remedial
Accuracy Rate, Accuracy Rate, Accuracy Rate, Incremental IncrementalUsing Placement All Students In All Students In Validity vs. Validity vs.
Test Cutoffs Developmental College Level All Dev Ed All Coll. Lev
MathEarned B or higher in GK 0.701 0.691 0.309 0.010 0.391Earned C or higher in GK 0.585 0.496 0.504 0.089 0.081Passed GK (D- or higher) 0.487 0.348 0.652 0.140 -0.165
EnglishEarned B or higher in GK 0.650 0.650 0.350 0.000 0.301Earned C or higher in GK 0.463 0.379 0.621 0.085 -0.158Passed GK (D- or higher) 0.381 0.275 0.725 0.105 -0.344
Table 2
Predicted Placement Accuracy Rates Using Placement Test Scores,Versus Placing All Students in College Level or Remedial
Accuracy Rate, Accuracy Rate, Accuracy Rate, Incremental IncrementalUsing Placement All Students In All Students In Validity vs. Validity vs.
Test Cutoffs Developmental College Level All Dev Ed All Coll. Lev
MathEarned B or higher in GK 0.701 0.691 0.309 0.010 0.391Earned C or higher in GK 0.585 0.496 0.504 0.089 0.081Passed GK (D- or higher) 0.487 0.348 0.652 0.140 -0.165
EnglishEarned B or higher in GK 0.650 0.650 0.350 0.000 0.301Earned C or higher in GK 0.463 0.379 0.621 0.085 -0.158Passed GK (D- or higher) 0.381 0.275 0.725 0.105 -0.344
Table 2
Predicted Placement Accuracy Rates Using Placement Test Scores,Versus Placing All Students in College Level or Remedial
Accuracy Rate, Accuracy Rate, Accuracy Rate, Incremental IncrementalUsing Placement All Students In All Students In Validity vs. Validity vs.
Test Cutoffs Developmental College Level All Dev Ed All Coll. Lev
MathEarned B or higher in GK 0.701 0.691 0.309 0.010 0.391Earned C or higher in GK 0.585 0.496 0.504 0.089 0.081Passed GK (D- or higher) 0.487 0.348 0.652 0.140 -0.165
EnglishEarned B or higher in GK 0.650 0.650 0.350 0.000 0.301Earned C or higher in GK 0.463 0.379 0.621 0.085 -0.158Passed GK (D- or higher) 0.381 0.275 0.725 0.105 -0.344
Predicted Placement Accuracy Rates Using Placement Test Scores, Predicted Placement Accuracy Rates Using Placement Test Scores,Versus Placing All Students in College Level or Remedial, Versus Placing All Students in College Level or Remedial
Sample Restricted to Narrow Range Around Current Cutoffs
Accuracy Rate, Accuracy Rate, Accuracy Rate, Incremental IncrementalUsing Placement All Students In All Students In Validity vs. Validity vs.
Test Cutoffs Developmental College Level All Dev Ed All Coll. Lev
MathEarned B or higher in GK 0.643 0.722 0.278 -0.079 0.365Earned C or higher in GK 0.519 0.512 0.488 0.007 0.031Passed GK (D- or higher) 0.423 0.352 0.648 0.071 -0.225
EnglishEarned B or higher in GK 0.600 0.628 0.372 -0.028 0.228Earned C or higher in GK 0.430 0.345 0.655 0.085 -0.226Passed GK (D- or higher) 0.368 0.253 0.747 0.115 -0.379
Table 1. Summary of the Evidence on Placement Accuracy Ratesfor COMPASS® and ACCUPLACER®
Success Criterion: B or Higher Success Criterion: C or Higher
Test Target CourseCorr. Coeff.
Average/ Median
Accuracy Rate
Median Increase in Accuracy
RateCorr. Coeff.
Average/ Median
Accuracy Rate
Median Increase in Accuracy
Rate
COMPASS (Source: ACT Inc., 2006, pp. 103-104)Writing Skills Composition n/a 66 19 n/a 67 2Reading Skills Composition n/a 60 10 n/a 67 2Reading Skills Psychology n/a 68 31 n/a 67 4Numerical Skills/Prealgebra Arithmetic n/a 70 16 n/a 72 4Numerical Skills/Prealgebra Elementary Algebra n/a 67 25 n/a 63 6Algebra Intermediate Algebra n/a 71 25 n/a 68 5Algebra College Algebra n/a 72 43 n/a 67 20
ACCUPLACER (Mattern & Packman, 2009, p. 4)Sentence Skills Composition, Reading 0.19 59 n/a 0.13 75 n/aReading Comprehension Composition, Reading 0.17 62 n/a 0.10 80 n/aArithmetic Basic Math to Precalc 0.29 66 n/a 0.23 84 n/aElementary Algebra Basic Math to Precalc 0.27 65 n/a 0.25 73 n/a
Caveats
• Maximizing placement accuracy rates may not be the goal
• Our computation treats false positives and false negatives equally, but may care more about one than the other
• Values about which type of error is worse can be inferred from where the cutoff is placed– Ex: Pr(passingGK) for math at cutoff is 67%– This means those that are just below cutoff are wrongly placed—
false negatives—67% of the time– Could increase placement accuracy by lowering cutoff– But if we think failing someone in GK is 2x worse than making
someone take developmental unnecessarily, then cutoff is in the right spot
Figure 1
Probability of Gatekeeper Success, By Math Part 2 Scores
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 91 93 95 97 99
Math Part 2 Score
Per
cen
t M
eeti
ng
Su
cces
s C
rite
rio
n
Passed GK Earned C or Better Earned B or Better
Predictive Validity:Take-Away Messages
• Placement tests are much better at predicting who is likely to do well in gatekeeper than at predicting who is likely to fail
• Placement tests are more predictive of gatekeeper success in math than in english
• High school academic measures are almost as predictive as math test scores, and more predictive than english test scores
• Placement accuracy rates are only modestly higher in some cases, and substantially worse in others, than what would result if no tests were used– But weighting false positives and false negatives differently may
change this conclusion• Analysis of effectiveness of remediation still to come
For more information:
Please visit us on the web at http://ccrc.tc.columbia.edu,where you can download presentations, reports,
CCRC Briefs, and sign-up for news announcements.
CCRC is funded in part by: Alfred P. Sloan foundation, Bill & Melinda Gates Foundation, Lumina Foundation for Education, The Ford Foundation, National Science Foundation (NSF), Institute of Education Sciences of the U.S.
Department of Education
Community College Research CenterInstitute on Education and the Economy, Teachers College, Columbia University
525 West 120th Street, Box 174, New York, NY 10027 E-mail: [email protected]: 212.678.3091