Copyright © 2015 by Educational Testing Service. All rights reserved. ETS, the ETS logo, LISTENING. LEARNING. LEADING. and SUCCESSNAVIGATOR are registered trademarks of Educational Testing Service (ETS). All other trademarks are the property of their respective owners.

27563

1

Driving Student Success:Course Acceleration and Holistic Advising

ATTAINED 35%

ENROLLED 19%

NO DEGREE,

NOT ENROLLED

46%

Students Starting at Two‐Year Institutions

1National Center for Educational Statistics’ Beginning Postsecondary Students Survey (BPS: 04‐09)2National Center for Education Statistics, Integrated Postsecondary Education Data System (IPEDS), Spring 2004.

Degree Attainment for Beginning College Students: Starting in 2003-04 Academic Year, followed through 20091

ATTAINED 64%

ENROLLED 12%

NO DEGREE,

NOT ENROLLED

24%

Students Starting at Four‐Year Institutions

Of the 1.4 million students2 who enrolled in 4 ‐ year schools in the Fall of 2003, an

estimated 525,686 STUDENTS

had not received a degree as of 2009.

Of the 1.1 million students2 who enrolled in 2 ‐ year schools in the Fall of 2003, an

estimated 743,909 STUDENTS

had not received a degree as of 2009.

More than 1.7 million students began community college in Fall 2014...

But are they prepared?• 50% of community college students

need remedial courses.1

• 20% of four‐year college students also start in remediation. 1

• 30% of these students who are placed into remedial courses fru‐ strated by their low placement don’t‐ even bother to enroll in classes. 2

• Fewer than 1 in 4 students who enroll in developmental courses complete a degree.2

1Complete College America (2012) 2Bailey, Jeong, & Woo‐Choo (2008)

Developmental Education: Jumping through Hoops

Practical Challenges:

Students face a long, difficult

sequence, sometimes

containing 4‐5 additional semesters

of coursework.

Academic Challenges:

Students enter with deficiencies, and are placed into the same

class settings that created those deficiencies.

Noncognitive Challenges:

Time management and study skills help

students manage college l‐ evel

expectations and work load.

4

• Curricular alignment with K-12Russell (2008)

• Course redesign (e.g., co-requisite models)Adams, Gearhart, Miller, & Roberts (2009)

• Curricular redesign (e.g., emporium models)Twigg (2009)

• Course accelerationComplete College America (2012); Edgecombe (2011); Scott-Clayton (2012)

• Improved/holistic assessment and placementBoylan (2009); Burdman (2012); Conley (2007); Levine-Brown, Bonham(2008); Saxon, & Boylan (2008) 5

How can we address developmental education?

What are noncognitive skills?

Institutional Commitment

Conscientiousness

Social Suppor t

Teamwork

Self-efficacy

6

Motivation

Metacognition

Study Skills

Goal Setting

Response to Stress

Test Taking Strategies

7

Construct rGPA

Academic-related Skills .129

Academic Goals .155

Academic Self-efficacy .378

Institutional Commitment .108

Social Support .096

SES .155

ACT® or SAT® Scores .368Robbins, et al. (2004) meta-analytic correlations with retention, GPA

Construct rGPA rretention

Academic-related Skills .129 .301

Academic Goals .155 .212

Academic Self-efficacy .378 .259

Institutional Commitment .108 .206

Social Support .096 .204

SES .155 .212

ACT® or SAT® Scores .368 .121

n=713Success rates for students

with low readiness but high effort more than three times as those for low effort, low

readiness7

Li., K., et al. (2013). Readiness, behavior, and foundational mathematics course success. Journal of Developmental Education, 13 (1), 14 22.‐

Achievement, Behavior, and Success in Math Courses

Effort Level(participation, attendance, complete assignments)

High Medium Low

Math Readiness

High 92% 80% 59%

Medium 86% 67% 29%

Low 74% 50% 19%

Holistic Assessment and Developmental Education: The SuccessNavigator®

Assessment

• A 30‐minute online, nonproctored, noncognitive assessment for incoming college students: Low stakes, diagnostic and developmental

• Can be used with or without academic markers (SAT® scores, HS GPA, course placement score)

• Assessments are scored and available immediately to students and advisors. Administrators have access to reports and data within the online system.

• Uses in a developmental education, course acceleration context:– Provides a composite score, independent of placement tests, that

recommends students who are likely to succeed when placed into higher level courses.

– Provides noncognitive scores and feedback, as well as tailored action plans, tools and tips, and recommendations for campus resources that help students focus on skills and behaviors that can improve success

9

General Skill

10

Subskill Definition Example Items

Academic SkillsTools and strategies for academic success

Organization Strategies for organizing work and time.

I make a schedule for getting my school work done. I take due dates seriously.

Meeting Class Expectations

Doing what’s expected to meet the requirements of your course including assignments and in‐class behaviors.

I attend almost all of my classes.I complete the reading that is assigned to me.

CommitmentActive pursuit towardan academic goal

Commitment to College Goals

Perceived value and determination to succeed in and complete college.

One of my life goals is to graduate college.The benefit of a college education outweighs the cost.

Institutional Commitment

Attachment to and positive evaluations of the school.

This is the right school for me.I’m proud to say I attend this school.

Self ‐management Reactions to academicand daily stress

Sensitivity to Stress

Tendency to feel frustrated, discouraged or upset when under pressure or burdened by demands.

I get stressed out easily when things don't go my way.I am easily frustrated.

Academic Self ‐ Efficacy

Belief in one’s ability to perform and achieve in I'm confident that I will succeed in my courses thisan academic setting. semester.

I can do well in college if I apply myself.

Test Anxiety

General reactions to test‐taking experiences, including negative thoughts and feelings (e.g., worry, dread).

When taking a test, I think about what happens if I don't do well.Before a test, my stomach gets upset.

SocialSupport Connecting with people and studentsresources for success

Connectedness A general sense of belonging and engagement. I feel connected to my peers. People understand me.

Institutional Support

Attitudes about and tendency to seek help from established resources.

If I don't understand something in class, I ask the instructor for help.I know how to find out what's expected of me in classes.

Barriers to Success

Financial pressures, family responsibilities, conflicting work schedules and limited institutional knowledge.

Family pressures make it hard for me to commit to school.People close to me support me going to college.

Feedback: Determine bymore specific “facet” scores (see next page).

Action Plans: Suggested interaction with programs and services on campus.

Success Indices: Separate indices for both classroom and enrollment success. Based on background, cognitive and psycho ‐social information and supported by statistical relationships with success.

Background Information: Communicate key student information from both SuccessNavigator® and SIS to faculty/advisor.

Domain Scores: Four general areas of student strengths and weaknesses. Scores are presented normatively.

11

Academic Success: Classroom success broken into ability to meet expectations and organizational skills.

Commitment: Measures

Self mana‐ gement: Ability to handle stress, testing anxiety and academic self ‐efficacy.

Social Support: Tendency to seek help, attitude toward barriers to success and ability to relate to peers in school.

.both commitment tocollege and commitment to the specific institution.

12

13

Holistic Assessment and Course Acceleration

Level

COM

PASS Score IWCC Math Course(s)

1 0 – 39MAT 075

2 40 – 44MAT 102, MAT 110

3 45 – 52 MAT 157

4

53

or aboveM

AT 121, MAT 129

Level COMPASS Score

IWCC Math Course Based on SuccessNavigator® Course Acceleration Indicator

Yellow – Caution Green ‐ Accelerate

1 0‐35 MAT 075

Decision Zone 1/2 36 ‐ 39MAT 075 or

MAT 102/110MAT 102/110

2 40*** MAT 102/110

Decision Zone 2/3 41 – 44MAT 102/110 or MAT 157

MAT 157

3 45 – 48 MAT 157

Decision Zone 3/4 49 – 52MAT 157 or

MAT 121/129MAT 121/129

4 53 or above MAT 121/129

• Studied 70,000 first-time degree-seekers from a large, urban CC system

• Used regression discontinuity modeling to predict performance of dev. ed. students in college level courses using multiple measures

• Major findings:1. One quarter of math students and one third of English students

are “severely misplaced”2. Using multiple measures could lower remediation rates by 8% in

math and 12% in English3. Roughly 20% of students in math and 30% of students in English

are severely under-placed, meaning they could were predicted to receive a B or better in the college-level course

Scott‐Clayton, J. (2012). Do high‐stakes placement exams predict college success?(CCRC Working Paper No. 4). New York: Community College Research Center.

15

Does course acceleration actually work?Evidence from the Community College Research Center

ETS Course Acceleration Case StudyStudy Description

• Spring 2014: 1,549 students from 4 CC’s in a large urban system

• 40% African American, 20% Hispanic/Latino

• 25% Credit bearing• 65% Remedial credit• 15% Non c‐ redit

Placement Test Scores

Decision Zones

+ SuccessNavigator

= COURSEACCELERATION

1. Students with green acceleration indices (56%) were more likely to pass their math courses than

students with yellow indices (46%; p < .01).

RESULTS2. This effect was larger at

higher l‐ evel math courses:

3. Students who were accelerated into college ‐level math courses (SN + placement test) passed

at rates similar to students who placed

“naturally” (placement test only).

See Rikoon et al. (2014)

Green Yellow

College 66.8 37.7

Dev. Ed. 1 53.4 48.3

Dev. Ed. 2 47.3 47.4

Dev. Ed. 3 53.3 50.7

• The decision zone model is designed to simplify and standardize the placement process

• There are several ways of creating these zones:– A normative approach – e.g., the top 20% of students in a

placement level– A distribution approach – e.g., within one standard

deviation of the cut score– A measurement approach – e.g., within one standard

error of the cut score

• Factors to consider when choosing a decision zone include the size of the band and the number of students who would be eligible for acceleration

17

Creating Decision Zones

18

Combining Holistic Placement and Holistic Advising

Course Placement• Students face a long,

complex sequence of developmental courses.

Early Academic Success• Students lack the

effective behaviors (organization, study skills) that are needed to succeed in college l‐ evel courses.

Persistence Over Time• Without well ‐

developed and aligned goals, self ‐management skills and social connections, students may fail to persist to a degree.

19

Where We Lose Students ...

TESTS +The Success Navigator® Assessment

PLACEMENT BANDS (?)

ACCELERATE

ACCELERATE WITH SUPPORTS?

ACCELERATION DECISION

1st Level Support

Intrusive Advising

2nd Level Support

Supplemental Instruction

DO NOT ACCELERATE

Instructional interventions at the class/program level

Interventions at the individual student level

26

Student J• 15 ACT®, 2.5 HSGPA• Strong Academic Skills• Moderate Commitment• Low Self-management• Strong Social Support• Why does Student J

have such a strong likelihood of success?

• What interventions might we recommend for Student J?

21

Student J• Academic Skills and

Commitment to College Goals are very relevant to both academic success and persistence

• Low Self-management can actually increase students’ likelihood for persistence (a lack of “stress” can also be a lack of “engagement”)

• Moderate Academic Success Index is likely coming from tests scores and HSGPA: tutoring and other academic interventions are likely most important

22

Student J – Self-management Interventions

23

Understanding your institution’s resources...Academic Skills Commitment Self-management Social Support

CAPS Accessibility Resource

Center College Enrichment &

Outreach Programs

(e.g., CEP, SSS) Advisement

(University, Fine Arts,

Engineering) Academic Coaching Academic Level

Student Support (e.g.,

Engineering Student

Services, Student

Services, etc.) Faculty Office Hours

(Noted on course

syllabi)

Career Services Academic Departments Student Activities (i.e.,

Greeks, Student

Organizations, etc.) Advisement

(University,

Fine Arts, Engineering)

Alumni Support

Mentoring Program Student

Employment Residential Life

Student Health &

Counseling Center (SHAC)

CAPS

Center, Veteran’s

Resource Center)

Religious Organizations

(e.g., Newman Center, etc.)

College Enrichment &

Outreach Programs (e.g.,

CEP, SSS)

Academic Coaching

Recreational

Services_Johnson Center

Targeted Resource

Centers (African-American

Student Services,

American-Indian Student Targeted Resource

Centers (African-American

Student Services,

American-Indian Student

Services, El Centro,

LGBTQ Resource Center,

Men of Color Initiative,

Women’s Resource

Services, El Centro,

LGBTQ Resource Center,

Men of Color Initiative,

Women’s Resource

Center, Veteran’s

Resource Center) Student Activities (i.e.,

Greeks, Student

Organizations, etc.) College Enrichment &

Outreach Programs (e.g.,

CEP, SSS) Dean of Students Student Employment Recreational

Services_Johnson Centerhttp://success.unm.edu/success n‐ avigator/index.html

29

<<Institution>>

• Administration window: <<08/01/13–09/15/13>>• Total sample size = <<1,138>>

SUCCESS INDICESPredictions of student success are based on two criteria. First, ACADEMIC SUCCESS is a student’s likelihood of succeeding in the classroom, indicated byGPA. Second, ENROLLMENT SUCCESS indicates a student’s likelihood of returning to your institution for a second year. Both of these scores are modeled using a large, nationwide study across varying types of institutions and students. They have been shown to be highly predictive of student success.

The tables below show the proportion of your students who have fallen in each of three categories — high, medium and low likelihood of success.

Institution Report

25

Academic Success IndexSuccess Likelihood Definition Percent of Students Across

Comparable InstitutionsPercent of Students in the

Cohort/Institution

High Projected 1st Semester GPA> 2.97

26 33

Medium Projected 1st Semester GPA between 2.23 and 2.97

46 46

Low Projected 1st Semester GPA< 2.23

27 21

Retention Success IndexSuccess Likelihood Definition Percent of Students Across

Comparable InstitutionsPercent of Students at Your

Institution

High Probability of retention> 93%

21 42

Medium Probability of retention between 84.1% and 93%

48 29

Low Probability of retention < 84.1% 32 39

The pages that follow will provide subscores within each domain to explain them more thoroughly.26

Institution Report

GENERAL SKILL SCORES

Ross MarkleSenior Research & Assessment Advisor, Global Education Division, ETS

rmarkle@ets.org or 1-615-835-3483

Vikki MonaghanDirector of Strategic Initiatives, Global Education Division, ETS

vmonaghan@ets.org or 1-609-683-2965

Copyright © 2015 by Educational Testing Service. All rights reserved. ETS, the ETS logo, LISTENING. LEARNING. LEADING. and SUCCESSNAVIGATOR are registered trademarks of Educational Testing Service (ETS). All other trademarks are the property of their respective owners.

27563

27

Contact Information

• Adams, P., Gearhart, S., Miller, R., & Roberts, A. (2009). The Accelerated Learning Program: Throwing Open the Gates. Journal of Basic Writing (CUNY),28(2), 50-69.Bailey, T., Jeong, D. W., & Cho, S. W. (2010). Referral, enrollment, and completion in developmental education sequences in community colleges. Economics of Education Review, 29(2), 255-270.Boylan, H. R. (2009). Targeted intervention for developmental education students (TIDES). Journal of Developmental Education, 32(3), 14-23.Burdman, P. (2012). Where to begin? The evolving role of placement exams for students starting college [Report]. Boston, MA: Jobs for the Future.Complete College America. (2012). Remediation: Higher education’s bridge to nowhere. Washington, DC: Author. Conley, D. T. (2007). Toward a more comprehensive conception of college readiness. Eugene, OR: Educational Policy Improvement Center.Levine-Brown, P., Bonham, B. S., Saxon, D. P., & Boylan, H. R. (2008). Affective assessment for developmental students, part 2. Research in Developmental Education, 22(2), 1-4.Li, K., Zelenka, R., Buonaguidi, L., Beckman, R., Casillas, A., Crouse, J., Allen, J., Hanson, M.A., Acton, T., & Robbins, S. (2013). Readiness, behavior, and foundational mathematics course success. Journal of Developmental Education, 37(1), 14.Markle, R.E., Olivera-Aguilar, M., Jackson, T., Noeth, R., & Robbins, S. (2013). Examining evidence of reliability, validity, and fairness for SuccessNavigator. (ETS Research Report 13-12). Princeton, NJ: Educational Testing Service.Olivera-Aguilar, M., Markle, R., & Robbins, S. (April, 2014). Using latent profile analysis to identify profiles of college droupouts. Paper presented to the National Council on Measurement in Education: Philadelphia, PA.Rikoon, S., Liebtag, T., Olivera-Aguilar, M., Robbins, S., & Jackson, T. (2014). A pilot study of holistic assessment and course placement in community college: Findings and recommendations. ETS Research Memorandum (No. RM-14-10). Princeton, NJ: Educational Testing Service.Robbins, S. B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261-288.Russell, A. (2008). Enhancing college student success through developmental education. (Higher Education Policy Brief ). Washington, DC: American Association of State Colleges and Universities.Saxon, D. P., Levine-Brown, P., & Boylan, H. R. (2008). Affective assessment for developmental students, part1. Research in Developmental Education,22(1), 1-4.Scott-Clayton, J. (2012). Do high-stakes placement exams predict college success? (CCRC Working Paper No. 41). New York: Community College Research Center. Retrieved online from http://ccrc.tc.columbia.edu/Publication.asp?UID=1026. Twigg, C. A. (2011). The math emporium: Higher education’s silver bullet. Change: The Magazine of HigherLearning, 43(3), 25-34.

•

•

•

••

•

•

•

•

•

•

•

•

•

•

28

References