Holistic Assessment: From Research to Application

Copyright © 2013 by Educational Testing Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS). SUCCESSNAVIGATOR is a trademark of ETS. All other trademarks are the property of their respective owners.

Ross Markle, Ph.D., Senior Research and Assessment AdvisorEducational Testing Service

Renée Delgado-Riley, Ph.D., Program Planning OfficerThe University of New Mexico

Saturday, June 20, 2014

ATTAINED35%

ENROLLED19%

NO DEGREE,

NOT ENROLLED

46%

Students Starting at Two-Year Institutions

1National Center for Educational Statistics’ Beginning Postsecondary Students Survey (BPS: 04-09)2National Center for Education Statistics, Integrated Postsecondary Education Data System (IPEDS), Spring 2004.

Degree Attainment for Beginning College Students:Starting in 2003-04 Academic Year, followed through 20091

ATTAINED64%

ENROLLED12%

NO DEGREE,

NOT ENROLLED

24%

Students Starting at Four-Year Institutions

Of the 1.4 million students2 who enrolled in four-year schools in the fall of 2003, an estimated

525,686 STUDENTShad not received a degree as of 2009.

Of the 1.1 million students2 who enrolled in two-year schools in the fall of 2003, an estimated

743,909 STUDENTShad not received a degree as of 2009.

2

“Underserved Populations”

Cohort starting in:

WhiteAfrican

AmericanHispanic/

LatinoWhite

African American

Hispanic/ Latino

2000 60.2 42.1 49.1 31.5 26.1 30.1

2002 60.2 40.1 48.9 30.4 24.2 30.7

2004 61.5 39.5 50.1 29.0 22.9 26.3

Percentage of first-time, full-time undergraduate students receiving a degree/certificate within 150% of normal FTE. (Source: : U.S. Department of Education, National Center for Education Statistics, Integrated Postsecondary Education Data System.)

4-Year Institutions 2-Year Institutions

4

“Underserved Populations”

- We’ve done a very good job of identifying who succeeds and who fails, but not a very good job of understanding why (Bean & Eaton, 1995).

- Demographic and background characteristics predict success, but they don’t suggest the underlying causes of attrition or what we can do to improve persistence and graduation.

Carnevale & Strohl, (2010)

Course Placement:

• Students face a long, complex sequence of developmental courses.

Early Academic Success

• Students lack the effective behaviors (organization, study skills) that are needed to succeed in college-level courses.

Persistence Over Time

• Without well-developed and aligned goals, self-management skills, and social connections, students may fail to persist to a degree.

Where we lose students ...

5

6

Challenges to supporting student success

Course PlacementProblem: High remediation rates lead to the “Bridge to Nowhere” (CCA, 2012)

Need: Identify students who might succeed on an accelerated path

AdvisingProblem: Traditional indicators of student success are insufficient

Need: Better understanding of students’ likelihood of success, tied to course of action

Institutional PlanningProblem: A host of programs and services based on “best practices”

Need: Data to target and structure programs and services to the population-specific strengths and weaknesses

So what’s the solution?

What are noncogntive skills?

Conscientiousness

Teamwork

Institutional Commitment

Motivation

Metacognition

Factors outside of“academic ability” or

“academic intelligence” that contribute to or are part of student learning

Study Skills

Goal Setting

Self-Efficacy

Social Support

Response to Stress

Test-Taking Strategies

8

9

K-8 High School College

Conscientiousness 0.28 0.21 0.23

Agreeableness 0.3 0.05 0.06

Cognitive ability 0.58 0.24 0.23

Socioeconomic Status 0.32 0.32 0.32

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Co

rre

lati

on

w/

Stu

de

nt

Succ

ess

Poropat (2009) Meta-analysis

Construct rGPA

Academic Skills .129

Academic Goals .155

Academic Self-Efficacy .378

Institutional Commitment .108

Social Support .096

SES .155

ACT/SAT Scores .376

Construct rGPA rretention

Academic Skills .129 .298

Academic Goals .155 .210

Academic Self-Efficacy .378 .257

Institutional Commitment .108 .204

Social Support .096 .199

SES .155 .173

ACT®/SAT® Scores .376 .079

Robbins, et al. (2004) meta-analytic correlations with retention, GPA

10

11

Effort Level (participation, attendance, complete assignments)

High Medium Low

Math Readiness

High 96% 91% --

Medium 94% 77% 82%

Low 86% 61% 38%

n=411

Developmental Education Class Success Rates Based on Math Readiness and In-course Behavior Levels

Success rate not

calculated due to

insufficient sample size

Success rates for students with low readiness but high effort more

than twice as high as for low effort, low readiness

Research shows that behavioral monitoring and intervention are key to student success

Source: ETS RR- 13-12, Examining Evidence of Reliability, Validity, and Fairness for the SuccessNavigator™ Assessment , July 2013

12

Working with Subpopulations

• Many view noncognitive factors as key to improving success, particularly for traditionally disadvantaged populations (e.g., academically underprepared, first-generation or low-SES students)

• However, there are several possible ways in which noncognitive variables might relate to these populations, each of which has different implications for guiding intervention

Group Status

SuccessAcademic

Achievement

Noncognitive factors

Group Status Success

Academic Achievement

Noncognitive factors

A holistic assessment solution

• Targets students between pre- and early enrollment (post-admissions)

• Inclusion of both cognitive and psychosocial (“noncognitive”) factors

• Alignment between assessment and institutional practices

• Added value:– Respects the whole student (examines cognitive and noncognitive factors)

– Focuses on the unique characteristics of each student

– Focuses on factors educators can control: such as motivation, social connectedness, as opposed to socioeconomic, situational factors

13

The SuccessNavigator™

Assessment

. 14

Use of the SuccessNavigator™ Assessment

• Low stakes: diagnostic and developmental: A 30-minute online, nonproctored noncognitive assessment for incoming college students.

• Can be used with or without academic markers (SAT® scores, HS GPA, course placement score)

• Three primary intended uses:– Identifying likelihood of persistence to degree and academic success– Informing course placement acceleration decisions– Feedback for advising, FYE, etc.

• Assessments are scored immediately and will be sent to the candidates and academic advisors. Administrators have access to appropriate reports within the online system.

1515

16

Domain Facet Definition Example Items

Academic SkillsTools and strategies for academic success

Organization Strategies for organizing work and time.I make a schedule for getting my school work done.I take due dates seriously.

Meeting Class Expectations

Doing what’s expected to meet the requirements of your course including assignments and in-class behaviors.

I attend almost all of my classes.I complete the reading that is assigned to me.

CommitmentActive pursuit towardan academic goal.

Commitment to College Goals

Perceived value and determination to succeed in and complete college.

One of my life goals is to graduate college.The benefit of a college education outweighs the cost.

Institutional Commitment

Attachment to and positive evaluations of the school.

This is the right school for me.I’m proud to say I attend this school.

Self-Management Reactions to academicand daily stress

Sensitivity to StressTendency to feel frustrated, discouraged or upset when under pressure or burdened by demands.

I get stressed out easily when things don't go my way.I am easily frustrated.

Academic Self-Efficacy

Belief in one’s ability to perform and achieve in an academic setting.

I'm confident that I will succeed in my courses this semester.I can do well in college if I apply myself.

Test AnxietyGeneral reactions to test-taking experiences, including negative thoughts and feelings (e.g., worry, dread).

When taking a test, I think about what happens if I don't do well.Before a test, my stomach gets upset.

Social SupportConnecting with people and students resources for success

Connectedness A general sense of belonging and engagement.I feel connected to my peers.People understand me.

Institutional Support

Attitudes about and tendency to seek help from established resources.

If I don't understand something in class, I ask the instructor for help.I know how to find out what's expected of me in classes.

Barriers to SuccessFinancial pressures, family responsibilities, conflicting work schedules and limited institutional knowledge.

Family pressures make it hard for me to commit to school.People close to me support me going to college.

16

SuccessNavigator™ Score Reports

Advisor/Faculty Report

• General Skill scores, Success Indices around academic and enrollment success

• Granular feedback report based on facet scores

• Used to foster conversations and interventions with student

Student Report

• Feedback at domain level, NO SUCCESS INDICES

• Action plans around programs and services on campus

Institutional Report (Institution Aggregate Report and Custom Institution Report)

• Population and subpopulation statistics (based on normative samples) at domain and facet level

• Used for strategic planning, tailoring co-curricular programs and services

1717

18

RECOMMENDATION

Student Name: [Name]

Student ID: [ID#]

Score Date: [Date]

The Student’s Background

• Gender: [Male/Female]• Race: [Race]• Age: [Age]• Is English Best Language? [Yes/No]

Academic Skills – Tools and strategies to succeed in the classroom

Skills• Sometimes uses strategies to effectively manage time and

assignments• Occasionally misses class or comes unprepared

Tools/Tips

Refer the student to the [Tutoring Center] to provide goal-setting strategies and organizational tools. For additional ways of working with the student, click here.

Commitment – Active pursuit toward an academic goal

Skills• Sees great value in a college degree• Is very attached to the college

Tools/Tips

The [Career Center] can help the student align degree goals with career goals. For additional ways of working with the student, click here.

Social Support – Connections with people and resources for student success

Skills• Holds strong connections to people and resources• Effectively balances the demands of college and personal life

Tools/Tips

The [Office of Student Life] can connect the student with valuable student groups on campus. For helpful tools and tips about student connections, click here.

Self-Management – Reactions to academic and other stressors

Skills• Has difficulty managing stress in a positive, productive

manner• Doubts personal skills and abilities

Tools/Tips

Refer the student to the [Wellness Center] on campus to help manage pressures and succeed. For effective ways of helping the student manage everyday academic stress, click here.

MODERATE

ACADEMIC SUCCESS INDEX*

HIGH

RETENTION SUCCESS INDEX*

* Course Acceleration: Please see Technical User’s Guide to fully understand how to make an informed course placement.* Academic Success Index: Weighted composite of student’s SuccessNavigator™ profile and other academic indicators of student preparedness, such as self- or institution-

reported high school GPA, SAT/ACT, etc. * Retention Success Index: Projected likelihood that student will return for a second semester or year at the institution.

Advisor Report

SKILL REPORT

HIGH

HIGH

COURSE ACCELERATION*

MODERATE

LOW

COURSE ACCELERATION*

GLISH

MATH:CAUTION

ENGLISH: YES

Domain Scores: Four general areas of student strengths and weaknesses. Scores are presented normatively.

Feedback: Determine by more specific “facet” scores (see next page).

Action Plans: Suggested interaction with programs and services on campus.

Background Information: Communicate key student information from both SuccessNavigator™ and SIS to faculty/advisor.

Success Indices: Separate indices for both classroom and enrollment success. Based on background, cognitive and psycho-social information and supported by statistical relationships with success.

Using the SuccessNavigator™

Assessment

19

20

Advising

• Problem: Traditional indicators of student success do not capture many of the critical factors

• Need: To better understand students’ likelihood of success so that faculty and staff can determine the appropriate level of engagement

• Solution: Integrated holistic assessments with accurate and intuitive score reporting

21

22

From the Resource Library:Supports for Academic Skills and

Commitment

23

24

Course Placement

• Problem: Too many students are placed into developmental courses, effectively placing them on a “Bridge to Nowhere”

• Need: Identify students who are likely to succeed if placed in higher-level courses (i.e., shorter paths to success)

• Solution: Holistic assessments that indicate students who will put forth effort, attend class, study and succeed (even when their cognitive test scores might not suggest such)

Placement Test Score

SuccessNavigator™ Course Acceleration Indicator

Yellow – Caution Green - Accelerate

0-30 Low Developmental Course Low Developmental Course

31-40 Low Developmental Course* High Developmental Course

41-70 High Developmental Course High Developmental Course

71-80 High Developmental Course* College-Level Course

Above 80 College-Level Course College-Level Course

25

TESTS + The SuccessNavigator™

Assessment

PLACEMENT BANDS (?)

ACCELERATE

ACCELERATE WITH SUPPORTS?

ACCELERATION DECISION

1st Level Support

Intrusive Advising

2nd Level Support

Supplemental Instruction

DO NOT ACCELERATE

Instructional interventions at the class/program level

Interventions at the individual student level

26

27

Institutional Planning

• Problem: Institutions develop a panoply of services based on best practices that may or may not fit their population

• Need: Data to target programs and services to the specific strengths and weaknesses of students at a particular institution

• Solution: Aggregate reports of relevant psychosocial skills that are tied to institutional practice

• Tutoring• Advising• FYE (Study Skills)

• Advising• Career Counseling• FYE (Goal Setting)

• Counseling• Testing Center• FYE (Test-Taking Skills)

• Student Life• Clubs and Organizations• Orientation, FYE• (Help seeking)

GENERAL SKILL SCORES AT COMMUNITY COLLEGE of NJ

28

29

New Mexico at a Glance

Population: 2.1 million

• 1 of 4 minority-majority States, 57.9% minority population

• 46.7% of total population is Hispanic

New Mexico

UnitedStates

Living in Poverty 20.4% 15.3%

Bachelor’s Degree 14.2% 17.6%

Some College 24.9% 20.6%

Speak More Than One Language 35.8% 19.6%

(U.S. Census Bureau, 2010)

30

The University of New Mexico is a public, four-year degree-granting institution located on Old Route 66, in the heart of Albuquerque, New Mexico. Current enrollment is approximately 36,510 students.

UNM is a Hispanic-Serving Institution and is classified by the Carnegie Foundation as a Research One, Very High Research University. UNM offers a total of 210 degree and certificate programs, including 93 baccalaureate level degree programs, 68 master’s programs, and 37 doctoral programs.

History of UNM

Copyright © 2014 by Educational Testing Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS). SUCCESSNAVIGATOR is a trademark of ETS. All other trademarks are the property of their respective owners. 25841

31

32

UNM at a Glance

UNM Hispanic Student Challenges

Hispanic White

Hispanic Postsecondary Completion (6 years)42% 49%

Postsecondary Drop Out Rates29.5% 15.1%

Student: Advisor Ratio 335:1

(UNM Office of Institutional Research, 2011; UNM Provost Report, 2007; U.S. Census Bureau, 2010)

Copyright © 2014 by Educational Testing Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS). SUCCESSNAVIGATOR is a trademark of ETS. All other trademarks are the property of their respective owners. 25841

33

Driving Student Success Through the SuccessNavigator™ Assessment

Early Alert Coaching & AdvisingTarget Student

Populations

• Identify early risk• Develop strategies for

intervention• Provide programming to increase

persistence

• 1-on-1 support and guidance• Resources to navigate through

college• Connecting students to on-

campus resources and programming inside and outside the classroom

• Develop profiles of noncognitivescores based on race/ethnicity and gender

• Develop interventions thatspecifically align to those skills and profiles

At UNM: Action for Every Student

High Skill:

Leadership

Moderate Skill:

Refinement

Low Skill:

Readjustment

Copyright © 2014 by Educational Testing Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS). SUCCESSNAVIGATOR is a trademark of ETS. All other trademarks are the property of their respective owners. 25841

34

35

Academic Skills Commitment Self-Management Social Support

CAPS

Accessibility Resource

Center

College Enrichment &

Outreach Programs (e.g.,

CEP, SSS)

Advisement (University,

Fine Arts, Engineering)

Academic Coaching

Academic Level Student

Support (e.g., Engineering

Student Services, Student

Services, etc.)

Faculty Office Hours

(Noted on course syllabi)

Career Services

Academic Departments

Student Activities (i.e.,

Greeks, Student

Organizations, etc.)

Advisement (University,

Fine Arts, Engineering)

Alumni Support Mentoring

Program

Student Employment

Residential Life

Student Health &

Counseling Center (SHAC)

CAPS

Targeted Resource Centers

(African American Student

Services, American Indian

Student Services, El Centro,

LGBTQ Resource Center,

Men of Color Initiative,

Women’s Resource Center,

Veteran’s Resource Center)

Religious Organizations

(e.g., Newman Center, etc.)

College Enrichment &

Outreach Programs (e.g.,

CEP, SSS)

Academic Coaching

Recreational

Services_Johnson Center

Targeted Resource Centers

(African American Student

Services, American Indian

Student Services, El Centro,

LGBTQ Resource Center, Men

of Color Initiative, Women’s

Resource Center, Veteran’s

Resource Center)

Student Activities (i.e.,

Greeks, Student

Organizations, etc.)

College Enrichment &

Outreach Programs (e.g.,

CEP, SSS)

Dean of Students

Student Employment

Recreational

Services_Johnson Center

Early Alert

http://success.unm.edu/success-navigator/index.html

36

Coaching & Advising

Copyright © 2014 by Educational Testing Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS). SUCCESSNAVIGATOR is a trademark of ETS. All other trademarks are the property of their respective owners. 25841

37

Comparing Men of Color to the General Population

General

Mean

(n=800)

M.O.C.

Mean

(n=209)

Total SDCohen's

d

Background

ACT® Composite 22.9 22.3 4.1 -0.15

H.S. GPA 3.4 3.3 0.5 -0.30

Expected Family Contribution 14,710 10,344 19,933 -0.22

SuccesNavigator™

Subskills

Organization 102.2 94.7 15.2 -0.49

Sensitivity to Stress 102.1 107.3 13.6 0.38

Test Anxiety 100.6 105.9 14.3 0.37

Barriers to Success 101.9 100.8 12.9 -0.09

Campus

Engagement

Visits to Tutoring, Advising 0.0 0.3 0.8 0.42

Faculty Mentors, Study Groups,

Clubs/Organizations0.0 0.0 0.7 -0.04

Showing up, Completing Assignments 0.0 0.2 0.8 0.37

Outcomes

Completing 1st Math Course 84.7% 83.9% 0.4 -0.02

Completing 1st English Course 93.8% 90.2% 0.3 -0.14

First Semester GPA 3.19 2.93 0.9 -0.30

Persistence to Spring 93.1% 89.0% 0.3 -0.15

Targeting Student Populations

38

• Campuswide involvement in the process• Thorough exploration and dissemination of the data• External and internal analyses and presentations of data• Sharing and offering data with programs • Upcoming Initiatives:

First Year Required Course Student Success Center Restructure New Student Orientation Gamification of Student Engagement

Holistic Student Success

Copyright © 2014 by Educational Testing Service. All rights reserved. ETS, the ETS logo and LISTENING. LEARNING. LEADING. are registered trademarks of Educational Testing Service (ETS). SUCCESSNAVIGATOR is a trademark of ETS. All other trademarks are the property of their respective owners. 25841

39

Embracing a Holistic Viewpoint

• Improving student success will require us to consider a broader set of skills.

• We must transform data to information, and information to action.

• We will also have to integrate student success efforts into a holistic plan of action.

Recognize you can begin to increase retention and success rates as early as your next incoming class.