LONGITUDINAL DATA ANDHIGHER EDUCATION ACCOUNTABILITY

Tom Schenk Jr., Consultant

National StudentClearinghouse

StudentRecords

StudentCourses

StudentAwards

Employee

Records

FacultyPositions

UnemploymentInsurance

Iowa College Student

Aid Commission

AS-28 [Program Information]

K-12[Project EASIER][Iowa Testing Services][CTE Plus]

PROJECT LEAD THE WAYEVALUATION

4 11th Grade ITED: Math

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 950%

2%

4%

6%

8%

10%

12%

14%

16%

18%

20%

NonparticipantsAverage = 65

ParticipantsAverage = 79

5 11th Grade ITED: Science

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 950%

2%

4%

6%

8%

10%

12%

14%

16%

18%

20%

6 Project Lead The Way

7 Project Lead The Way

8 Selection Bias

Socio-demographic Background

Cognitive ability

School-level factors}

9 Propensity Score Matching

y = φ(x1,…,xn)

Binary outcome

Logit Function

10

Propensity Bins

0% - 25% 25.1% - 50% 50.1% - 75% 75.1% - 100%

11

Subsequent Analysis

z = β1x1,…, βnxn, βyy

Outcome Propensity Bins

12

Enrollment by Gender

84%

16%

49%50%

Participants

NonparticipantsFemale Male

13

Percentage of Whites by Cohort

2008 2009 2010 20110.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%Participants

Nonpartici-pants

14

8th Grade ITBS: Math

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 950%

1%

2%

3%

4%

5%

6%

7%

8%

9%

NonparticipantsAverage = 58

ParticipantsAverage = 80

15

8th Grade ITBS: Science

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

-1%

0%

1%

2%

3%

4%

5%

6%

7%

8%

9%

NonparticipantsAverage = 61

ParticipantsAverage = 79

16

Evidence of Selection Bias

Participants are more likely to be male, by a wide margin.

Participation by gender holds constant around 90 percent.

Students are in the 80th percentile in math and science.

17

LABOR SUPPLY

18 Wages for employees in most sectors.

Provided on a quarterly basis.

Includes multiple jobs.

Excludes workers in some sectors (e.g., military).

Does not include hourly wages or full-time status.

Only includes employees within Iowa.

Unemployment Insurance

19

Matching Wage Data

• Educational data

MIS

20

Human Capital Theory

WagesCompleters

Direct Costs

Time

Leavers

21

Returns on Investment

Σ yi - xi (1+r)t

t=1

T

- C

22

Returns on Investment

Track wages over a set period of time, T.

Find the difference between wages between completers and leavers on the student level.

Find the cost of tuition.

If you assert an interest rate, r, then solve to find Net Present Value.

If the interest rate if left unknown, then solve to find the Internal Rate of Return.

23

Methodology

Assemble a cohort of graduates (completers) and those who left college without a degree (leavers).

Stagger the cohorts so completers are finishing their final year in college as leavers are in their first year in the workforce.

Exclude students who are found in any postsecondary institution.

Estimate the tuition expenses for completers in their final year.

Cohorts from 2002 and 2006.

24

Wages by Year

Net Present Value for Completers was $3,131.Internal Rate of Return was 6 percent.

25

Returns by ClusterN

PV

26

Returns by Select Clusters

28

In-state Retention Rate

In-stateOut-of-state

Working

School

WorkingSchool

HIGHER EDUCATIONACCOUNTABILITY

31

Slightly over 52 percent of first-time, full-time students entering in 2006 either transferred or graduated within three years (by 2008).

Success Rate

2005 20060%

10%

20%

30%

40%

50%

60%

32

33

The three year graduation rate for first-time, full-time students rose slightly to 39.1 percent for the 2006 cohort. The graduation rate has fluctuated, but stayed ahead of the national graduation rate (33 percent).

Graduation Rate

2001 2002 2003 2004 2005 20060%

10%

20%

30%

40%

50%

34

Time-to-degree rose slightly to 2.4 years for the 2004 cohort from 2.3 years.

Time to Degree

2003

2004

0 1 2

35

Accountability Systems

Measuring the effectiveness of institutions through student outcomes.

Desirable qualities of accountability measures:1. Specifically measures the effectiveness of the institution,

not other factors.2. Measures improvement.3. Flexible to accommodate a variety of outcomes.

The current traditional measures cannot capture these elements.

36

Accountability Systems

Issues with graduation/success rate:1. Denominator debate.2. Selection bias.3. Positive feedback mechanism.

Solutions: Use the same methods that assist with Project Lead The Way evaluations.

37

50%

Estimating Success

50%

50%

50%

38

Simulated Example

OBTAININGDATA

40

Obtaining Data

Contact your major professor and/or the Department of Education.

Education data can be obtained from the Department of Education, other data (e.g., UI records) will need special permission.

Data is distributed de-identified and must be returned or destroyed at the completion of the research study.

Obtaining data is relatively low-cost, nominal fee (<$100) for a secure flash drive.

Tom Schenk Jr.Consultant – Institutional Effectiveness &

AccountabilityIowa Department of EducationPhone: 515-281-3753E-mail: tom.schenk@iowa.gov