1 Topical Call Series: Improving Data Quality and Use CSPR Data Collection Tuesday, September 15,...

1

Topical Call Series: Improving Data Quality and UseCSPR Data CollectionTuesday, September 15, 2015

2

Who Is on the Call?

What is your role?• Title I, Part D State Coordinator• EDFacts or CSPR Coordinator at SEA• Other SEA staff• State-run facility staff (Subpart 1)• Locally run facility staff (Subpart 2)

3

Agenda

• Overview of CSPR Data Collection• Overview of CSPR Resources and Tools• CSPR Collection: Table-by-Table

– Review data indicators– Data quality checks

• State Data Collection Tools and CSPR Timelines in States

NDTAC is funded through a contract with U.S. Department of Education. The content of these slides and presentation does not necessarily reflect the views or policies of the U.S. Department of Education, nor

does it imply endorsement by the U.S. Government.

4

Overview of CSPR Data Collection

5

Statutory Requirements

Each State agency and local educational agency shall—

(1) submit evaluation results to the State educational agency and the Secretary; and

(2) use the results of evaluations under this section to plan and improve subsequent programs for participating children and youth.

State and local agencies receiving Title I, Part D funds must evaluate their programs’ impact on the ability of students:

(1) to maintain and improve educational achievement;

(2) to accrue school credits that meet State requirements for grade promotion and secondary school graduation;

(3) to make the transition to a regular program or other education program operated by a local educational agency;

(4) to complete secondary school (or secondary school equivalency requirements) and obtain employment after leaving the correctional facility or institution for neglected or delinquent children and youth; and

(5) as appropriate, to participate in postsecondary education and job training programs.

Source: Title I, Part D, Statute, Subpart 3

6

Federal Collection Requirements

All Title I, Part D-funded programs must collect data to report to the U.S. Department of Education (ED).

Program categories:• At-Risk programs—Subpart 2 only

• Other Programs

• Neglected Programs

• Juvenile Detention

• Juvenile Corrections

• Adult Corrections—Subpart 1 only

7

Federal Reporting Systems:CSPR and EDFacts

• CSPR is a data collection instrument administered annually by ED’s Office of Elementary and Secondary Education (OESE).

• EDFacts is an ED initiative to collect, analyze, report on, and promote the use of high-quality performance data.

• Most of the Title I, Part D data are now reported through EDFacts. Eventually, all data will be reported through the EDFacts’ online EDEN Submission System (ESS).

8

Relation Between Federal Reporting Systems

Category/Area

CSPR Tables

EDFactsSpecification FilesSubpart 1 Subpart 2

Facility Counts/Programming

2.4.1.1 & 2.4.1.1.1

2.4.2.1 & 2.4.2.1.1

Not included

Student Count/Demographics 2.4.1.2 2.4.2.2

S1 = C119 (Data Group 656)S2 = C127 (Data Group 657)

Transition Services2.4.1.3.1 2.4.2.3.1

S1 & S2 = C182 (Data Groups 786, 787)

Academic/Vocational Outcomes

2.4.1.3.2 2.4.2.3.2

In Program = C180 (Data Groups 782, 783)Exited Program = C181 (Data Groups 784, 785)

Pre- and Posttesting in Reading and Math

2.4.1.6.1 & 2.4.1.6.2

2.4.2.6.1 & 2.4.2.6.2

S1 = C113 (Data Group 628)S2 = C125 (Data Group 629)

9

Why Is Data Quality Important?

Trusting your data is important for informing:

• funding and other decisionmaking;

• technical assistance (TA) needs;

• subgrantee monitoring; and

• student programming.

10

Role of the Part D Coordinator in Improving Data Quality

Ultimately, coordinators cannot “make” the data be of high quality, but they can implement systems that make it more likely by:• understanding the collection process;• providing TA in advance;• developing relationships;• developing multilevel verification processes;• tracking problems over time;• using the data; and• linking decisions (funding, hiring, etc.) to data evidence.

11

Overview of CSPR Resources and Tools

12

EDFacts Initiative Web Page

http://www2.ed.gov/about/inits/ed/edfacts/index.html

• Overview of EDFacts

• EDFacts File Specifications

• CSPR Documentation (a.k.a. Data Reporting Forms)



13

Partner Support Center

Provides assistance to EDFacts customers

who submit data to the Education Data

Exchange Network (EDEN) or use its

analytical and reporting tools.

Contact Information:

Toll Free: 877-457-3336(877-HLP-EDEN)

Fax: 888-329-3336(888-FAX-EDEN)

TTY: 888-403-3336(888-403-EDEN)

E-mail: [email protected]

mailto:[email protected]

14

CSPR Resources: CSPR Data Reporting Forms for School Year 2014–15

• Official Part II reporting forms are on the ED Web site.

• Part D (Section 2.4) can be found on page 35.

http://www2.ed.gov/admins/lead/account/consolidated/csprpart21415.doc



15

CSPR Resources: The Instructional Guide to Reporting Title I, Part D Data in the CSPR for SY 2014–15

• Also known as “The CSPR Guide”

• The “encyclopedia” of the CSPR Data Collection

http://www.neglected-delinquent.org/sites/default/files/NDTAC_CSPR_guide_201314.pdf

• Topical call on how to utilize “The CSPR Guide”:

http://www.ndcommunities.org/calls/ndtac-resources-meet-technical-assistance-needs-call- 1




http://www.ndcommunities.org/calls/ndtac-resources-meet-technical-assistance-needs-call-1




16

CSPR Resources: TIPD Data Collection List for SY 2014–15

• Lists data indicators needed for CSPR and EDFacts data collection

• Briefly defines each item

It’s a good place for your “shopping list” of data

items.

http://

www.neglected-delinquent.org/

sites/ default/files/CSPR_collection-list.doc

http://www.neglected-delinquent.org/sites/default/files/CSPR_collection-list.doc





17

CSPR Resources: TIPD Data Collection and Submission Timeline for SY 2014–15

This will help you see the bigger picture during planning and explain data collection process to stakeholders.

http://www.neglected-delinquent.org/sites/default/files/TIPD_DataCollectTimeline.pdf

http://www.neglected-delinquent.org/sites/default/files/TIPD_DataCollectTimeline.pdf

18

CSPR Resources: Checklist for Performing Data Quality Reviews of TIPD Data

• Can be used for in-depth data reviews

• Checklists for:– programs and facilities;– students served;– transition services;– academic and vocational

outcomes; and– academic performance in

reading and math.

• Can be used by anyone responsible for collecting, entering, or reviewing the data (at school, LEA, SA, or SEA levels)

http://www.neglected-delinquent.org/sites/default/files/NDTAC_TIPD_DataQualityChecklist.pdf



19

Reporting Tool: Creating a Weighted Average Length of Stay

• Instructions with examples that guide calculations for reporting average length of stay

• Can be used by anyone responsible for collecting and entering data (at LEA, SA, or SEA level)

http://www.neglected-delinquent.org/ sites/default/files/NDTAC_ReportingTool_AverageLengthofStay.doc

http://www.neglected-delinquent.org/sites/default/files/NDTAC_ReportingTool_AverageLengthofStay.doc



20

Reporting Tool: Collecting and Reporting Racial/Ethnic Data in Seven Categories

• Describes how data should be collected (two-part question) and reported (seven categories), per ED requirements

• Can be used by anyone responsible for collecting and reporting data (at school, LEA, SA, or SEA level)

http://www.neglected-delinquent.org/administering-title-i-part-d/reporting-and-evaluation




21

Reporting Tool: Reporting Complete Pre- and Posttest Results for Reading/Mathematics

• Provides sample calculations to help avoid common data collection errors

• Can be used by anyone responsible for collecting, reporting, or reviewing data (at school, LEA, SA, or SEA level) to identify inconsistencies in performance data

http://www.neglected-delinquent.org/sites/default/files/ReportingTool_PrePostResults.pdf




22

CSPR Data Collection:Table-by-Table

23

2.4.1.1/2.4.2.1Programs and Facilities

Program/Facility Type# Programs/

FacilitiesAverage Length of

Stay in Days

Neglected Programs

Juvenile Detention

Juvenile Corrections

Adult Corrections/At-Risk

Other

Total (Auto calculated) //////////////////////////////

24

2.4.1.1.1/2.4.2.1.1: Programs and Facilities That Reported

Program/Facility Type # Reporting Data

Neglected Programs

Juvenile Detention



Other

Total (Auto calculated)

25

Examples From Colorado: What To Do When a Facility Closes or Refuses Future Funds

• Colorado had an S1 facility submit an annual count in SY 2013–14, but it chose to refuse funds for SY 2014–15. – SEA had the facility submit a letter saying that they

were turning down the funds for SY 2014–15 and that they would still submit data for the youth they served in SY 2013–14.

• Colorado had an S2 neglected facility close in January 2014 (mid SY 2013–14).– Going through the district, the SEA obtained the

SY 2013–14 data from the facility that closed.

26

2.4.1.1/2.4.2.1 and 2.4.1.1.1/2.4.2.1.1Data Quality Checks

2.4.1.1.1 Programs and FacilitiesThat Reported2.4.1.1 Programs and Facilities

Program/Facility Type

# of Programs/ Facilities

Neglected Programs 2

Juvenile Detention 3

Juvenile Corrections 2

Adult Corrections 4

Other 2

Total 13


#Reporting

Data




Adult Corrections 4

Other 1

Total 12

27

2.4.1.1/2.4.2.1 and 2.4.1.1.1/2.4.2.1.1Data Quality Checks

2.4.1.1.1 Programs and FacilitiesThat Reported2.4.1.1 Programs and Facilities


# of Programs/ Facilities




Adult Corrections 4

Other 2

Total 13


#Reporting

Data




Adult Corrections 4

Other 1

Total 12

28

2.4.1.2/2.4.2.2 Students Served: Unduplicated Count and Long-Term Students

• “Unduplicated” means that the student is counted only once, even if admitted to a facility multiple times within the year.

• “Long-term” refers to students with a length of stay of 90 consecutive calendar days or more within the reporting year. Do not add together separate enrollments. The number of long-term students reported should not exceed the unduplicated count for each program.

# of Students ServedNeglected Programs

Juvenile Detention


Adult Corrections/

At-RiskOther

Programs

Total Unduplicated Students Served

Long-Term Students Served

29

2.4.1.2/2.4.2.2Student Subgroups

Student Subgroups

Neglected Programs

Juvenile Detention


Adult Corrections/

At-RiskOther

ProgramsStudents With Disabilities (Individuals with Disabilities Education Act of 1990 (IDEA))

Limited English Proficiency (LEP) Students

30

2.4.1.2/2.4.2.2Students Served: Race/Ethnicity and Sex

Race/EthnicityNeglected Programs

Juvenile Detention


Adult Corrections/

At-RiskOther

Programs

American Indian or Alaska Native

Asian

Black or African American

Hispanic or Latino

Native Hawaiian or Other Pacific Islander

White

Two or More Races

Total (Auto calc) (Auto calc) (Auto calc) (Auto calc) (Auto calc)

Sex

Male

Female


31

2.4.1.2/2.4.2.2Students Served: Age

Age in YearsNeglected Programs Juvenile Detention



Other Programs

3 through 5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21


32

2.4.1.2/2.4.2.2: Race/Ethnicity

Polling Question: You have a facility that served 100 students, but 5 students did not self identify their race/ethnicity. What do you do?

• Have the facility submit what they think the race/ethnicity is when a respondent refuses to self identify their race/ethnicity

• Submit the data for the 95 students without a comment

• Submit the data and provide a comment about the missing data for five unidentified students

• Include the five students in a randomly chosen race/ethnicity category

33

Example From Alaska: How To Submit Data on Students Who Age Out of a Program

• Alaska had a few students who aged out of the program (i.e., turned 22 during the school year). Initially, Alaska submitted the age data without these students and added a comment.

• NDTAC provided guidance to include these students as under age 21, because you are to report the age the student was when he/she left the facility/program (i.e., the student was no longer served by Part D after turning 22).

• Alaska has since updated its data collection tool to reflect the guidance they received.

34

2.4.1.2/2.4.2.2Data Checks

# of Students Served

Neglected Programs

Juvenile Detention


276 556


Juvenile Detention


10 25

Asian 20 25


90 150

Hispanic or Latino 50 100


20 100

White 90 150

Two or More Races 8 6

Total 278 556

SexNeglected Programs

Juvenile Detention

Male 200 354

Female 76 200

Total 276 554

35

2.4.1.2/2.4.2.2Data Checks

# of Students Served

Neglected Programs

Juvenile Detention


276 556


Juvenile Detention


10 25

Asian 20 25


90 150

Hispanic or Latino 50 100


20 100

White 90 150

Two or More Races 8 6

Total 278 556

SexNeglected Programs

Juvenile Detention

Male 200 354

Female 76 200

Total 276 554

36

2.4.1.3.1/2.4.2.3.1Transition Services

In the first row of the table below, indicate whether programs/facilities receiving Subpart 1 funds within the State are legally permitted to track student outcomes after leaving the program or facility by entering Yes or No. In the second row, provide the unduplicated count of students receiving transition services that specifically target planning for further schooling and/or employment. If not, provide more information in the comment field.

FAQ on facilities collecting data on student outcomes after exit: If only some, but not all, facilities in the State can collect data on student outcomes after exit, enter “yes” for the first question and provide a comment indicating why some facilities are unable to collect these data.

Neglected Programs

Juvenile Detention


Adult Corrections/

At-RiskOther

Programs

Are facilities in your State permitted to collect data on student outcomes after exit? (Yes or No)

Number of students receiving transition services that address further schooling and/or employment

37


Are facilities legally permitted to collect data after exit?

What to submit in table 2.4.1.3.1/2.4.2.3.1

Yes, all facilities are permitted to collect data after exit.

Yes

Some, but not all of the facilities in my State are permitted to collect data after exit.

YesAlso provide a comment indicating why some facilities are unable to

collect these data.

No, none of the facilities are permitted to collect data after exit.

No

38


• If facilities are legally permitted to collect student outcome data after exit (as indicated by a “yes” in table 2.4.1.3.1/2.4.2.3.1), but lack the resources or capacity to collect the data (as indicated by zeros/blanks in table 2.4.1.3.2/2.4.2.3.2), please explain via comment.

• The leading indicators for Part D include two transition- related indicators: – Percentage of students who enrolled in a local district

school after exit from a Part D program.– Percentage of students served who earned high school

course credits up to 90 days after exit.

39

2.4.1.3.1/2.4.2.3.1Data Checks


Juvenile Detention


276 556

Neglected Programs

Juvenile Detention


Yes 0

Number of students receiving transition services that address further schooling and/or employment.

279 556

40

2.4.1.3.1/2.4.2.3.1Data Checks


Juvenile Detention


276 556

Neglected Programs

Juvenile Detention


Yes 0

Number of students receiving transition services that address further schooling and/or employment.

279 556

41

2.4.1.3.2/2.4.2.3.2Academic and Vocational Outcomes (1)

The table of academic and vocational outcomes is reorganized into three smaller tables to group outcomes by (1) the setting in which the outcomes are achieved (in facility vs. out of facility) and (2) how many times students can achieve them. The instructions for the tables have been altered to reflect the new groupings:

The first table includes outcomes a student can achieve only after exit. In this table, provide the unduplicated number of students who enrolled, or planned to enroll, in their local district school within 90 calendar days after exiting. A student may be reported only once, per program type.

Outcomes(once per student,

only after exit)Neglected Programs

Juvenile Detention


Adult Corrections/

At-RiskOther

Programs

# of Students Who 90 daysafter exit

90 daysafter exit

90 daysafter exit

90 daysafter exit

90 daysafter exit

Enrolled in their local district school

42


The second table includes outcomes a student can achieve only one time. In this table, provide the unduplicated number of students who attained the listed outcomes either while enrolled in the LEA program/facility column (“in fac.”) or in the “90 days after exit” column. A student may be reported only once across the two time periods, per program type.

Outcomes(once per student)

Neglected Programs

Neglected Programs

Juvenile Detention

Juvenile Detention





Other Programs

Other Programs

# of Students Who In fac. 90 days after exit

In fac.

90 days after exit In fac. 90 days

after exit In fac. 90 days after exit In fac. 90 days

after exit

Earned a GED

Obtained high school diploma

43


The third table includes outcomes that a student may achieve more than once. In the “In fac.” column, provide the unduplicated number of students who attained academic and vocational outcomes while enrolled in the LEA program/facility. In the “90 days after exit” column, provide the unduplicated number of students who attained academic and vocational outcomes within 90 calendar days after exiting. If a student attained an outcome once in the program/facility and once during the 90-day transition period, that student may be reported once in each column.

Outcomes(once per student per time period)

Neglected Programs

Neglected Programs

Juvenile Detention

Juvenile Detention





Other Programs

Other Programs

# of Students Who In fac. 90 days after exit

In fac.

90 days after exit In fac. 90 days

after exit In fac. 90 days after exit In fac. 90 days

after exit

Earned high school course credits Enrolled in a GED program Accepted and/or enrolled into postsecondary education

Enrolled in job-training courses/programs

44

2.4.1.3.2/2.4.2.3.2Data Checks


Juvenile Detention


276 556


Neglected Programs

Neglected Programs

Juvenile Detention

Juvenile Detention

# of Students Who In facility 90 days after exit In facility 90 days after

exit

Earned a GED 78 200 16 82Obtained high school diploma

14 78 13 32

45

2.4.1.3.2/2.4.2.3.2Data Checks


Juvenile Detention


276 556


Neglected Programs

Neglected Programs

Juvenile Detention

Juvenile Detention

# of Students Who In facility 90 days after exit In facility 90 days after

exit

Earned a GED 78 200 16 82Obtained high school diploma

14 78 13 32

46

2.4.1.6.1/2.4.2.6.1 and 2.4.1.6.2/2.4.2.6.2Academic Performance in Reading and Mathematics

Performance Data(Based on most recent

pre-/posttest data)Neglected Programs

Juvenile Detention


Adult Corrections/ At-Risk

Other Programs

Long-term students with negative grade-level change from the pre- to posttest exams (optional for At-Risk)

Long-term students with no change in grade level from the pre- to posttest exams

(optional for At-Risk)

Long-term students with improvement up to one full grade level from the pre- to posttest exams


Long-term students with improvement of more than one full grade level from the pre- to posttest exams


47

Example From California:Where are we and what do we want to accomplish?

Purpose of Title I, Part D Funds• To provide opportunity for youth to meet the same

academic content standards as other students• To operate high quality programs to prepare youth for

secondary school completion, training, employment, or further education

• To provide activities to facilitate the transition of youth from correctional facilities to further education or employment

• To operate programs in local schools for youth returning from correctional facilities; and programs which may serve at-risk youth

48

Example From California:How are we doing?

Educational programs of the correctional facility help students meet state academic content standards.

At least once every three years, the agency evaluates the Neglected or Delinquent Program, disaggregating data by gender, race, ethnicity, and age, to determine its effectiveness with students’:• Ability to maintain and improve educational achievement. • Completion of secondary school requirements and ability to obtain employment. • Accrual of credits toward promotion and graduation. • Transition to a regular program or other education program. • Participation in postsecondary education and job training programs as

appropriate.

The agency uses multiple and appropriate measures of student progress in evaluating Neglected or Delinquent Programs.

The agency uses evaluation results to improve programs.

49

2.4.1.6.1/2.4.2.6.1 and 2.4.1.6.2/2.4.2.6.2Data Checks

# of Students Served Neglected Programs Juvenile Detention

Long-Term Students Served 8 4

Performance Data(Based on Most Recent Pre-/Posttest Data)

Neglected Programs

Juvenile Detention

Long-term students with negative grade-level change from the pre- to posttest exams

1 1


1 1


3 2


3 2

Total 8 6

50

2.4.1.6.1/2.4.2.6.1 and 2.4.1.6.2/2.4.2.6.2Data Checks

# of Students Served Neglected Programs Juvenile Detention

Long-Term Students Served 8 4

Performance Data(Based on Most Recent Pre-/Posttest Data)

Neglected Programs

Juvenile Detention

Long-term students with negative grade-level change from the pre- to posttest exams

1 1


1 1


3 2


3 2

Total 8 6

51

State Data Collection Tools and CSPR Timelines in States

52

State Data Collection Tools

Polling Question: How does your SEA collect data from its subgrantees?• Word document• Excel spreadsheet (no rollup feature)• Excel spreadsheets with rollup feature• Survey tool (Survey Monkey, SurveyGizmo, etc.)• Customized online tool • Other

53

State Data Collection Tools

Polling Question: How does your SEA conduct data quality checks?• The Title I, Part D Coordinator conducts data quality

checks “by hand” (e.g., eyeballs it, does rough calculations, recreates Excel equations each year).

• Data quality checks are programmed to check data once they are compiled across state.

• Data quality checks are programmed to pop up when each subgrantee submits data.

• Other.• I’m not sure how the data quality checks are done.

55

State CSPR Data Collection Timeline

• Polling Question: When does the SEA collect data from the subgrantees?

1 Topical Call Series: Improving Data Quality and Use CSPR Data Collection Tuesday, September 15,...

Documents

Transcript of 1 Topical Call Series: Improving Data Quality and Use CSPR Data Collection Tuesday, September 15,...