Youth Performance Accountability

Data Accuracy and Reporting for WIA and WIOA*

CWA ConferenceLong Beach, CAJanuary 27, 2015

*As known today1

Objective

Mutual Understanding of Data Collection-Entry-Reporting accountability from Local Areas to State to DOL.

Encourage discussion of data collection and reporting requirements, procedures, and guidance.

Establish open forums for communication for technical assistance.

2

Background Oversight agencies like GAO and OIG

cite data quality issues with ETA’s data (2002 and 2008)

Guidance issued annually containing report submission deadlines and source documentation requirementsTEGL 6-14 for PY13/FY14 Reporting and

Data Validation

3

Policies/Procedures and Policies/Procedures and TrainingTraining

Data management and the resultant quality of reported data are derived from and influenced by the policies, procedures and protocols utilized at the state and/or local levels

Grantees should develop guidance for staff and sub-grantees involved in the collection of data:

Definitions of data elements Sources of information Participant record and documentation requirements Procedures for collecting, entering and reporting data and

associated “business rules” that cover timeliness and completeness

Procedures for entering data into an automated database Procedures for correcting data

4

Training and MonitoringTraining and Monitoring Data collection and data entry:

Routine training should be provided for data management guidance

All staff involved in the collection or entry of data should be trained in the procedures

The data entry process should include steps for verifying entered data against original sources on a sample basis or for entire population of records

5

REPORTING

If Data is not collected and reported, it didn’t happen!!

6

Performance Performance AccountabilityAccountability

*unless State ‘early implementer’*unless State ‘early implementer’

WIA (Workforce Investment Act) effective through PY15

‘Services’ based Participation and Exit

Data Validation required

Reporting Cohort primarily 1st to 3rd Qtr after Exit

Nine Common Measures

Reporting Participant Information

Sequence of Services – Core, Intensive, Training

WIOA (Workforce Investment and Opportunity Act) effective PY16*

‘Services’ based Participation and Exit

Data Validation codifiedReporting Cohort

extended 2nd to 4th Qtr after Exit

Twelve Primary Indicators of Performance

Expanded Reporting Participant Information

‘Career Services’ and Training

7

WIOA Youth Program Design

Focus on Out of School Youth75% of formula youth funds minimum

Work Experience Expenditure20% of local area funds minimum

Eligible Out of School Youth Age16-24 year olds

New Program Elements

8

WIOA Youth Program Transition Youth PY 2015 Funding

April 1, 2015 WIOA Implementation

July 1, 2015

While WIOA is effective July 1, 2015 and the Youth funds allotments are provided April 1, 2015, and it will take some time to transition, States and Local Areas must begin to incorporate strategies for recruiting and serving more OSY with the receipt of PY 2015 funds. Progress must be documented, including plans to achieve 75% expenditure rate with future funds.

9

Participation

10

Participant An individual determined eligible to

participate in the program who receives a service funded by the program in either a physical location (e.g., One-Stop Center) or remotely through electronic technologies.

Three ComponentsDetermined eligible to participate in the

programReceives a funded service In either a physical location or

through electronic technologies11

Components of Components of ParticipantParticipant Individual determined eligible to participate

Depends on program/funding; doesn’t apply in the case of W-P, which is based on universal access

Receives a service Not all services trigger participation; it’s important to

understand the distinction between those that do and those that don’t

In a physical location or remotely Many substantial services are remotely accessed; this

needs to be captured

12

Multiple Program Participation

Counting Participants in Multiple Programs

Earliest date of service Can participate in several programs

simultaneously Counted as a participant in each of those

programsThe participant won’t exit from the program

unless there is a gap of no service for 90 days

13

WIOA Youth Program Participant When does participation begin? What services commence participation

and inclusion in performance accountability?

14

A Service Is:

Any core, intensive or training activity

made available to eligible participants that

allows them to benefit from specific

programs in the workforce system.

15

When a Service is included in Performance?

Core, intensive or training services made available to eligible participants and require significant staff involvement who exit the program.These aforementioned individuals are included

in the performance measuresThose WIA Adult and DW program participants

who only receive self service or informational activities are excluded from performance

16

Services that Do Not Begin or Extend Participation Eligibility determination Case management administrative

activities to obtain information regarding employment status, educational progress, need for additional services, etc.

Income maintenance or Support payments Visitors to One Stop Centers, etc., for

reasons other than its intended purposes Follow-Up Services

17

Exit

18

Exiter A participant who hasn’t received a

program or partner-funded service for 90 consecutive days and no future services are scheduled

Three componentsHasn’t received a serviceFor 90 consecutive daysNo future services scheduled

19

Components of Exiter The participant hasn’t received a service

Could be program- or partner-funded

For 90 consecutive calendar days A gap in service can stop the 90-day clock if based on

specific/allowable circumstances

No future services scheduled Specific services and activities as allowable Does not include any follow-up services or

circumstances where the participant voluntarily withdraws or drops out of the program

20

Illustration: Participation and Exit

Participation No Services or Follow-Up Services

Exit DateParticipation

Date

Eligible and Receives Service

End of 90-Day Period

Last Service

21

Non-Compliance with EXIT Requirements

Exit dates not reflective of dates of last service

‘Case management’ used to extend exit date Hard exits utilized

Date of last contact = Exit dateDate of employment = Exit date

Services provided within 90 days Lack of common exit date (across core

workforce programs) Exit dates not consistent with dates in MIS

22

How to Prevent Unintentional Exits: Gaps in Service

Three allowable circumstances - the condition must exist for at least 90 days

Delay before beginning of training Health/medical condition of participant

or family member Temporary move from the area that

prevents participation23

Exclusions

Institutionalized Health/Medical or Family Care Deceased Reserved Forces called to Active Duty Relocated to Mandated Program (Youth) Invalid/Missing Social Security Number

24

WIOA Youth Program Exit

When is it appropriate to exit Youth participants?

Common or Program Exit?

25

Follow Up Services Follow-up begins after the expected last

service

Youth are required to receive at least 12 months of follow-up services, which are triggered at exit (the only exclusion is for summer youth employment)

Not intended to take responsibility away from case managers for WIA. Case managers do not have to wait 90 days, for instance, to begin providing follow-up services. 26

Total Youth Common Measures

27

PLACEMENT IN EMPLOYMENT OR EDUCATION

Employed, in the military or in post-

secondary education at participation?

Has a qualifying outcome* in the 1st quarter after exit?

NO

YES

Excluded

YES

YES

Numerator

Denominator

NO

Qualifying Outcomes: employment, military, enrolled in post-secondary education, advanced training, or occupational skills training 28

Placement in Employment or Education Reporting Time periods

For the Quarter ending Sept 2014

20132013 20142014

Jan-Jan-MarMar

Apr-Apr-JuneJune

July-July-SeptSept

Oct-Oct-DecDec

Jan-Jan-MarMar

Apr-Apr-JuneJune

July-July-SeptSept

Oct-Oct-DecDec

Exit CohortExit Cohort            

   Employment/Education QtrEmployment/Education Qtr         

         2 Qtr Wage Delay2 Qtr Wage Delay   

                    

Nov Nov 15th15th

29

LITERACY/NUMERACY GAINS

Definition: Of those out-of-school youth who are basic skills deficient, the percentage who increase one or more educational functioning levels within one year of participation

30

Literacy/Numeracy DefinitionsBasic Skills Deficient

The youth computes or solves problems, reads, writes, or speaks English at or below the eighth grade level or is unable to perform these tasks at a level necessary to function on the job, in the individual’s family, or in society.

States and grantees may develop their own definition, but it must include the language above.

31

Definitions (Continued)Out-of-School Youth

High School Dropout (no diploma or equivalent)

High School Graduate (or equivalency) not in postsecondary

High School Graduate (or equivalency) in postsecondary – but basic skills deficient

32

About the Assessments All out-of-school youth must be assessed in basic reading, writing and math

Pre-testing must occur within 60 days of the first youth program service; can use pre-test from up to six months prior to date of first youth service

The same standardized assessment must be used for pre- and post-testing

Youth should be post-tested by the end of one year of participation and compared to pre-test results obtained during initial assessment

33

About the Assessments If a youth continues to be basic skills deficient after the first 12 months of participation, they should continue to receive training in literacy and/or numeracy skills

Youth should be post-tested and included in the measure at the completion of the 2nd year only if they complete two full years in the program

34

Literacy/Numeracy Gains - 1st Year of Participation Of those out-of-school youth who are basic skills deficient, the percentage who increase one or more educational functioning levels within one year of participation

Number of youth participants who increase one or more educational functioning levels

Number of youth who completed a year of

participation (based on date of 1st youth service)

Number of youth who exit before completing a year of participation

+35

Literacy/Numeracy Gains2nd and 3rd Years of Participation Of those out-of-school youth who are basic skills deficient, the percentage who increase one or more educational functioning levels within one year of participation

Number of youth participants who increase one or more educational functioning levels

Number of youth who completed a second or third full year of participation (based on anniversary date of 1st

youth service) 36

ATTAINMENT OF DEGREE OR CERTIFICATE

Enrolled in education at participation or any

time during participation?

Attained diploma, GED or certificate by the end of

the 3rd quarter after exit?

YES

NO Excluded

YES

YES

Numerator

DenominatorNO

37

Credentials Defined

Awarded in recognition of an individual’s attainment of measurable technical or occupational skills necessary to obtain employment or advance within an occupation.

Work Readiness Credentials and those awarded by workforce investment boards are NOT included in this definition.

38

Credentials

TEGL 15-10 established ‘credential’ as the umbrella term encompassing postsecondary degrees, diplomas, licenses, certificates, and certifications. All states, regardless of the common measure waiver, must collect information on credential attainment in the WIASRD

39

Credentials: Approved Awarding Institutions

A state educational agency, or a state agency responsible for administering vocational and technical education within a state Institution of higher education (including community colleges) A professional, industry, or employer organization or a product manufacturer or developer A registered apprenticeship program A public regulatory agency (e.g., FAA aviation mechanic certification) A program approved by DVA (Veterans’ Affairs) to offer education and training to veterans and other eligible persons under the Montgomery GI Bill Office of Job Corps Institutions of higher education which are formally controlled, or formally sanctioned or chartered by the governing body of an Indian tribe(s)

40

WIOA Youth Performance Measures Education/Employment

2nd quarter after exit

Education/Employment 4th quarter after exit

Earnings Median earnings 2nd quarter after exit

Credential Rate Up to one year after exit If HS Dip/GED, must include Employment/Postsecondary

In-Program Skills Gain Achieving measurable skills gains

Employer Effectiveness before PY16

41

What’s Eliminated Literacy/Numeracy indicator for youth

Although utilized in development of Skills Gain measure

Customer Satisfaction as statutory measure

State Incentive Funds

But Governor’s reserve may be used for local incentives

42

Additional Provisions State Targets

Must use statistical adjustment model—use now codified (Sec. 116(b)(3)(A)(viii))

Targets for first two years included in State Plans

Additional Information required in Annual Reports Example: Amount of funds spent on each type of service Data Validation now codified (Sec. 116(d)(5))

43

Additional Provisions Sanctions

State Level If a state fails performance, Secretaries shall provide TA

(used to say will provide TA upon request) If a state fails for 2nd consecutive year or fails to submit

their Annual Report, it can lead to a reduction in statewide funds (stronger language)

Local Level If failure continues for a 3rd consecutive year, the

Governor must take corrective action which shall include development of a reorganization plan (and new local board)

44

Eligibility and Data Validation Documentation

45

Data Validation and Eligibility States have the burden of identifying

required Program Eligibility documents, or have given that responsibility to the Local Area.

While not required, many States use the Data Validation Source Documentation Guide as the required documents listing for Program Eligibility.

46

Source Documentation

Whether scanned, paper, or system cross-match, the purpose of source documentation is to have an auditable trail that documents the participant, services delivered and outcomes received.

47

Common Source Documentation Issues

Using documents that are not on ETA’s list of Acceptable Source Documents;

Checkbox in MIS; Date of Birth; Youth who Needs Additional Assistance; Failure to Accurately record ‘Dates’; and Poor Case Notes.

48

Questions?

49