ED 300 645

AUTHORTITLE

INSTITUTION

SPONS AGENCY

PUB DATENOTEPUB TYPE

EDRS PRICEDESCRIPTORS

IDENTIFIERS

ABSTRACT

DOCUMENT RESUME

CE 051 320

Dickinson, Katherine P.; And OthersEvaluation of the Effects of JTPA PerformanceStandards on Clients, Services, and Costs. ExecutiveSummary. Research Report No. 88-15.Berkeley Planning Associates, Calif.; SRIInternational, Menlo Park, Calif.National Commission for Employment Policy (DOL),Washington, D.C.Sep 8835p.; For the full report, see CE 051 321-322.Reports - Evaluative/Feasibility (142)

MF01/PCO2 Plus Postage.*Client Characteristics (Human Services); EducationalPolicy; *Employment Services; *Evaluation Criteria;Federal Government; Government Role; GovernmentSchool Relationship; Outcomes of Education; *ProgramCosts; Public Policy; *State Standards*Job Training Partnership Act 1982; *PerformanceStandards

A study examined whether the state and federalperformance standards fpr Job Training Partnership Act (JTPA)programs have influenced Service Delive:y Area (SDA) and serviceprovider decisions about program design and implementation practicesin ways that have affected the clients served, the types of servicesprovided, and the costs of the services. Only a few SDAs appeared togive priority to their performance goals at the expense of otherprogram objectives. It was only in those SDAs without well-definedclient or service goals that performance standards had notableunintended effects. A great deal of variation was found in the waystates have chosen to use their authority to establish JTPAperformance standard policies and to set priorities for the types ofclients served. States that established target groups for JTPAprograms did influence SDAs to enroll more hard-to-serve clients.There was no evidence to suggest that federal standards for theentered employment rate and wage rate for adults generally hadunintended effects on clients or services. On the other hand, federalcost standards did have a number of unintended effects and were theleast comparably measured of all federal performance measures.(MN)

******************************************g***************************** Reproductions supplied by EDRS are the best that can be made *

* from the original document. *

x**********************************************************************

Members of the National Commission for Employment Policy

GERTRUDE C. McDONALD, ChairmanFremont, California

LEORA G. DAYBoise, Idaho

HENRY A. DUFFYLithonia, Georgia

JOHN GARTLANDPotomac, Maryland

VIRGINIA S. MILNERBeverly Hills, California

JERRY NAYLORAgoura, California

A. WAYNE ROBERTSBurlington, Vermont

MAX HUGEL JOHN A. ROCCOWindham, New Hampshire Cherry Hill, New Jersey

DONALD W. JONES WILLIAM M. TAYLORSpringfield, Missouri Jacksonville, Florida

J. MICHAEL LEVESQUEProvidence, Rhode Island

FRANK D. McDONALDThousand Oaks, California

JAMES "Vf . WINCHESTERPass Christian, Mississippi

Section 106(0 of the Job Training Partnership Act states "The National Commission forEmployment Policy shall ... evaluate the usefulness of ... standards as measures of desiredperformance, and evaluate the impact of ... standards (intended or otherwise) on the choice ofwho is served, what services are provided, and the cost of such services in service deliveryareas."

EVALUATION OF THE EFFECTS OF JTPAPERFORMANCE STANDARDS ON CLIENTS,SERVICES, AND COSTS

Executive Summary

Prepared by:

SRT InternationalKatherine P. DickinsonRichard W. West

Berkeley Planning AssociatesDeborah J. KoganDavid A. DruryMarlene S. FranksLaura SchlichtmannMary Vencill

Research Report No. 88-15

September 1988

National Commission for Employment Policy1522 K Street N.W., Suite 300, Washington, D.C. 20005

4

This report was prepared under contract to the National Commission for EmploymentPolicy. Any opinions or conclusions are those of the authors and do not necessarily represent theopinions or policies of the Commission or any other agency of the United States Government.

5

.11

NATIONAL COMMISSION FOR EMPLOYMENT POLICY1522 K Street, NW, Suite 300

Washington, D.C. 20005

(202) 724-1545

September 1988

TO THE PRESIDENT AND THE CONGRESS OF THE UNITED STATES

The National Commission for Employment Policy (NCEP) is pleased to present to you thisExecutive Summary of our major research study entitled, Evaluation of the Effect of JTPA Per-formance Standards on Clients, Services, and Cost. This is the first national evaluation ofacornerstone of the Job Training Partnership Act (JTPA)the implementation of clear,measurable program goats that exemplify "the return on investment" strategyupon which the en-tire program is built. In fact, Congress itself believed that performance standards were of suchimportance that it mandated an evaluation of their impact in the enabling legislation. Section106(0(2) of the Act directs the Commission to "evaluate the impact of such standards (intendedor otherwise) on the choice of who is served, what services are provided, and the cost of suchservice in service delivery areas."

I believe this report is one of the major contributions of my tenure as Chairman of the Com-mission. Contractors working on the study, SRI International and Berkeley Planning Associates,are to be commended for their superb job of gathering and analyzing the diverse and compli-cated data. As one of our reviewers notes "this is a tremendously valuable piece of work, break-ing new ground in national studies." I am confident you will agree that the findings detailed arevery useful, having implications that go well beyond the performance management system andbeyond JTPA. Many of the findings can be helpful when enacting legislations for other humanresource programs.

The Commission believes that programmatic performance standards have been a successfulstrategy to focus the management of JTPA toward the goals of the legislation finding produc-tive employment for disadvantaged people. This not to say that no adjustments are needed to im-prove the performance management system; however, we feel that, overall, the performancemanagement system is having its intended effect of guiding JTPA resources toward meaningfulinvestment in the Nation's most precious resourceits human capital.

At this time, I am transmitting only the Executive Summary to you. It contains an overviewof the study, major findings, and of the specific findings. In addition, the Commission ispleased to have the opportunity to present you both general and specific recommendations thatwe feel would make the performance management system even more valuable to the Nation'slargest job training system. Should you, or your staff, be interested in the full report, I would behappy to forward it to you.

Sincerely,

c tfd, e eGertrude C. McDonP11Chairman

CONTENTS

Preface vii

Acknowledgments ix

Role of Performance Standards in the JTPA Title II-A Program 1

The Federal Role in Performance Standards 2The State Rolein Performance Standards 2The SDA and PIC Role in Performance Standards 3

Objectives of the Study and Key Policy Questions 3

Major Findings 4

Ucal Response to Performance Standards 4Effects of State Policies 5

Effects of Federal Policies 5Summary 6

Detailed Findings 6

Overview of Evaluation Design 6Results from the Quantitative Analysis 7Results from the Qualitative Analysis 11

Effects on Local Program Objectives 11

Effects on Client Goals 12Effects on Program Services and Costs 12Effects on Management Practices 14

Implications for the Performance-Standards System 15

Federal Policies 15

Choice of Performance Measures 15

Cost Measures 15Youth Measures 16

Level of Standards 16DOL Adjustment Models 16Technical Assistance 16Reporting 16

v

State Policies 17

State Performance Standards 17Additional Adjustment Procedures for the Federal

Standards 17Hard-to-Serve Policies 18Incentive Policies 18S ar.c don Policies 18Technical Assistalce 18

Conclusions 18

Recommendations 21

vi

PREFACE

This Executive Summary is a departure from the traditional abstract of research reportpublished by the National Commission for Employment Policy. This extended Executive Sum-mary is provided for the reader who may not have the time or inclination to review the entirereport with its appendices.

The Executive Summary is divided into six sections. The first section provides an overviewof the role of performance standards in JTPA. Next, the paper presents the objectives of thestudy. Major findings are summarized in the third section with detailed findings presented in thefourth part of the summary. The fifth section includes the implications of the findings from theperspective of the authors of the report. Finally, the recommendations of the Commission forchanges to the performance management system are presented.

The Commission hopes that the Executive Summary provides a flavor of the valuable infor-mation contained in the complete report. Many individuals interested in JTPA and the effect ofperformance standards will also want to read the full report, which contains extensive informa-tion from the case studies and the quantifiable factors shown to influence who is served inJTPA, the type of services received, and the cost of such services. The full report and the appen-dices are available free of charge from the Commission.

vii

ACKNOWLEDGMENTS

The National Commission for Employment Policy wishes to thank the many individualswho contributed to this report. First, we are grateful to the the Service Delivery Area (SDA)Directors and their staff who completed the lengthy director's questionnaire and the fiscal ques-tionnaire, without which the thorough analysis would not have been possible. State staff werealso exceptionally cooperative in furnishingour contractors with copies of their policies and pro-cedures relating to their performance management systems, enabling us to obtain informationfrom all 50 States.

Special appreciation must go to the performancemanagement staff in the States of Califor-nia, Maryland, Missouri, Massachusetts, Missouri, New Mexico, New York, North Carolina,and Ohio and to the State Job Training Coordinating Council staff in those States for spendingtime with our contractors during the site interviews. Although we agreed not to identify the SDAstaff, Private Industry Council members, and local service providers who were interviewed forthis report to maintain their anonymity, we owe them particular gratitude for allowing our con-tractors to obtain so much useful information from them about the way JTPA operates in theirareas and the manner in which performance standards impacton their programs.

The Commission would like to acknowledge a very able expert panel who reviewed the ini-tial design of the evaluation as well as made extensive, thoughtful comments on the final report.John Raisian of the Hoover Institute, Carl Simpson of Western Washington University, andJohn Wallace of the Manpower Demonstration Research Corporation spent numerous hoursproviding assistance both at the beginning and end of the project. Their comments were insight-ful and enabled the final product to be both technically sound and useful for both JTPAevaluators and practitioners.

Staff of the Department of Labor were very cooperative in ensuring that we had access tovarious data sources and in providing comments on the final report. Karen Greene and her staffwithin the Division of Adult and Youth Performance Standards deserve special thanks for theirassistance.

Finally, special appreciation to the Pinellas County Private Industry Council in Clearwater,Florida for the use of their logo for the cover of this report.

ixl 0

Executive Summary

Role of Performance Standardsin the JTPA Title II-A Program

Performance standards are an integralpart of the Job Training Partnership Act(JTPA) programs operated under Title II-A.The standards implemented during the first 5years of program operation [Transition Year1984 through Program Year 1987(PY 87)]were intended to further a variety of Federalprogram priorities. These included

To hold service delivery areas(SDAs) accountable for programoutcomes,

To encourage the achievement ofquality employment outcomes,

To encourage the achievement of cost-effective outcomes,

To create an incentive for effectivemanagement by local program ad-ministrators, and

To foster acceptance of the programby the business community.

The focus on accountability and the con-

cern with outcome measuresrather thanprocess measureswere new to employmentand training programs. The performancestandards grew out of the legislative goal of"measuring the return on the JTPA programinvestment" in terms of increases in theemployment and earnings of economicallydisadvantaged individuals and reductions inwelfare dependency. Although not intendedto be an accurate measure of net program im-pacts, the performance standards are used toindicate the extent to which individual SDAsare managing their resources in order toachieve important program outcomes.

Starting in PY 88, several additional per-fonntince standards have been implemented,including measures of the participants' status3 months after leaving the JTPA program.The new post-program standards are intendedto provide greater emphasis on the objectiveof improving job retention and longer termemployability of participants than do the cur-rent measures, which measure the immediateemployment status of participants on leavingthe program.

The Federal Role inPerformance StandardsAlthough the performance standards are

one expression of Federal goals for the JTPAprograms, the performance-standards systemhas been carefully designed to minimize theday-to-day Federal role in local programdirection and admini tration. The JTPA sys-tem is highly decentralized, enabling localPrivate Industry Councils (PICs) and SDAsto design programs relevant to the needs andemployment opportunities in their localareas. In particular, the local agencies aregiven wide discretion in the types of programservices provided and the types of clientsserved.

The Federal role in influencing therecruitment and selection of JTPA clients islimited to several requirements in the legisla-tion: (1) at least 90% of the enrollees beeconomically disadvantaged, (2) recipientsof Aid to Families with Dependent Children(AFDC) and high school dropouts receivean equitable share of 'TPA services, and (3)40% of all JTPA service dollars be expendedfor services to youth.

The Federal role in determining the typesof program services provided is even morelimited, allowing SDAs substantial discretionin designing their JTPA service mix. Al-though public service employment is not anallowab't service and expenditures on sup-portive services are restricted (limiting theability of SDAs to offer stipends to class-room training participants), a broad range ofservices is permitted by the JTPA legislationand Federal program regulations.

Consistent with he limited Federal rolein designing the JTPA program, the intentionof Federal performance-standards policies ist,o foster accountability and cost-effective-ness without undue influence on SDA designdecisi-.,n5: The Federal performance-stand-

-ds policies include the choice of measures,numerical level at which the standards.C.A, and the provision of optional models

2

for adjusting standards.Careful effort went into choosing Fedelal

performance measures that would not con-strain SDAs in their decisions about whom toserve and Ciat services to offer. The numeri-cal level of most Federal standards is set at a"minimally acceptable" level so that SDAscould generally meet their standards withoutdistorting their program designs. The adjust-ment models developed for the Federal stand-ards are intended to ensure the "neutrality" ofthe standards with respect to decisions aboutclient targeting and to "hold harmless" SDAsthat choose to serve mere difficult clientsthrough downward adjaitments to thenumerical levels of their standards.

The State :tole inPerformance StandardsStates can play a very visible and impor-

tant role in the JTPA program as a whole andin the performance-standards system in par-ticular, S ;ciflcally, States are responsiblefor setting the levels of standards for SDAs.States decide whether or not to use theDepartment of Labor's (DOL) adjustmentmodel to set SDA standards and make adjust-ments beyond the model to those standards.States may also develop and implement addi-tional performance standards to further Stateprogram goals and priorities and may decidehow to weight the different State and Federalstandards in making incentive awards. Statesalso determine what portion, of the 6% fundswill be used for technical assistance toSDAs, what port:,,n will be used for incen-tive awards to SDAs that exceed the stand-ards, and what portion will be used forincentives to serve the hard to serve. Statesdetermine whether the incentive awards willencourage performance that just, exceeds thestandard level or whether there will be astrong financial incentive for SDAs to per-form at the highest level possible. Fiolly,States determine policies to sanction SDAsthat fail to meet their standards.

Although the Federal performance-stand-

ards system was designed to encourage SDAdiscretion about who to serve and what ser-vices to offer, State governments are en-couraged to use the performance-standardssystem to further State client and servicepriorities Some States have welcomed theopportunity to take a leadership role in theseareas, while other States have decided to playa less active role and defer to the preferencesof local SDAs.

The SDA and NC Role inPerformance StandardsSDAs, with guidance from PICs, have the

responsibility for actually implementingJTPA programs and for making majorprogram-design decisions about service andclient mix. In making these decisions, SDAstry to balance the influence of local goals,their assessment of local conditions, the per-formance-standards incentives offered by theState, and the State and Federal programregulations. SDAs are also motivated to per-form well on the performance standards for anumber of reasons, including the desire to (1)run cost-effective programs that achieve high-quality outcomes, (2) appear to be account-able to local elected officials and privatesector representatives on the PIC, (3) avoidreorganization as a consequence of failing toperform at acceptable levels for 2 years insuccession, and (4) receive incentive awards.

SDA and PICs may respond to perfor-mance-standards incentives in three key areasof program design: (1) decisions about thetypes of clients to serve and procedures torecruit and enroll clients; (2) decisions aboutthe types of services to provide and thelength of those services; and (3) decisionsabout how to provide services, including th..:choices of service providers, type of contract,and contract terms.

Objectives of the Study andKey Policy Questions

The National Commission for Employ-ment Policy (NCEP) has funded a corn-

3

prehensive evaluation of several important is-sues, in keeping with its legislative mandateto advise the Secretary of Labor on thedevelopment of the JTPA performance stand-ards and to evaluate the impact of standardson the choice of who is served, what servicesare provided, and the cost of such servicesat the local SDA level (Section 106 (f)). Theoverall goal of dr; study is to assess whetherthe performance standards have influencedSDA and service-provider decisions aboutprogram design and implementation practicesin ways that have affected the clients served,the types of services provided, and the costsof the services.

The study is not a detailed assessmentof whether the performance standards haveresulted in an increase in JTPA programproductivity and cost-effectiveness. Rather,the study's purpose is to test whether the in-centives created by the performance-stand-ards system have caused SDAs to emphasizemeasured performance objectives at the ex-pense of unmeasured or unrewarded objec-tives that are also valued. In particular, thegoal is to determine whether emphasis on theperformance standards as a vehicle forprogram management has led SDAs to avoidserving more difficult clients or to reduce theintensity of services offered, which in turnmay have affected the types of participantswho could benefit from the program.

The evaluation was intended both todetermine the effects of performance-stand-ards policies and to investigate how those ef-fects come about. Thus, the evaluation wasdesigned to meet several goals. First, thestudy was designed to assess the effect of theoverall Federal performance-standards sys-tem on SDA and service-provider behavior.Our examination of the Federal performance-standards system distinguished among the ef-fects of (1) the kinds of performancemeasures selected, (2) the level at which per-formance expectations were set, and (3) theadequacy of the adjustment model in holding

13

SDAs "harmless" for their client and servicechoices.

Second, the study was designed to assesshow various State performance-standardspolicies and procedures have affected SDAand service-provider behavior and how theseeffects have influenced the clients st rved, theservices provided, and the costs of the ser-vices. This evaluation distinguished the ef-fects of several aspects of State policies,including (1) choices about procedures to ad-just standards, (2) policies for serving hard-to-serve clients, and (3) policies for awardingincentives for good performance andsanctioning for poor performance payments.

When we found evidence that perfor-mance standards did appear to be influencingthe clients, services, or costs, the study wasdesigned to explore how and why those in-fluences occurred and whether the influenceswere intended (for example, causing SDAs toserve hard-to-serve clients and to offer ser-vices that will make a difference in the longrun) or unintended (for example, causingSDAs v .ve fewer hard-to-serve clients orto offer less intensive services).

Finally, the study was designed togenerate suggestions for refining the perfor-mance-standards system so that any un-desirable impacts of performance standardscan be prevented in the future.

Major FindingsLocal Response toPerformance StandardsThe JTPA legislation gives authority to

SDAs to design programs to meet localneeds. Consistent with this intent, this evalua-tion found dramatic differences among SDAsin the design and operation of JTPAprograms, and many of th?,se differences af-fected clients, services, and costs. Althoughperformance-standards policies did affectsome SDA design decisions, many elementsof local JTPA design reflected local condi-tions and constraints beyond the control of

4

the SDA. The local unemployment rate inparticular had a strong influence on the typeof clients in JTPA programs. In tight labormarkets, most of the clients interested inJTPA programs were those who had littlework experience or major barriers to employ-ment, often requiring basic skills remedia-tion. In contrast, in areas with highunemployment, individuals seeking JTPAservices generally had more job skills and re-quired help in retraining for new industries oroccupations. The local availability of serviceproviders and of alternative services also af-fected the types of services that were offeredthrough JTPA programs, resulting in notabledifferences between urban and rural SDAs.

Within the local context, we also foundsubstantial variation in program design thatreflected SDAs' choices of goals for theJTPA program. Generally, we found SDAsto be influenced by three different objectives:(1) commitment to serving specific types ofclients; (2) commitment to responding tolocal employer needs and interest; and (3)commitment to achieving specific levels onJTPA performance standards. The relativeemphasis that SDAs placed on these three ob-jectives varied a great deal.

To the extent that SDAs stressed clientobjectives, they tended to design program ser-vices appropriate to the needs of those clientsand to choose management practices to en-sure achievement of their client goals. To theextent that SDAs smssed employer needs,they gave priority to designing and im-plementing a service mix to train individualsfor the more highly skilled jobs and enrolledparticipants appropriate for that training.

In general, most client-oriented andemployer-oriented SDAs were able to meettheir performance goals without majorrevisions to their other goals. In conjunctionwith clearly identified client and servicegoals, performance standards appeared tohave their intended effects of increasing ef-ficiency and accountability.

r4

This evaluation found only a few SDAsthat gave priority to their performance goals(generally goals of maximizing incentiveawards) at the expense of other program ob-jectives. Only in those SDAs without well-defined client or service goals didperformance standards have notable unin-tended effects. In a vacuum, performancestandards can produce unintended effectsof rec ucing service to the hard to serve anddecrea ',ing the intensity of services.

Effects of State PoliciesThe JTPA legislation gives considerable

authority to the States, both to establish per-formance-standards policies and to setpriorities for the types of clients to be servedin JTPA programs. There is a great deal ofvariation in how States have chosen to usethat authority. This evaluation examined howSDAs responded to these differences in Statepolicies.

We found that States can provide an ef-fective leadership role in setting clientpriorities for JTPA programs. States that es-tablished target groups for JTPA programsdid influence SDAs to enroll more hard-to-serf,: clients. Often States used performance-standards policies to convey their clientpriorities, including setting State standardsfor service to target groups or using 6%funds for service to those groups. States thatused the optional DOL adjustment models,which adjust standards for the types of clientsserved, also influenced SDAs to enroll morehard-to-serve clients.

Some of the ways that States have usedtheir authority in establishing performance-standards policies have had unintended ef-fects. In particular, incentive policies thatemphasize exceeding standards rather thansimply meeting standards tend to reduce ser-vice to some hard-to-serve groups and reducethe amount of basic skills remediation ser-vices. Emphasis on exceeding standardsseems to have institutionalized sometimes in-tense competition among SDAs and con-

5

veyed the message that "cheaper is better."The effects of these policies, however, arenot large and affect SDA choices at themargin.

Effects of Federal PoliciesFederal performance-standards policies

are intended to be neutral with respect to thetypes of clients served. They are intended toguide SDAs to choose cost-effective servicesbut not to reduce local flexibility in designingservices appropriate to local needs. Thisevaluation found that the Federal standardsfor the entered-employment rate and wagerate for adults generally did not have unin-tended effects on clients or services. Further-more, no evidence showed that the welfareentered-employment-rate standard inhibitedservice to welfare recipients.

The Federal cost standards, however, hadthe most unintended effects and were theleast comparably measured of all the Federalperformance measures. This evaluation foundthat SDAs in States that placed more weighton the Federal cost standard tended to servefewer hard-to-serve clients and that SDAsconcerned about exceeding the cost standardstended to design less-intensive services. Atthe same time, this evaluation found seriousmeasurement problems with the cost stand-ards. We found large differences in the extentto which SDAs were leveraging JTPA funds,either by using funds from other programs tohelp fund JTPA Title II-A programs or byusing service providers that had alternativefunding sources. As a result, it is difficult tocompare the cost of services received byJTPA participants across SDAs.

There was also considerable confusionamong SDAs in how to manage youthprograms to meet all the youth standards. Be-cause the goals of in-school and out-of-school programs were so different, SDAs thatemphasized one type of service often found itdifficult to meet at least one youth standard.Thus, the Federal youth standards tended toconstrain SDA choices about serving in-

school versus out-of-school youth.

SummaryThis evaluation found that performance-

standards policies can influence SDAs'choices about clients, services, and costs.Some effects are the intended results of Stateleadership in setting priorities for the JTPAprogram; some effects are the unintendedresults of State and Federal policies that tendto reduce service to hard-to-serve groups orreduce the intensity of service. The unin-tended effects, however, are neither large norinevitable. Performance standards do notpreclude SDAs from enrolling hard 0-serveclients or from providing intensive servicesbut do affect some SDA tradeoffs at the mar-gin. Furthermore, there is substantial varia-tion in how SDAs choose to react tostandards. Most SDAs can meet their perfor-mance goals without major revision to theirclient and service goals. Only in SDAswithout client or employer goals do perfor-mance standards have substantial unintendedeffects.

Detailed FindingsOverview of Evaluation DesignTo conduct a comprehensive and

thorough evaluation of performance stand-ards, this study was composed of two com-ponents, a quantitative evaluation based onsurveys of all States and SDAs and a qualita-tive evaluation based on indepth case studiesof 8 States, 30 SDAs, and 87 serviceproviders.

The quantitative evaluation examinedwhether differences in State performance-standards policies affected SDAs' choicesabout the types of clients to be served, theservices provided, or the program costs. Thisanalysis provided objective, statistically validevidence of the impacts of State performance-standards policies for the system as a whole.The analysis also examined the influence ofSDA enrollment criteria and service-providerarrangements on clients, service, and costs.

6

Data for the quantitativc: component wasobtained from the following sources:

Review or all State pe.-formance-standards policies for PY 86 to deter-mine incentive and sanction policies,adjustment procedures, additionalState performance standards, andpolicies for serving the hard to serve.

Survey of all SDA directors to obtaininformation on PIC and local electedofficial (LEO) concerns and in-fluence, intake procedures, enroll-ment criteria, and service-providerarrangements. (The response rate tothis survey was 87%.)

Survey of fiscal/management informa-tion system (MIS) staff to obtain in-formation on resources, expenditures,and number of participants receivingservices in various program activitiesin PY 86. (The response rate to thissurvey was 74%.)

Census data on the characteristics ofthe eligible population in each SDAas well as other published data onlocal characteristics, including the un-employment rate, percentage ofpopulation with income below pover-ty, population density, and area wagelevels.

JTPA Annual Status Report (JASR)data on the characteristics of in-dividuals served in JTPA programsand program expenditures for PY 86.

The analysis estimated the impacts ofState policies and SDA practices on clients,services, and costs, controlling for charac-teristics of the local economy and of the localeligible population.

The second component of this evaluationwas a qualitative analysis, based on indepthcase studies that included extensive onsite in-terviews and observations. The purpose ofthis component was to explore the influence

16

of Federal performance-standards system onStates and SDAs and to supplement the quan-titative analysis in understanding the extentto which State performance-standardspolicies influenced SDAs. The qualitativeanalysis also examined the mechanisms bywhich performance-standards policies af-fected clients, services, and costs and iden-tified how performance standards interactedwith other influences in determining State,SDA, and service-provider behavior.

The qualitative component was based ona sample of 30 SDAs and 87 serviceproviders in 8 States. Although necessarilylimited in number, these agencies werechosen to be as representative as possible ofthe JTPA system as a whole.

The eight States included California,Maryland, Massachusetts, Missouri,New Mexico, New York, NorthCarolina, and Ohio. These Stateswere chosen on a random basis thatalso ensured variation in State perfor-mance-standards policies andgeographic region.

The 30 SDAs were selected, 4 withineach State (except in New Mexico,where 2 of the 3 SDAs wereselected), on a random basis that alsoensured variation in SDA perfor-mance relative to their model-ad-justed standards.

Eighty-seven service providers werechosen, approximately three perSDA. These providers were selectedpurposively to ensure variation in thetype of provider, type of service, andtype of contract.

Extensive interviews were conductedwith State Job Training Coordinating Coun-cils (SJTCC) and agency staff at the Statelevel, with PIC representatives, SDA direc-tors, and staff at the SDA level, and withprogram administrators and direct servicestaff at the provider level.

7

Results from the Quantitative AnalysisThe quantitative analysis identified how

variations in State performance-standardspolicies have affected the clients, services,and costs of local JTPA programs. The quan-titative analysis traced the effects of State per-formance-standards policies in two ways.First, we measured the total effects of Statepolicies on clients, services, and costs, con-trolling for a variety of other variables, in-cluding local economic conditions,characteristics of the JTPA- eligible popula-tion in each local area, and variations in theroles played by PIC members and LEDs.Second, we examined the impact of severalSDA design decisions on the types of clientsserved including procedures to recruit andenroll different type of clients, criteria forselecting clients, the type of service providersused, and the type and terms of contracts. Wethen examined the influence of performance-standards policies on those design decisions.

The analysis of the impact of perfor-mance-standards policies showed that severaldimensions of State performance-standardspolicies do significantly affect the extent towhich SDAs serve members of hara-to-servegroups and the types and average duration ofJTPA services provided in the local area.Policies that increase service to hard-to-serveclients in JTPA include the following:

Use of the DOL adjustment modelsignificantly increases the percentageof both adult welfare recipients anddropouts served. Use of the adjust-ment model also significantly in-creases enrollment of youth with"other barriers to employment" (thatis, offenders, handicapped in-dividuals, and limited-Englishspeakers).

State policies on adjustments beyondthe model have less consistent effectsbut increase services to adultminorities and increase services to

17

youth welfare recipients andminorities.

State policies on use of 6% for hardto serve (setting aside 6% funds forservices to hard-to-serve groups, re-quiring that some 6% funds be usedto serve hard-to-serve groups, or ex-empting 6%-funded projects from theperformance standards if they areused for hard-to-serve groups) in-crease service to adult welfarerecipients and to adults with "otheremployment barriers" but decreaseservices to women. For youth, thesepolicies increase enrollment of wel-fare recipients and in-school youth.

State policies encouraging services tospecific client groups (a formal policystatement or a 6% policy that rewardsservice to a particular group) are ef-fective in increasing enrollment ofwelfare recipients, dropouts, andolder adults.

Policies that decrease service to hard-to-serve clients in JTPA include the following:

Incentive policies that emphasize ex-ceeding standards include (1) reserv-ing most of the incentive awards forsubstantially exceeding standards, (2)having SDAs compete for 6% fundsby comparing performance levelsacross SDAs, (3) not "capping" theperformance level beyond which addi-tional incentives would not be earned,and (4) requiring a large number ofstandards be exceeded to qualify forincentive awards. These policies sig-nificantly reduce service to welfarerecipients and minorities for bothadults and youth and reduce serviceto older adults and in-school youth.

Placing greater weight on the coststandards reduces service to adult wel-fare recipients and adult dropouts.For youth, this practice was estimated

8

to reduce services to welfarerecipients, dropouts, and youth withother barriers to employment.

These results indicate that some Statepolicies have their intended effects but thatthe incentive policies in some States are unin-tentionally reducing service to some hard-to-serve groups.

State performance-standards policies alsoinfluence the types of services offered inJTPA programs, including the relative em-phasis on types of program activities and theaverage duration of JTPA services.

Several State polities are generally as-sociated with more intensive or longer termservices:

Adjustment procedures. Use of theDOL model significantly increasesthe length of adult programs andreduces the amount of job-search as-sistance provided to youth. Statepolicies that specify procedures foradditional adjustments significantlyincrease provision of basic skills train-ing to youth and reduce job-search as-sistance, although they have anunexpected negative impact on thelength of participation.

State policies for hard-to-servegroups. State policies for serving wel-fare recipients significantly increasethe amount of basic skills training of-fered for both adults and youth andreduce the amount of on-the-job train-ing pm, significantly so for adults.Policies for serving dropouts sig-nificantly increase the average lengthof services for adults and youth, butpolicies for serving dropouts also in-crease the provision of job-search as-sistance, reducing OJT for adults andwork experience for youth.

Placing greater weight on the wagestandard. Policies emphasizing thewage standard significantly increase

.1.8

provision of classroom training in oc-cupational skills and reduce OJT foradults.

State policies that were found to reduceservice to hard-to-serve groups also affectedprogram services. These policies generallyeither reduced the intensity of services or in-creased the employment focus of the pro-gram. These policies include the following:

Emphasis on exceeding standards.State policies that emphasize exceed-ing standards lead SDAs to provideless basic skills training and moreclassroom training in occupationalskills for both adults and youth. Foryouth, these policies also reduce pre-employment/work maturity trainingand reduce the average length ofprogram participation.

Placing greater weight on the coststandards. State incentive policies thatplace greater weight on cost standardsreduce the average length of servicesfor adults and increase the provisionof pre-employment/work maturitytraining for youth.

Finally, State performance-standardspolicies appear to have fewer effects onprogram costs than on clients or services:

Adjustment policies. Use of the DOLmodels has no significant effects onprogram costs, although State proce-dures to allow for adjustments sig-nificantly increase the amount spentper terminee for both adults andyouth.

Emphasis on exceeding standards.State policies that emphasize exceed-ing performance standards increasecosts per terminee, significantly foradults, although not cost per enteredemployment or cost per positive ter-mination. Thus, these policies leadSDAs to enroll less hard-to-serveclients but to provide them with more

classroom training in occupationalskills, the most expensive service.

Placing greater weights on cost stand-ards. The weights placed on the coststandards do not significantly affectcosts per terminee or SDA perfor-mance on the cost standards.

One reason that the quantitative resultsare smaller for costs may be that there isserious noncomparability of reported costsacross SDAs. Some SDAs extensivelyleverage JTPA resources with funds fromother programs, as discussed in the qualita-tive analysis.

The second phase of the quantitathanalysis examined the mechanisms throughwhich the State performance-standardspolicies influenced SDA behavior. In thefirst step of this examination, we examinedthe influence of a variety of SDA implemen-tation practices on clients, services, andcosts. In the sccond step, we examined therelationship between the State performance-standards policies and those SDA practicesthat appear to influence clients, services, andcosts.

SDA policies and practices that werefound to affect the types of clients served byJTPA include the following:

PIC influence. PIC influence per sedoes not reduce service to the hard toserve. The results suggest that PICsthat see their role as guiding thedesign of the program are associatedwith greater enrollment of the hard toserve while PICs that are more in-volved in contracting are associatedwith less service to some hard-to-serve groups.

Enrollment criteria. SDAs that use ob-jective basic skills or educationalcriteria serve more hard-to-serveclients, probably because thesecriteria are used to slot participantsinto appropriate activities. In contrast,

9 19

SDAs that use subjective judgments,require previous work histories, or ac-cept "reverse referrals" fromemployers for OJT slots serve sig-nificantly fewer hard-to-serve clients.

Service-provider arrangements. Theuse of community-based organiza-tions (CBOs) as service providers isassociated with greater service toseveral hard-to-serve groups, includ-ing both adults and youth dropouts.The percentage of expenditures in per-formance-based contracts does notreduce service to hard-to-servegroups, although there is a weak pat-tern that setting stringent contractterms reduces the service to somehard-to-serve groups in JTPAprograms. SDAs that vary the termsof their contracts serve significantlymore adult welfare recipients anddropouts.

Program services. The types ofprogram services offered by SDAshave a strong influence on the typesof clients enrolled in JTPA programs.Basic skills remediation has thestrongest association with enrollmentof hard-to-serve groups, followed byclassroom training in occupationalskills, then job-search assistance.SDAs that provide more OJT tend toserve significantly fewer hard-to-serve clients.

After identifying the different effects thatthese various SDA designs and implementa-tion practices have had on client outcomes,the quantitative analysis examined whetherthe practices themselves were related to varia-tions in State performance-standards policies.The findings are consistent with the overallrelationships between State policies andclient patterns and help elucidate themechanisms by which these relationshipsare realized.

10

SDAs in States that use the DOL model,which was positively associated with ser-vices to hard-to-serve clients, were less likelyto use subjective judgments as enrollmentcriteria or to allow employers to preselectOJT trainees, two practices found to dis-courage services to hard-to-serve clients. Fur-thermore, SDAs in these States were morelikely to target both youth and adult servicesto welfare recipients and groups with otherbarriers to employment.

SDAs in States with specific policies en-couraging services to welfare recipients weremore likely to establish procedures to recruitwelfare recipients for both adult and youthparticipants. On the other hand, the localdecision of whether to target dropouts wasnot significantly affected by State policies en-couraging service to doouts.

SDAs in States win_ policies that em-phasize exceeding the standards, which wereassociated with reduced levels of services tohard-to-serve groups, were more likely to usesubjective judgements about the likelihoodof completing as a client enrollmentcriterion, which was also associated withreduced levels of services to hard-to-serveclients. These SDAs also made increased useof for-profit providers, which reduced thelevels of service to some hard-to-servegroups. Furthermore, SDAs in States with acrong emphasis on exceeding the standards

used more performance-based contracts withhigher "holdbacks" for placements andhigher wage rates. It is not clear from thequantitative analysis that these last two prac-tices had a direct impact on clients c,rved bythe )TPA program, although higher requiredwage rates did appear to discourage servicesto some hard-to-serve groups.

SDAs in States that give greater thanaverage weight to the cost standards, whichwas associated with reduced services to hard-to-serve groups, also were more likely toallow employers to make prescreened refer-rals of clients for OJT slots and were less

20

likely to use special procedures to recruit wel-fare recipients into the JTPA program.

Overall, these results indicate that perfor-mance-standards policies affect the types ofclients served, in part through their influenceon the types of services offered and in partthrough their influence on SDA enrollmentand contracting practices.

The results of the quantitative analysis in-dicate several State performance-standards af-fect SDA decisions about whom to serve andthe types of services to provide. These resultsare statistically reliable and have importantpolicy implications. Nonetheless, tin effectsof these policies are not large. These policiesdo not preclude service to the hard to serve orprevent provision of intensive services.Rather, these policies affect SDA tradeoffs atthe margin.

Results from the Qualitative AnalysisThe analysis of the qualitative case study

data examined how and why the 30 casestudy SDAs varied in their reactions to theperformance standards. In tracing the effectsof the Federal and State performance-stand-ards policies facing each SDA, we examinedseveral elements of SDA program-design andimplementation decisions: (1) the develop-ment of client goals and SDA plans for ser-vices to particular groups; (2) the develop-ment of overall service designs, includingwhat services to offer, how much of each ser-vice to provide, how intensive to make eachservice, and how services relate to each otherin the development of service plans for in-dividual clients; and (3) the design ofmanagement practices for selecting serviceproviders and overseeing program implemen-tation. Within each of these program ele-ments, we used the case study data to addressquestions about how the performance stand-ards influenced SDA and service-providerdecisions, how the performance standards in-teracted with other factors, and how theresulting SDA practices influenced clients,services, and costs.

11

Effects on Local Program Objectives.There was considerable variation both in theextent :o which the performance standards in-fluenced local performance goals and in theways that standards produced these effects.About one-third of the case study SDAs hada goal of meeting the standards and "stayingout of trouble" (that is avoiding sanctions).Another one-fourth of the SDAs indicated agoal of exceeding the standards slightly, tohave a performance safety margin and to real-ize some incentive awards. The remainingsites indicated a goal of performing at as higha level as they could on the standards,primarily because it was important to maxi-mize their potential incentive awards. Pridewas also mentioned by all types of SDAs asan important incentive to perform at or abovethe level of performance standards.

The relative importance of the perfor-mance standards in shaping local goals andpriorities was conditioned in part by the im-portance to the SDA of receiving the maxi-mum amount of incentive funds. Some SDAsplaced high priority on receiving funds be-cause they viewed them as compenoting forshrinking 78% allocations and tight ad-ministrative limits. Others valued the awardsfor their public relations value. Still othersused the awards to undertake projects thatthey could not fund with 78% money, suchas projects serving high-risk clients ormarketing or outreach projects that only in-directly contributed to outcomes.

Several typologies of responses to the per-formance standards help explain the varia-tions in how the performance standardsinfluenced clients, services, and costs withinthe case study sample. Client-oriented SDAsgave priority to their client objectives andoriented their program-design and manage-ment decisions around how this would affecttheir client goals. Employer-oriented SDAsgave priority to designing and implementinga service mix that would train individuals forjobs in demand in the local labor market. Per-

21

formance-oriented SDAs gave priority tomeet4ng or exceeding the performance stand-ards (or other State or local performancemeasures). These were ST -As without strongemployer or client goals to counterbalancetheir goals to exceed their performancestandards.

Some client-oriented or employer-oriented SDAs were indifferent to the stand-ards when they designed their progarns,either because they did not care if they mc,the standards or because the standards wereso easy to meet or exceed. Others adoptedrisk management strategies that enabled themto meet their performance goals while stillfurthering local client and service objectives.Performance-oriented SDAs tended to adoptrisk avoidance strateg; -ts 'hat increased thelikelihood that they w.luld meet the perfor-mance standards by avoiding hard-to-serveclients or adjusting the service mix forreasons that were standards oriented.

Finally, SDAs varied in the extent towhich the SDA assumed responsibility formeeting its performance goals, shared thatresponsibility with service providers, orpassed on the primary responsibility for per-formance to the service-provider level.

In balancing these different goals, the per-formance standards played an important rolein some SDAs and a minor role in others.Only a few SDAs in the case study samplepurely relied on "risk avoidance" rather than"risk management" in their response to theperformance standards. Most SDAs couldmeet their performance goals without havingto resort to a major revision of otherprogram goals.

Effects on Client Goals. The qualitativeanalysis suggested that the performancestandards do not influence SDA client goalsdirectly but rather influence clients servedthrough their influence on SDA servicedesigns and implementation practices. Otherfactors that did influence client goals in-cluded (1) State equitable-service require-

12

ments and State policies for serving specifichard-to-serv., groups; (2) the 40% youth ex-penditure requirement; and (3) the charac-teristics of the applicant pool, which werestrongly influenced by environmental factorsbeyond the SDA's control, particularly thelocal unemployment rate.

Moreover, the case studies demonstratedthat the performance standards did notprevent SDAs that had a strong commitmentto serving hard-to-serve groups from target-ing and ::ruing those groups. Welfarerecipients and dropouts were served out ofproportion to their incidence in the popula-tion in a number of SDAs. Strategies for serv-ing these clients included (1) designingspecial training classes that integratedremediation and world-of-work orientationwith occupation skills curricula, (2) leverag-ing other JTPA funds (for example, 8% and6%) or non-JTPA funds (for exaizple, Statefunds and foundation funds), (3) targetingemployers that offered health benefits, and(4) making explicit tradeoffs to balance ser-vices to hard-to-serve clients with less expen-sive services to other groups to enable theSDA to meet its performance requirements.In addition, groups with other barriers toemployment (the handicapped, offenders,and limited-English speakers) were targetedfor services by some SDAs. These groupswere usually served in special programsoperated by providers with a history of work-ing with these groups.

Despite this general pattern, some SDAsthat were highly motivated to exceed theirstandards were influenced by fi: ,lrfor-mance standards to develop 5 isk avoid-ance" strategies. These SDAs tended to ad-dress the needs of some subset of eligible ap-plicants that did not include those with thegreatest barriers to employment.

Effects on Program Services andCosts. Performance standards had slightlymore influence on the service mix offered toadults by SDAs. A few SDAs mentioned the

22

standards when explaining their emphasis onOJT rather than classroom training.However, at least equally important influen-ces on service mix were (1) the SDA's his-torical pattern of service delivery, as evolvedfrom the Comprehensive Employment andTraining Act (CETA) program; (2) theSDA's intentions about serving differentclient groups; and (3) PIC preferences aboutwhat services to emphasize in the SDA's ser-vice package.

SDAs exhibited wide variation in the in-tensity and duration of JTPA services. In ad-dition, the costs-of-JTPA-serviteg-varieddramatically among SDA::, and average costfigures often disguised tremendous variationwithin an SDA in the duration and intensityof services received by different participants.The cost standards had some influence onthis aspect of service design, because SDAsoften wanted not only to meet but to exceedthese standards to maximize their incentiveawards. For the SDAs that were highlymotivated to "overperform" on the standards,the standards influenced them to design ser-vice programs that had short durations,yielded high entered-employment rates, andresulted in low unit costs. Thus, in theseSDAs, the initial reaction to the standardswas to emphasize quantity over quality. Anumber of these SDAs are beginning to shifttoward more intensive services, however, par-tially in response to the implementation ofthe follow-up standards.

However, other factors were equally asimportant in determining the intensity andduration of services. These included (1) theavailability of other funds to supplementTitle II-A funds in purchasing services tobenefit Title II-A enrollees, (2) the cost of thevarious service packages available from theservice providers in the SDA, (3) the SDA'sgoals about the types of clients it wanted toserve and the intensity and types of servicesthose clients required, and (4) the desire tospread JTPA funds over as large a client pool

as possible.There were two reasons why SDAs were

very concerned about reducing averageJTPA costs. A majority of SDAs visited wereexperiencing declining JTPA allocations (dueto declining unemployment rates), which re-quired a decision about whether to serve asmaller volume of participants and keep theaverage cost of services stable or to try toserve as many participants as possible by cut-ting back the average cost of the servicesreceived. Almost universally, the SDAsvisited had decided to try to reach as many in-dividuals-as-possible, without diluting the ser-vice intensity beyond reason.

The second reason for an emphasis oncost reduction was the SDAs' desire to ex-ceed the cost performance standards. Al-though the vast majority were in no danger ofmissing these standards, the emphasis onoverperforming grew out of a desire to maxi-mize their incentive awards.

The existence of performance standardsfor youth did not seem to affect the extent ofSDA resources devoted to youth programsper se. Performance standards did, however,affect SDA decisions about serving in-schoolversus oat-of-school youth. There is an inten-sive debate among and within SDAs aboutthe extent to which JTPA programs shouldbe offering employment-oriented or com-petency-oriented training. SDAs that chose toemphasize competency-oriented servicesfound it difficult to meet their entered-employment-rate standards while those em-phasizing employment-oriented programsfound it difficult to meet their positive-termination-rate standards.

Nonetheless, local goals about whether torun employment-oriented or competency-oriented programs for youth, as well as theFederal requirement to spend 40% of JTPAfunds on youth, appeared to have greater in-fluences than the standa,a. However, a num-ber of SDAs had instituted a youth employ-ment-competency system to meet their posi-

13 23

tive-termination-rate standards.Effects on Management Practices.

SDA management practices were designed tofurther a variety of management goals, in-cluding ensuring (1) that the planned mixof clients was actually served, (2) that theplanned mix of !:2rvices was actuallyprovided, (3) that each client was referred toappropriate services, (4) that serviceproviders enrolled clients that wereappropriate for that service and had areasonable likelihood of success, (5) thatcosts would be kept within reasonable limits,and (6) that the desired performance levelswould be achieved.

To further their management goals,SDAs instituted a number of practices for en-rollment and assignment to services, place-ment, and selection of service providers.Some of these practices grew out of their ap-proach to the performance standards.

Most of the SDAs visited maintainedcentralized control over the outreach, assess-ment, and referral of clients to service com-ponents, with a smaller number leavingresponsibility for these services to serviceproviders. Contrary to expectations, the en-rollment of hard-to-serve clients was not cor-related with the SDA's control over enroll-ment; SDA and service-provider goals aboutserving the hard to serve seemed to be moreimportant than who controlled enrollment.

Some SDAs protected themselves againstthe risk of serving hard-to-serve individuals,however, by delaying enrollment until aftersome services had been delivered to avoidhaving to include early dropouts in theirstatistics.

Performance standards did affect SDAs'service-provider arrangements-in particular,service providers' past performance often in-fluenced the size of their contract in the nextfunding period. However, a number of addi-tional factors influenced provider selectionand retention, often more strongly than theperformance standards per se, including the

14

availability of alternative service providers,the characteristics of the local economy, theamount and categories of services that theSDA contracted out, and the goals andphilosophy of the PIC or SDA.

Two-thirds of the ise-study SDAs hadsome current perforo mce-based contracts.Although the performance standards were afactor in the spread of performance-basedcontracting among the case study SDAs, the15% administrative limit was at least as im-portant; this was especially true in the con-text of declining Title II-A allocations.

Among performance-based and cost-reim-bursement contracts alike, the great majorityincorporated performance criteria as either re-quirements or goals. In about half of the casestudy SDAs, contracts built in the SDAs'standards (usually with a "cushion") directly.Most of the others varied key contract termsamong their providers but took care that, ag-gregating across the providers, they met orexceeded the SDA's own standards.Providers facing high placement expecta-tions, high-wage goals, high placementholdbacks, and caps on maximum serviceduration were inclined to be cautious aboutthe clients they would accept. If high-percent-age-holdback contracts become moreprevalent, "riskier" die its, who have moresevere employment bar iers, may be morelikely to be screened it of service programs.

Contract terms should not, however, beunderstood as simple outgrowths or reflec-tions of the standards. Many of the key fea-tures of these contracts predate JTPAprograms, as does the use of performance-based contracts in several of the case studySDAs. The relationship between contractterms and the SDA's performance standardswas affected by several factors. These in-cluded how difficult a given SDA foundrecruiting and retaining clients, the relativeemployability of available clients, theavailability of alternative service providers,the status and direction of the SDA's Title II-

A allocation, its access to non-JTPA funding,and the philosophy and goals of the PIC andthe SDA staff. Finally, even when the formof performance-based contracts is a recentdevelopment, they often essentially continuethe mutual expectations that an SDA and aprovider had developed in earlier years.

Service providers varied in their reactionsto the performance expectations placed onthem by the SDAs. In a few cases, contrac-tors faced pressure.: to perform at high levels,which caused greater caution in client selec-tion. Most providers, however, had a commit-ment to serving hard-to-, erve clients, andtheir contract terms gave . hem enoughflexibility to meet that commitment.

For the most part, the management prac-tices found in the case study SDAs are consis-tent with "risk management" strategiesrather than "risk avoidance" strategies. Thatis, using these practices to safeguard perfor-mance levels, SDAs could and did, serve dif-ficult clients under the JTPA piogram whileattaining their performance goals.

Implications for the Performance-Standards System

The findings of this evaluation indicatethat the performance standards do notpreclude SDAs from enrolling hard-to-serveclients or from providing intensive services.Nonetheless, the results of this evaluationalso indicate that not all aspects of the perfo.mance-standards system are working as in-tended and that some policies have importantunintended effects. In this section, we ex-amine the implications of the results for bothFederal and State performance-standardspolicies.

Federal PoliciesChoice of Performance Measures.

Cost Measures. The JTPA legislation man-dates that cost standards be included asFederal performance standards. Both thequantitative and qualitative results, however,indicate that the cost standards had the most

unintended effects and were the least com-parably measured of all the Federal stand-ards. There were unintended effects on bothclients and services: States that placed a highweight on the cost standard lead SDAs toserve fewer welfare recipients and dropouts;in our case study sample, some SDAs con-cerned about exceeding their cost standardsdesigned short-term, less intensive services.

The measurement problems in the coststandards were also serious. SDAs that wereable to leverage JTPA Title II-A funds withother program funds or that were able to relyheavily on service providers with alternativefunding sources had much lower measuredcosts than those SDAs that relied solely onTitle II-A resources to train their participants.Some quantitative results supported this find-ing, and our case studies found dramatic dif-ferences in the leveraging of JTPA funds. Asa result, basing incentive paymeis on thecost standards probably rewards differencesin the local availability of other trainingresources as much as differences in manage-ment quality.

Clearly, cost-effectiveness is an essentialgoal of any program, and an increase in theextent of leveraging, which reduces redun-dant services, may be a desired effect of thecost standard. Nonetheless, we found thatmany SDAs were already very concernedabout costs because of a strong desire toserve as many participants as possible vithlimited (and often declining) JTPA funds.

The results of this evaluation suggest thatalternatives to the cost standards should beexplored. Out of concern for the unintendedeffects of the cost standards, DOL set muchmore lenient costs standards for PY 88. Thispolicy is not likely to be effective, however,in States that strongly emphasize exceeding,not just meeting, standards, particularly whenthe cost standards are weighted more heavily.An alternative that requires legislative chan-ges would be to set maximum costs perentered employment (or positive termination)

1525

that SDAs could spend but not to base incen-tive payments on cost standards. In effect,this policy would treat costs as a compliancerather than al_erformance-standards isstYouth Measures. The only other problem inthe choice of measures was a less serious con-cern raised about the youth measures. SDAswere generally confused about how tomanage the youth programs to meet both theentered-employment-rate and positive-ter-mination-rate standards. Because the goals ofin-sc.,,,o1 and out-of-school programs wereso different, SDAs that emphasized one typeof service often felt pinched by at least oneyouth standard. One possible solution is :oset separate standards for in-school programsand out -of- school programs so that theSDAs' choice of program mix would not af-fect their ability to meet the youth standards.

A related issue is the inclusion of youthemployment competencies in the positive-ter-mination measure. Although evaluating thequality of the competency systems wasbeyond the scope of this study, we found thata substantial majority of the case study SDA,had made a conscientious effort to establishmeaningful systems. However, many SDAsreported that they would not have adoptedthe competency-based approach in the ab-sence of dr.; positive-termination-rate stand-ard. Thus, SDAs naturally tended to developsystems that would reflect well on theprograms that they were running. Despite thetemptation to implement a superficial youthemployment competencies (YEC) system,only 2 of the 30 case study SDAs hadadopted a "quick and dirty" competency sys-tem, and 1 of those used it only as a back-upoutcome for their employment-orientedyouth services.

Level of Standards. The results of thisevaluation do not indicate the need for chan-ges in the general levels at which perfor-mance standards are set. The low levels ofmost standards generally allow SDAs tomeet the standards and design their programs

16

for local needs, while providing a clear incen-tt for good performance The only excep-

tion is a concern among a number of SDAsthat the positive-termination-rate standard istoo 1101 for serving very at-risk youth, manyof whom may drop out of the program.

DOL Adjustment Models. The use ofthe DOL adjustment models appears to bevery effective in increasing service to hard-to-serve groups and in increasing the length ofadult serviTeT.-These models, therefore, ap-pear to have their intended effects of holdingSDAs harmless for their local decisions ofwhom to serve in 3TPA programs.

The models adjust for ;haracteristics ofclients served and the length of services, aswell as local economic conditions. Themodels do not, however, adjust for the typ..,sf cervices offered. This evaluation has

found that the types of services offered sig-nificantly affect the types of clients enrolledin JTPA programs and that some of the unin-tended effects of performance standards, par-ticularly the cost standards, are *.o reduce theprovision of intensive services. Furthermore,we found that the types of services sig-nificantly affect program costs, with class-room training in occupational skills and basicskills remediation costing considerably morethan job-search assistance.

Because the adjustment models are effec-tive tools in holding SDAs harmless for localdecisions, these results imply that seriousconsideration should be given to includingsome adjustments for program activities inthe models for the cost standards. In par-ticular, adjusting for basic skills remediationwould enable SDAs to spend more moneywhen they include basic skills remediation intheir service package.

Technical Assistance. The evaluationfindings suggest that additional technical as-sistance from DOL could be used to improvethe effectiveness of the performance-stand-ards system. First, DOL could provide assis-tance to States to develop their full

leadership potential in setting client and ser-vice priorities for their SDAs and to developperformance - standards polic:.:s consistentwith those goals. Second, DOL could provideadditional assistance to SDAs to developstrategies to balance performance goals withother client and service goals. Third, al-though there is a moderate level of under-standing of the adjustment model, manySDA respondents felt unsure that they fullyunderstood how the process worked. Givenhow important the model is in encouragingservice to the hard to serve, DOL efforts toimprove understanding of the model shouldincrease the intended effects of the model.

Reporting. We found very little evidencethat the Federal reporting requirements neces-sary to support the adjustment models were aburden to SDAs, providers, or participants.Most of the information required was infor-mation that the providers needed to managetheir programs or for equal-opportunitydocumentation. Furthermore, most SDAsmaintained computerized participant-leveldata bases (often supported by the State), sopreparing reports on participant characteris-tics was not difficult. In fact, some respon-dents indicated that the greatest burden wasthe delay in issuing the JASR revisions be-cause of the prolonged negotiations betweenDOL and the Office of Management andBudget (OMB).

However, two reporting practices reducedthe comparability of the performancemeasures. First, SDAs typically delayed en-rollment of participants until after an orienta-tion period so that individuals who droppedout early were not enrolled. This pre-enroll-ment period usually lasted 3 to 5 days, but insome SDAs went on for several weeks. Wefound that SDAs with longer pre-enrollmentperiods have significantly higher measured-entered-employment rates. These results sug-gest that some monitoring of this process atthe State or Federal level would increase thecomparability of measured performance.

170)

Second, as discussed above, the fact thatmany SDAs leverage JTPA funds with otherresources substantially reduces the com-parability of the cost standards. The problemcannot be rectified by reporting requirementsbecause SDAs would find it very difficult toestimate the value of other resources (forexample, the State subsidy to communitycolleges).

State PoliciesState Performance Standards. States

are increasingly adopting State performancestandards to further their goals for the JTPAsystem. The study demonstrated that Statescan affect the direction of the JTPA programby implementing State perfonnance stand-ards to supplement the Federal standards. Inparticult-r, State standards for service to tar-get groups furthered the State's clientpriorities for the JTPA program. The successof these policies should encourage otherStates to attempt to play a leadership role ifthey desire to do so.

Additional Adjustment Procedures forthe Federal Standards. Although the effectsare not entirely consistent, the qt 'tativeresults suggest that State policies : specifyprocedures to obtain additional adjustmentsbeyond the model do increase service tosome hard-to-serve groups and encourageprovision of basic skills remediation toyouth. However, in our case studies, wefound that both State and SDA staff were notparticularly comfortable with the adjustmentprocess as it now exists. State staff felt un-sure about how to establish equitable criteriafor adjustments beyond the model and abouthow to determine the appropriate size of ad-justments. SDA staff often felt that they didnot understand the statistical basis of themodels well enough to justify adjustments.In addition, some staff felt that requesting ad-justments was admitting failure and, par-ticularly when SDAs competed for funds,that it was not fair to "go begging."

One SDA director suggested an alterna-tive process that he felt would be more ac-ceptable. Specifically, he suggested thatSDAs be allowed to request waivers from theperformance standards for projects that metwell-specified criteria, for example, for start-ing up a program for high-risk clients. Thiswould be more acceptable, he said, becausethe standards for the remaining projectswould not be affected, so that all SDAswould still be abiding by the same rules.

Hard-to-Serve Policies. State policies toencourage service to hard-to-serve groups in-clude additional State performance standards,integrating service to hard-to-serve groupsinto the calculation of incentive payments,and identifying priority groups. The evalua-don indicates that States can play an effec-tive leadership role in focusing JTPAprograms on the needs of particular hard-to-serve groups through a variety of incentiveawards adjustments, in addition to the im-plementation of special State standards.

Incentive Policies. The JTPA legislationrequires that incentive payments be awardedbased on the extent to which standards are ex-ceeded. States policies vary widely in howthat requirement is implemented. Policiesthat place a strong emphasis on exceedingstandards are found to lead SDAs to reduceservice to hard-to-serve groups, reduceprovision of basic skills remediation, and in-crease the employment focus of both theadult and youth programs. These are largelyunintended effects of both the legislative re-quirement and of the specific State policies.The results of this evaluation strongly sug-gest that these policies should bere-examined.

Another important aspect of incentivepolicies is the weight placed on each stand-ard. Fish weights on the cost standard werefound to reduce service to welfare recipientsand dropouts. States that equally weight thepercentage by which each standard is ex-ceeded implicitly give substantially greater

18

weight to the cost standards because costs aremore variable among SDAs. These policiesshould also _ re-examined.

The case studies also indicate that incen-tive payments are very important to SDAs be-cause of two other Federal policies. First, theallocation formula bases funding on the un-employment rates. Incentive payments helpSDAs cushion the decline in JTPA fundingbrought on by an improving economy.Second, many SDAs are squeezed by the15% limit on administrative costs. Because30% of incentive funds can be used for ad-ministrative costs, these funds are particular-ly valuable to many SDAs.

Sanction Policies. Many States have notdeveloped specific policies for sanctioningSDAs for poor performance, and even whenthese policies are in place, States are reluc-tant to enforce them. Tie, threat of reorganiza-tion has little effect on SDA behaior.

Technical Assistance. The study find-ings suggest that SDAs would benefit fromadditional State technical assistance to clarifythe State's intention in designing its perfor-mance-standards policies and to discuss howthe performance standards should interactwith other client and service priorities in fur-thering JTPA program priorities.

Conclusions1. The performance standards need to be

balanced by local client and servicegoals to provide a useful guide tomanaging JTPA programs. If taken asthe sole statement of program goals,the standards can lead to unintendedeffects of reducing service to hard-to-serve groups and decreasing the inten-sity of JTPA services.The study found that States, SDAs, PICs,

and many service providers express strongsupport for the idea of cost-effectiveness andaccountability for outcomes as an importantpart of the JTPA program. When SDA stafftalk about the effect that performance stand-

ards have had on the design and operation oftheir programs, they generally do not ques-tion the premise that they should be account-able for program outcomes.

The idea of performance accountability,however, has been embraced by some in-dividuals in the JTPA system to the extentthat performance goals sometimes receivemore emphasis than other program objec-tives, such as client priorities and goals aboutwhat types of services to offer. The absenceof clear client and service goals in someStates and SDAs is unfortunate, because itcan result in performance goals receivingundue emphasis. Performance standards hadnotable unintended effects of reducing ser-vice to the hard to serve and decreasing theintensity of services only in SDAs withoutwell-defined client or service goals.

In general, however, the performancestandards are not preventing SDAs from pur-suiig their local client and service goals andare not dramatically altering the types ofclients served by JTPA programs or the typesof services being offered.2. Although the performance-standards

system as a whole appears to be work-ing, the cost standards are moreproblematic than the other standards.Reducing the emphasis on the coststandards would reduce unintended im-pacts of the standards.The Federal cost standards have the most

unintended effects and are the least com-parably measured of all the Federal perfor-mance measures. In addition, SDAs in Stateswith incentive policies that emphasize perfor-mance in excess of the standar? often findthat reducing costs is the easiest and mostdramatic way to perform beyond their stand-ards.

Thus, concerns about exceeding the coststandards tend to reduce the average intensityof services offered by some SDAs. Thesefindings suggest that alternatives that reducethe emphasis on the cost standards should be

19

explored.3. State incentive policies have unin-

tended effects and need to bere-examined. States should be assistedin developing policies that enable themto fulfill their potential of providingleadership to the JTPA system.The State incentive policies that en-

courage SDAs to perform as high as possibleon all the standards have unintended effects.The quantitative analysis showed that thesepolicies have modest but consistent effects ofreducing service to hard-to-serve clients andreducing the provision of expensive programservices, such a basic skills remediation.

Because the tt impacts are probably notintended, States should carefully review theirperformance-standards policies. One solutionmay be for States to establish more explicitlyclient priorities in their performance-stand-ards policies. Another solution may be forStates to develop policies that are moreneutral in their effect on SDA client targetingdecisions and service design choices.

Solutions should be explored to permitStates to reward SDA performance that is"exemplary" rather than "average" whileavoiding unintended effects on clients andservices.4. The youth standards are currently in-

fluencing SDA decisions about clienttargeting and program design. Onechange that would reduce unintendedstandards impacts would be to estab-lish separate standards for in-schooland out-of-school youth.There is intense debate within and among

SDAs about whether youth programs shouldbe oriented toward immediate employment ortoward employment-competency outcomes.We saw no signs of an emerging consensuson this issue.

The performance-standards system re-quires SDAs to report both employment andemployment - compete. 7y outcomes for allyouth participating in the JTPA program.

This compromise, however, has not beencompletely satisfactory from the SDAs'perspective. SDAs that emphasized servicesto in-school youth often found it difficult tomeet the youth entered-employment-ratestandard, while SDAs that emphasizedemployment-oriented services to youth whohave left school often found it difficult tomeet the youth positive- termination -ratestandard.

Although most SDAs have developedprogram designs that enable them to meet theyouth performance standards, there is littleei.thusiasm and considerable confusion at theSDA level over the youth standards. Onechange that might increase local discretionwould be to establish separate standards forin-school youth and out-of-school youth.

Recommendations of the Commission

The National Commission for Employ-ment Policy believes that this report is an ex-tremely valuable contribution to research onthe Job Training Partnership Act (JTPA), andone whose findings have implications that gowell beyond the scope of this projecttodetermine the i.npact of performance stand-ards on clients, services, and costs. Insightsgained from the comprehensive analysis ofboth quantitative and qualitative data call foractions from national, State, and localpolicymakers that are not limited to perfor-mance standards. Therefore, theCommission's recommendations includeboth general ones as well as specific recom-mendations concerning changes to the perfor-mance management system itself.

General RecommendationsNational PolicyThe Act balances legislative and Federal

objectives for the program with local discre-tion designed to allow the program to meetlocal needs. The Commission believes thattargeting of JTPA programs works very well

21

but that the Congress must be very clearabout who the Act is intended to serve whilemaintaining the balance between national ob-jectives and local discretion. Further claritymay be needed in defining the three major tar-get groups mentioned in the Actwelfarerecipients, dropouts, and youth. For example,with respect to welfare recipients, since legis-lation affecting both the welfare system andJTPA's service to welfare recipients has beenunder consideration by the Congress, it maybe timely to clarify that long-ierm welfarerecipients (those receiving welfare for 2 yearsor longer) are the welfare recipients bestservc :n JTPA programs. In the area ofdropouts, it may be necessary to definedropouts as youth (since the lack of a highschool credential, for a youth more so thanfor adults with work experience, is a sig-nificant barrier to employment). For alldropouts, basic skills deficiencies appear tobe more of a barrier to full workforce par-ticipation than educational status,

Probably more than any other vehicle,performance standards have been used to

31

transmit national policy in Jr . programs.Sometimes these signals, transmitted throughthe choice of performance measures and na-tional standards, have not always been clearto State and local program managers. Recent-ly, much progress has been made in usingperformance standards to reinforce alreadystated policy goals and objectives.

However, we believe that performancestandards should reinforce policy, not estab-lish policy goals. The Commission recom-mends that the Department of Labor clearlyarticulate (1) policy objectives for theprogram apart from performance standards,(2) specific policy on the provision of basicskills for both youth and adults, (3) objec-tives about the intensity of training (and thetype of training appropriate for JTPA par-ticipants) that are desirable, (4) appropriateoutcomes for both in-school and out-of-school youth, and (5) objectives concerningthe State role in promoting such objectives.

The Department of Labor needs to giveadditional attention to the collection of anddefinition of data elements to support policyobjectives and work with the Office ofManagement and Budget to implement amore carefully defined management informa-don system.

State PolicyThis report has provided new insight into

the role of States in the performance manage-ment system. As the report indicates, Statepolicies can either promote or dissuade PICsand SDAs from providing intensive serviceto the hard to serve. The Con: missionbelieves that better use can ba made of thesepolicy tools through the goals and objectivesdetailed in the Governor's Coordination andSpecial Services Plan, through incentiveaward policies for exceeding performancestandards, through sanctions for failure tomeet performance standards, and through theuse of incentives or serving the hard to serveas specified in Section 202 (b)(3)(B) of theAct. The Commission recommends that

States improve their incentive policies topromote the provision of service to thoseclients deemed to be most in need. One par-ticular mechanism to achieve a clearer focuson the hard to serve is through better use ofincentives for serving the hard to serve.

Local PolicyThe Commission is concerned that one

partner in the Act, the service provider, isstill largely overlooked in the performancemanagement system. Since JTPA servicesare often provided by entities other than theadministrative entity, we are concerned thatcontractors need to be included in both therisks and the rewards of programs. The Com-mission recommends that PICs developreward s;_ 'ems built into contracts for ser-vice providers who are successful in provid-ing intensive service to the most in need inthe JTPA population, particularly for welfarerecipients and dropouts, as well as othergroups the PICs target for service.

Specific Performance ManagementRecommendations

The Commission calls attention to thefact that all recommendations made here areprovided based on the performance manage-ment system in place during PY 86-87without the benefit of analyzing the effect ofthe implementation of post-program perfor-mance measures and standards. This reportnotes a fair degree of support for measuringthe erriployment and earnings of individualsafter they leave JTPA programs, but since nostandards were in place during the year ofthis study, we could not directly assess theireffect on who is served, the type of service,or the cost of such service.

We do believe, however, that most of therecommendations offered here will applyonce post-program standards have been fullyimplemented into the JTPA system.

Choice of Measures1. The Commission strongly supports the

concept of cost-effectiveness inherent in per-

,

2232

formance standards and recognized in theAct but believes that too much emphasis onefficiency measures (in this case two perfor-mance standards dnling with cost) may havehad the unintended effect of precluding the at-tainment of effective programs to meet thegoals of the Act. We are also concernedabout the comparability of the cost informa-tion across programs. JTPA programs havemade great strides in leveraging the resourcesof other systems to serve the economicallydisadvantaged; however, this may mean thatthe actual cost of serving some clients isunderreported in some programs.

Therefore, the Commission recommendsthat Section 106 (b)(4), which requires theSecretary to prescribe performance stand-ards relating gross program expenditures tovarious performance measures, be amendedto direct that cost-efficiency be monitored byStates.

2. The Commission endorses the conceptthat employment is the major objective of theAct and believes that the Act wisely em-phasizes increases in employment and earn-ings as major goals. However, the Act alsorecognizes the importance of outcomes otherthan employment for youthnamely, theattainment of youth employment competen-cies; completion of major levels of school;and enrollment in other training programs.

As the report's findings indicate, one ofthe intended effects of performance standardsfor youth that was realized was the develop-ment of programs to enhance a youngperson's future employability through the at-tainment of youth employment competencies.While the Commission favors this approachto youth programs, we are concerned aboutthe continuing problems in the youthmeasures. The Commission endorses preserv-ing local programs' discretion about the rela-tive emphasis of serving in-school andout-of-school youth. The new youth measureintroduced for PY 88 will not fully addressthe differing goals of in-school programs that

focus on improving basic skills competenciesand dropout prevention strategies and out-of-school programs that should focus onemployability development (particularly inthe area of basic skills) for youth for whomthe final outcome is employment. The Com-mission therefore recommends that theDepartment of Labor develop separatzreporting for in-school and out-of-schoolyouth programs and develop appropriateoutcomes for both.

Exceeding Performance StandardsThe evaluation of the effect of perfor-

mance standards also points to another areaof concern in the legislation that we believerequires remedy. Section 202 (b)(3)(B) of theAct directs the Governor to provide incentiveawards for programs "exceeding perfor-mance standards." As indicated in this evalua-tion, this language may have had theunintended effect of promoting overperfor-mance and setting up competition amongSDAs for incentive funds at the expense ofsome clients and services that might other-wise be offered for the hard to serve. One ofthe original, principles upon which the perfor-mance management system was based is thatSDAs should be judged against their ownlocal circumstances, that is, economic condi-tions, characteristics of the population to beserved, and so forth. The Commission recom-mends that the Congress clarify its intent topromote service to the hard to serve bychanging Section 202 (b)(3)(B) of the Act toprovide for incentive awards based on "meet-ing" performance standards. At a minimum,States should include provisions in their in-centive policies that do not promote overper-formance at the expense of service to thehard to serve.

The Commission believes that furtherdiscussion is needed about more specificguidance to the States concerning "incentivesfor serving the hard to serve." Therefore, theCommission recommends that an amendmentto Section 202 (b)(3)(B) be considered to re-

2333

quire States to provide such incentives.

Encouraging the Provisionof Basic SkillsWe are particularly concerned about en-

couraging the attainment of basic educationalskills for both youth and adults. We have sup-ported the efforts of the Secretary in em-phasizing the role of training programs likeJTPA in ensuring that basic skills remedia-tion is a significant part of any employabilityprogram. The Commission is also interestedin ways to provide incentives to JTPAprogram operators to link improvements inbasic skills with occupational training and ul-timately employment. One way to improvethe performance management system'sability to do so may be to collect data thatcould be used to adjust standards or providebonuses for programs that emphasize theprovision of basic skills in addition toemployment. The Commission thereforerecommends that the JTPA Annual StatusReport be changed to include information forboth adults and youth who, in addition toentering employment, attain basic skillswhile enrolled in JTPA progams (whetheror not attained with JTPA funds).

Adjustment to StandardsThe Commission is pleased that the ad-

justment models used by the majority ofStates to establish local SDA standards havethe intended effect of promoting service tosome target groups for which employmentmay be more difficult to achieve (the so-called hard to serve); however, we are con-cerned that the use of any further adjustmentsdoes not seem to be an effective approach fora variety of reasons. JTPA programs must en-courage innovative strategies to solve the dif-ficult problems faced by those served underthe Act. While we commend the Departmentof Labor's effort to promote additional adjust-ments, the Commission believes that it istime to consider alternative approaches.

We do not believe that enough work has

24

been done in this area to suggest specific ap-proaches, but some suggestions from recentCommission publications and from the workof other groups should be explored. Thesestudies suggest utilizing incentive approach-es rather than (or in addition to) the "holdharmless" approach currently used in the ad-justment models. Some of the specific op-tions that should be considered include theuse of incentive funds for serving at-riskclients based on pre-enrollment criteria suchas targeting individuals with limited priorwork experience, with basic skills deficien-cies, and who are long-term welfare recip-ients; and the use of waivers to performancestandards to encourage innovative strategiesfor programs that serve the most at risk. Awaiver could also be used for programs ex-pected to transcend program years, thuspromoting longer term programs whensuitable for the hard to serve. The Commis-sion recommends that the Department ofLabor explore such alternative approacheswhich may go "beyond the adjustmentmodels" to promote positive incentives toserving individuals most at risk of futureemployment problems.

Incentive AwardsThe Commission is concerned that some

incentive award policies have become socomplex that they may be having the unin-tended effect of dissuading service from thehard to serve simply because the policies aredifficult to understand. This report has noteda number of both positive and negative ap-proaches currently in use by States that affectwho is served, the type of service, and thecost of service. Translation of goals and ob-jectives through the use of the incentiveaward rystem needs to be kept as simple aspossible to have the desired effect on localprograms. The Commission therefore recom-mends that technical assistance be providedto States to ensure that incentive policieshave the effect States intended and thatStates clearly reward service to clients most

34

in need.

Technical AssistanceThe Commission is ;,'.eased that, for the

most part, Pies and SDAs have a fairly goodunderstanding of the performance standardssystem, but we are concerned that a differentstrategy in the delivery of technical assis-tance is needed. The Department of Labor istc be commended for providing assistance onperformance standards, including theirdevelopment and ways to adjust standards.However, we are concerned that their techni-cal assistance on standards does not suffi-ciently promote the policy objectives impor-

25

tant to system-wide program improvementaimed at better preparation of the workforce.The Commission recommends that futuretechnical assistance efforts be focusedon im-proving programs and developing innovativestrategies targeted to long-term employmentof the most in need. Performance standardsshould be included as one aspect of how toplan and manage programs. Such a strategyshould include PICs and service providersand must emphasize how to integrateprogram goals about who to serve and thetypes of service with performance expecta-tions into contracts.

,I5

* U.S. GOVERNMENT PRINTING OFFICE : 1988 0 - 223-605