SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE? Charles Smith, Ph.D. Executive Director, David P....

SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?

SYSTEM-BUILDING AND QUALITY:WHAT’S AT STAKE?

Charles Smith, Ph.D. Executive Director, David P. Weikart Center for Youth Program QualityVice President of Research, Forum for Youth Investment

May 9, 2013; 9:00-9:30 am


Agenda

• Quality Improvement Systems1. Building QIS

2. Site Level Process

3. System Accountabilities

• Local Models of Quality Improvement System Accountability

• Why Build Systems for Developmental Settings?

• APPENDIX


Building QIS


l

2012-2013 Dissemination

Pol

icy

Set

ting

Org

an

iza

tion

Se

ttin

gP

oint

of S

ervi

ce

Set

ting

>21,125 StaffEstimate based on mean of 6.5

staff per site in YPQI Study Sample

85 Networks/ Systems

>276,250 Child & Youth

Estimate based on mean daily attendance of 85 youth per day in YPQI

Study Sample

>3250 Sites


Building a QIS: Stages and tasks


Site Level Continuous Improvement Process


Instructional Practices“Quality” at the Point of Service Level Setting


Four Continuous Improvement PracticesOrganization Level Setting

(Plus 10 hours of TA/coaching for site managers to implement the four CI practices)


Targeted Staff Trainings for Instructional SkillsCI Practice #4


System Accountabilities


System Supports for CI Practices

Policy: Eligibility, Targeting, Low/high stakes

Training, TA & Coaching

EvaluationExternal Raters, Program Evaluation


System Accountabilities: Higher Stakes

Objective Data Publicity Action Improved

Outcomes

Higher Stakes

Accountabilities


Objective Data

Meaningful Information

Action/ Expertise

Improved Outcomes

Lower Stakes Accountabilities

Interpretive Community

•Team Self Assessment •Review external scores

Team Planning and Implementing

•Improvement planning•Performance coaching

Higher Stakes Accountabilities

System Accountabilities: Lower Stakes


Higher Stakes: System Needs and Challenges• System Needs1

– Standards beyond licensing regulations– Accountability policies based on assessment and monitoring– Program and practitioner outreach and support– Financing incentives specifically linked to compliance with

quality standards

• Challenges2

– Differences in structure and design (e.g. measures)– Lack of coordination across agencies and data systems– Policies lack clarity about goals, timing and expectations for

improvement1. National Child Care Information and Technical Assistance Center. (2009). Quality Rating Systems: Definition and Statewide Systems. Fairfax, VA: National Child Care Information and Technical Assistance Center.http://www.dataqualitycampaign.org/files/4_NCCIC_QRIS.pdf2. Tout, K., Zaslow, M., Halle, T., and Forry, N. (2009). Issues for the next decade of quality rating and improvementsystems. Washington, DC: Office of Planning, Research and Education.

http://www.dataqualitycampaign.org/files/4_NCCIC_QRIS.pdf




Local Models of Quality Improvement System Accountability


Requirements

Policy Stakes Lower Middle Higher

Accountabilities IndianapSeattle /

WAMI / OK 21

CCLCKansas City UW

VT Oakland AR QRIS

Participation Mandatory x x

Financial Incentives x x x x x x

Published Scores x x

CI Practices IndianapSeattle /

WAMI / OK 21

CCLCKansas City UW

VT Oakland AR QRIS

Self Assessment x x x x x optional x

External Assessment x x x x x x x

Planning with Data x x x x x x x

Perf Feedback & Coaching x x x x xTraining: Instruction Methods x x x x x x x


Incentives/Punishments Policy Stakes Lower Middle Higher

Incentive Structures Indianap

Seattle / WA

MI / OK 21

CCLC

Kansas City UW

VT Oakland

AR QRIS

Voluntary, emphasis on supports x x x x

Required, emphasis on supports x x x x

Required, participation monitored x x x x

Management clarity (“know what to do”) x x x x x x x

Information useful x x x x x x x

Financial incentives x x x x x x

Public ratings x x


Sources of Data for Public Ratings

• Oakland– Participation records– Program Quality Assessment (PQA)– Stakeholder surveys– Academic records– http://publicprofit.net/Services/Evaluation/

• Arkansas QRIS– Program or Business Administration Scale (PAS or BAS)– Traveling Arkansas Professional Development Registry (TAPP)– Program Quality Assessment (PQA)– School-Age Care Environmental Rating Scale (SACERS)– Various other criteria– http://www.arbetterbeginnings.com/wp-content/uploads/2011/12/

sagridswithdef.pdf

http://publicprofit.net/Services/Evaluation/


http://www.arbetterbeginnings.com/wp-content/uploads/2011/12/sagridswithdef.pdf

http://www.arbetterbeginnings.com/wp-content/uploads/2011/12/sagridswithdef.pdf


YPQI Study “Best” PracticesRequirements Accountabilities

Participation Mandatory xFinancial Incentives

Published Scores

CI Practices

Self Assessment xExternal Assessment xPlanning with Data xPerf Feedback & Coaching xTraining: Instruction Methods x

Incentives/Punishments

Voluntary, emphasis on supports

Required, emphasis on supports x

Required, participation monitored x

Management clarity (“know what to do”) x

Information useful x

Financial incentives

Public ratings


Prime Time Palm Beach County


Evolution

• Piloted QIS elements with four providers in 2010• Confirmed what we suspected: expertise in content;

opportunity to strengthen youth development• Included all enrichment programs in the modified QIS• Changed name of enrichment activities to “expanded

learning opportunities” in 2012 to reflect new expectations

• Moving to greater alignment with the school day through the Common Core framework

• Moving to more concrete learning measures and youth outcomes


Background

• Founded in 2000• Primary Areas of Service:

• Quality Improvement• Professional Development• Community Engagement and Supports

• Supported enrichment activities for nine years

• Belief that a variety of experiences is essential for positive youth development


QIS annual cycle


Entry• 12 months• Introduction to system• QA supported self-

assessment

Intermediate• 12-24 months• QA preparation of director

for maintenance role and requirements

Maintenance• Ongoing• Based on benchmark

scores and director accomplishments

QIS Level System

• Recognizes high quality programs and directors• Provides flexible time expectations based on needs


Why Build Systems for Developmental Settings?

A Frame for Developmental Systems


Frames• Positive Youth Development

– Substitution– Skill building

• Systems as…– Protective factors (Fragmentation = risk)– School reform (“Expanded Learning”)

• Reinventing Government / Social Sector– Regulating core processes (instead of inputs)– Building performance cultures


SYSTEM-BUILDING AND QUALITY:WHAT’S AT STAKE?

Charles Smith, Ph.D. Executive Director, David P. Weikart Center for Youth Program QualityVice President of Research, Forum for Youth Investment

[email protected]://cypq.org/ypqi

mailto:[email protected]

http://cypq.org/ypqi


APPENDIX


High Stakes Examples


Higher Stakes Models

• Oakland requires participation, scores are tied to funding, and reports go to the city government

• AR’s system is voluntary, but once in it, scores feed into 3 tier system that are used for incentive grants and published ratings for families

Policy Stakes Higher

Accountabilities Oakland AR QRIS

Participation Mandatory x Financial Incentives x xPublished Scores x xCI Practices Oakland AR QRIS

Self Assessment optional xExternal Assessment x xPlanning with Data x xPerf Feedback & Coaching x xTraining: Instruction Methods x x


Oakland Unified School District (OUSD) and Oakland Fund for Child and Youth (OCFY)

• http://publicprofit.net/Services/Evaluation/• OUSD and OCFY are funders• Public Profit is the intermediary and evaluator• System is voluntary for self assessment, required for external

assessment and planning• Process:

– All sites get external assessments of 2 program offerings– All sites receive individualized “planning with data” type meetings with the

evaluators, who go over their scores and work with them to create improvement plans

– Reports of external assessments are also sent to city government – All sites have access to Methods trainings– Programs with low scores receive additional coaching

• Incentives:– Programs with extremely low scores (scale scores under 2) that don’t improve

over the course of 2-3 years could lose their funding– No sites have lost funding almost no programs score that low




Arkansas Quality Rating and Improvement System (QRIS)• http://www.arbetterbeginnings.com/• Arkansas State University serves as intermediary for system for school-age care

programs• System is voluntary• Process:

– A three-tiered rating system that the PQA scores feed into, but it is only one of multiple measures.

– Components included in the ratings are:• Administration• Administrator/Staff Qualifications & Professional Development• Learning Environment• Environmental Assessment• Child Health & Development

• Incentives:– Ratings are used to offer both incentive grants and published ratings for families to use to

decide where to send their children– Incentive Grants are available upon meeting certification standards at each of the 3 levels

• At level 1 and level 2, it is renewable for a maximum of 9 years (not to exceed 6 years at either level 1 or level 2).

• At level 3, the Incentive Grant is available annually, as long as the facility continues to meet the standards.

• Incentive grant amounts are based on a combination of licensed capacities, current Level and the number of years spent at that level.

http://www.arbetterbeginnings.com/


Middle Stakes Examples


Middle Stakes Models

• MI/OK require process to maintain funding, but focus on supports and coaching

• Kansas City has 3 tier incentive system based on completion of YPQI elements to get funding at the different levels

• VT has 5 tier recognition program based on various practices where programs get funding, public awareness, discounts, and funding opportunities

Policy Stakes Middle

Accountabilities MI / OK 21 CCLC

Kansas City UW VT

Participation Mandatory x Financial Incentives x x xPublished Scores CI Practices MI / OK

21 CCLCKansas

City UW VT

Self Assessment x x xExternal Assessment x x xPlanning with Data x x xPerf Feedback & Coaching x xTraining: Instruction Methods x x x


Kansas City United Way

• http://www.unitedwaygkc.org/nonprofits/qualitymatters.html• UW is the intermediary, partners with Francis Institute for coaching

and University of Missouri, Kansas City for evaluation/external assessments

• System is voluntary• Half the sites that participate are United Way funded programs, but

United Way funds the YPQI process for all sites• Process:

– Have high fidelity to YPQI assess-plan-improve, with all sites doing assessments, planning and receiving coaching

• Incentives:– Have a 3 tier incentive system that is based on completion of the elements of the YPQI,

and they receive $300, $500 and $750 accordingly– Example: Participation Level 3: $750 – Completion of:

• Conduct a fall team based PQA self-assessment which includes observation and team consensus and enter data into the Online Scores Reporter by [DATE].

• Program Improvement Plan created and entered into Online Scores Reporter by [DATE].• 75% completion rate of goals set in Program Improvement Plan**by [DATE]• Attend 7 different workforce training (1-4 site staff may attend each training and must stay

for entire session) by [DATE]

http://www.unitedwaygkc.org/nonprofits/qualitymatters.html


Michigan 21st CCLC and Oklahoma 21st CCLC• http://www.michigan.gov/mde/0,4615,7-140-6530_6809-39974--,

00.html• http://www.ok.gov/sde/21cclc• Similarly structured systems, they are both funder and

intermediary• Participation is required since it is tied to funding• Process:

– Data from both the PQA and evaluation are incorporated into the QIS process

– Coaches are key component, offering comprehensive services to select programs

• Incentives:– Focus is on implementation and improvement supports– Require that all sites complete full process in order to maintain good

standing on grant– Scores are not used punitively, but can be used to target coaching

services

http://www.michigan.gov/mde/0,4615,7-140-6530_6809-39974--,00.html

http://www.michigan.gov/mde/0,4615,7-140-6530_6809-39974--,00.html

http://www.ok.gov/sde/21cclc


Vermont Center for Afterschool Excellence

• http://www.vermontafterschool.org/• http://dcf.vermont.gov/cdd/stars• Are intermediary, they serve both 21st CCLC and AHS/QRIS programs• Both systems encourage use of the YPQI, but is voluntary• Process:

– Have a 5 tier star system for recognition of programs, based on practices in these areas:• Compliance with state regulations• Staff qualifications and training;• Interaction with and overall support of children, families, and communities;• How thoroughly providers assess what they do and plan for improvements; and• The strength of the program’s operating policies and business practices.

• Incentives: – The benefits for the star system, tiered based on number of stars the program has earned:

• The Child Care Financial Assistance Program (CCFAP) pays a higher rate on behalf of families.

• Bonus payments for EACH level achieved:• Public awareness of STARS participation if requested. Options include: listing on the

STARS website, supply of STARS brochures, and a customized press release.• The opportunity to apply for grants open only to programs that are in STARS or are

nationally accredited• Discount on purchases from a list of corporate sponsors.

http://www.vermontafterschool.org/

http://dcf.vermont.gov/cdd/stars




Lower Stakes Example

Indianapolis Marion County Commission on Youth (MCCOY)


Lower Stakes Models

• Main incentive is access all supports, learning community, flexibility and choice

• MCCOY is very low stakes, all parts are suggested but optional

• Raikes also offers funding to programs to subsidize participation

Policy Stakes Lower

Accountabilities Indianap Seattle / WA

Participation Mandatory Financial Incentives xPublished Scores CI Practices Indianap Seattle /

WA

Self Assessment x xExternal Assessment x xPlanning with Data x xPerf Feedback & Coaching xTraining: Instruction Methods x x


Indianapolis Marion County Commission on Youth (MCCOY)

• http://www.mccoyouth.org/• MCCOY is intermediary• Very, very low stakes--Participation is entirely voluntary• Process

– Sites do the full Assess-Plan-Improve sequence– Sites can do self and external assessment, usually only once a year, but could do

more– No coaching or training in instructional coaching– Methods workshops are offered for all sites– Recruitment can be hard, but they focus on partnerships and Methods to get sites

engaged– Programs choose to join cohorts (2-3 cohorts per year)– Programs can choose to participate multiple times but there is no emphasis on

tracking year to year improvement.

• Incentives– Programs have access to supports and learning community, and a lot of flexibility in

choosing how to improve

http://www.mccoyouth.org/

http://www.mccoyouth.org/


Seattle & Washington State- Raikes Foundation and School’s Out Washington• http://raikesfoundation.org/Secondary.aspx?file%3daboutmission• http://www.schoolsoutwashington.org/index.htm• Raikes Foundation is the funder• Schools Out Washington is the intermediary, with other local supports• Has funded 1-3 cohorts across the state since 2008 • System is voluntary, programs apply• Programs span funding and accountability streams• Process

– There is an intensive application process that comes with funding to programs to subsidize their participation and all of the supports are free

– High level of fidelity to YPQI– Sites get less and less supports over a 3 year period…beyond that they can apply

for funding to get a al carte services

• Incentives:– Participation comes with funding to programs to subsidize their participation and

all of the supports are free

http://raikesfoundation.org/Secondary.aspx?file=aboutmission

http://raikesfoundation.org/Secondary.aspx?file=aboutmission

http://www.schoolsoutwashington.org/index.htm


YPQI Evidence


CQI Systems: Cross-Level RolesPo

licy

Sett

ing

Org

an

izati

on

S

ett

ing

Poin

t of

Serv

ice

Sett

ing

…engages in continuous improvement practices

…enacts continuous improvement practices

…engages standards and supports

…enact standards and supports

Youth engage in instruction and build skills

…enacts instructional practices

Settings Actors Behaviors

Network Leaders

Managers

Staff

Youth


0102030405060708090

100

Assess instruction quality

Create improv plan Coach staff on instruction

Meet with a TA/coach

Perc

ent o

f Man

ager

sManagement CI SkillsYPQI Study Baseline


0

0.2

0.4

0.6

0.8

1

Full Imp Mid Imp Non-imp

Mea

n Sco

re

Attend training Conduct program planning Conduct observation with tool

State of the Field• 10% of staff were engaged

in all CI practices• 22% were not engaged in

CI practices• 68% were engaged in some practices

Note: Profile of 3 exemplary clusters from an 6 cluster solution

Staff CI SkillsN=366, YPQI Study Baseline


Staff Instructional Skills3

N= 600 different youth workers and teachers

1

2

3

4

5

PYD I Staff Cent I Low Qual II

Q u a l i t ywelcome

inclusion

active learning

grouping

choices

planning

reflect

Occurred For All

Occurred For Some

Did Not Occur

Positive Youth DevelopmentN=166, 28%

Staff CenteredN=231, 39%

Low Quality N=193, 33%


CQI Systems: Cross-Level RolesPo

licy

Sett

ing

Org

an

izati

on

S

ett

ing

Poin

t of

Serv

ice

Sett

ing

…engages in continuous improvement practices

…enacts continuous improvement practices

…engages standards and supports

…enact standards and supports

Youth engage in instruction and build skills

…enacts instructional practices

Settings Actors Behaviors

Network Leaders

Managers

Staff

Youth

ES =.98

ES =.52

ES =.55

ES =1.87


0

10

20

30

40

50

60

70

80

90

100

Program Managers Direct Service Staff

Participant Satisfaction

N=128 site managers, 178 staff from Atlanta, Baltimore, Chattanooga, Maryland, Nashville, Richmond, Vermont, Washington

SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE? Charles Smith, Ph.D. Executive Director, David P....

Documents

Transcript of SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE? Charles Smith, Ph.D. Executive Director, David P....