Improving Performance Managementthrough the 

Next Generation of ROMA

NASCSP Conference I September 2014

Focus on Performance

EOA of 1964 OEO Instruction on CAA Mission of 1969 Voluntary reporting GPRA Act of 1993 COATS Human Services Act of 1998 GPRA Modernization Act of 2010 Accountable Government Initiative Federal Funding Accountability Act (FFATA) Head Start Competition Focus on Evidence

2

ROMA COE Overview

Key ROMA COE goals are to…

Increase Network capacity to execute the full ROMA cycle

Increase Network capacity to collect, report, and use performance data for decision making

3

Snapshot of Progress

ROMA Implementation Surveys

National Community Action Theory of Change

Local TOC Toolkit (Draft)

White Papers (3)

National Webinars, National Listening Sessions, Statewide Listening Sessions (over 50)

National Surveys on Proposed NPI Revisions (2)

Interim Pilot of Proposed NPI Revisions (9 states, 37 CAAs)

Case Study on Bundled Services (3 states)

Four ROMA COE Committees 175 members

State CSBG Offices, State CAA Associations, CAAs, National Partner Organizations, Network Consultants

4

Improved Performance Management

1. ROMA Next Generation is about strengthening the performance management culture of Community Action. 

2. ROMA Next Generation is about more robust data analyses to enable agencies to more effectively connect people and distressed communities with the services needed to achieve specific outcomes. 

5

Improved Performance Management

3. ROMA Next Generation asks agencies and states to establish a better understanding of what services produce the best outcomes for specific family groups and communities. 

4. ROMA Next Generation will ensure that agencies and states are using their data to focus their limited resources on what works best to achieve Community Action’s long‐term goals of self‐sufficiency and revitalized communities. 

6

CONSIDERING OUTCOMESWhat services work?

7

Service

Outcome

One Service ‐> One Outcome

.

Service

Service

Service

Service

Outcome

Multiple Services ‐> One Outcome

.

Service

Outcome

Outcome

Outcome

Outcome

One Service ‐> Multiple Outcomes

.

Service

Service

Service

Service

Service

Outcome

Outcome

Outcome

Outcome

Multiple Services ‐> Multiple Outcomes

.

Family Level Data Analysis: Example

People Services Outcomes

Case #1 Unemployed Food Assistance Obtained a job and maintained it for “X” days

Case #2 Unemployed Job Training Obtained a job and maintained it for “X” days

Case #3 Unemployed Food assistance,Child Care, Job Training, Education, Transportation

Obtained a job and maintained it for “X” days

12

Elements of Data Analysis (Family Level)

ROMA Next Generation connects outcomes, services and clients to facilitate robust analysis.

Outcomes: Obtained jobs, completed education, increased income, improved parenting, secured housing 

Services: Job Training, Work Experience, Scholarships, Financial Education, Support Group, Case Management, Housing Counseling, Food Assistance 

Population: Demographic information, income data, service and outcome data, exit dates, follow up.

13

DATA ANALYSIS USING THE ROMA CYCLE

14

15

WhatOutcomes DoWeWanttoAchieve?

NEEDSASSESSMENT

IncreasedStabilityandSelf‐SufficiencyofFamilies &RevitalizedCommunities

PLANNINGWhatPartnershipsDoWeNeedtoMakeToAchievetheOutcomes?

IMPLEMENTATION

WhatServices DoWeNeedtoBundleto BetterAchievetheOutcomes?

WhatData DoWeNeedtoCollect/Use?

ServicesProvidedtoLow‐IncomeIndividuals/Families

StrategiesImplementedtoChangeConditionsintheCommunity

PerformanceDataIsCollectedIndividualsServedAreTrackedServicesProvidedAreTracked

What Services Do We Need to Provide to Achieve the Outcomes?

WhatdifferencedidtheNetworkmake?

16

DATAANALYSIS/EVALUATION

Whichindividualsachievedparticularoutcomes?

Whatservicesweremostusedbyindividualsachievingtheoutcomes? Whatspecificmixofserviceswasmostusedbyindividualsachievingtheoutcomes?

Howwelldidwedeliverservices?(Timeliness,Accuracy,Cost,ClientSatisfaction)

Whatcanwesurmiseaboutwhatservicesaremosteffectiveforachievingtheexpectedoutcomes?

WhichNPIsdidtheagencymeet?Notmeet?

Whatservicesweremostusedbyindividualsnotachievingtheoutcomes? Didtheindividualsnotachievingtheexpectedoutcomesusedifferentservices?Notusespecificservices?

WhatchangesshouldwemaketoourServiceModel?Whatdomoreof,lessof?Anyservicestostopdelivering?Anyservicestodeliverdifferently?Any

servicestoadd?Doweneedtoimprovehowwedeliverservices?

DATA ANALYSIS/EVALUATION (CAAs)

17

WhatOutcomes(NPIs)weremetbyeachofthelocalagenciesacrossthestate?WhatOutcomeswerenotmet?

Isthedatareceivedfromthelocalagenciesaccurate?Reliable?(Unduplicatedcountsofindividualsserved?)

DATAANALYSIS/EVALUATION

Whattypesofserviceswereprovided?Howmanyofthesetypesofserviceswereprovidedacrossthestate?

Howmanypeoplewereservedacrossthestate?

Whatdoesthedatatellusaboutanypatternsandtrendsre:servicedeliveryandoutcomesachieved?

Takentogether,whatoutcomes(NPIs)wereachievedacrossthestate?

InSummary:WhatdidCommunityActionachieveinourState?“InthisState,weachievedXResults(Outcomes)forthismanylow‐incomeindividuals/families/communities(Output)bydeliveringtheseservices(Output),atthisCost.”

Whatservicedeliverymodelsappearmosteffective?

Whatlocalagencybestpracticesshouldbereplicatedinotherpartsofthestate?

Whatlocalagenciesshouldreceiveenhancedtrainingandtechnicalassistance?

DATA ANALYSIS/EVALUATION (States)

Where We’re Headed

Refined listing for national aggregation (Family, Community) Some NEW indicators

Measuring progress over time

Refined definitions Enhancing reporting quality and accuracy 

Refined focus on achievement  Six National ROMA Goals 

Self‐Sufficiency and Revitalized Communities

Focus on proving National Community Action TOC assumptions Continued focus on independent living for vulnerable populations Continued focus on Community Action’s role in creating community change Continued focus maximum feasible participation in the community

Please note the NPIs are still under development and review

18

Family Level NPIs

19

The Big Questions

What’s your impact on the lives of individuals and families? Are people better off as a result of Community Action 

intervention? What actually happened (what changed) to an individual or 

family as a result of the services you provided?

If Community Action is about moving families to self‐sufficiency, how many families have achieved self‐

sufficiency?

If families have not “moved” but have received services to maintain or achieve family stability, how do you measure 

this accomplishment? 

20

Basic Data Needed

How many individuals received services?  Of those served, how many received single or short term services?  

How many received multiple services?  Were the services integrated?  What were the services?

For how many of the total individuals served do you have follow up information? What outcomes did they achieve? What changed? REPORT VIA NPIS

Of the number who received services for whom you have follow up information, what percent achieved the outcome?  How does the percent that achieved the outcome compare to the 

number you  targeted  would achieve an outcome?

21

Family NPIs: National Reporting

NEW! The number and percent of individuals for whom family income increased.  The information secured on intake for Section G is collected 

again (second or multiple times) and compared with original information.

The number and percent of individuals who increased savings.  

NEW! The number and percent of individuals who reduced debts (liabilities). 

22

Family NPIs: National Reporting

The number and percent of individuals who obtained health insurance. 

The number and percent of individuals who obtained a job. 

Of the above, the number and percent who retained employment for at least 90 days.

Of the above, the number and percent who retained employment for at least 180 days. 

23

Family NPIs: National Reporting, cont.

The number and percent of adult individuals who increased education or skills: The number and percent who completed high school or 

post‐secondary education and obtained a certificate or diploma.

The number and percent who improve skills related to adult roles of parent/caregiver, citizen, worker.  

The number and percent of children and youth who achieved or maintained  appropriate developmental progress (social, emotional, academic).

24

Family NPIs: National Reporting, cont.

The number and percent of individuals who obtained safe and affordable housing.   

Of the above, the number and percent who maintained safe and affordable housing for 90 days.

Of the above, the number and percent who maintain safe and affordable housing for 180 days.

The number and percent of individuals who obtained reliable transportation. 

25

Family NPIs: National Reporting, cont.

The number and percent of seniors and individuals with disabilities who achieved or maintained an independent living situation.

The number and percent of individuals who were in a crisis and became stable, or who averted a crisis.  

26

Making Connections, Improving Impact 

Employment Data Analysis Example:  The agency can identify the various employment services 

received by a program participant

The agency can connect employment program participants with the services they receive in the other domain areas

The agency can identify whether employment program participants received only employment services, received services in other domain areas while also receiving employment services, or received employment services as a part of a “bundled services” approach.

27

Making Connections, Improving Impact 

The agency can identify employment and non‐employment outcomes achieved by those participating in employment related services

The agency can identify the average duration and intensity of services required before employment is obtained

The agency can connect employment program participant characteristics, or demographics, with services and outcomes

28

Making Connections, Improving Impact 

Emergency and Stabilization Data Analysis Example:

The agency can identify the number and percent of individuals achieving stability as evidenced by their ability to meet family basic needs.

The agency can identify the number and percent of individuals who entered a self‐sufficiency program or service as a result of stability or emergency service provision. (To develop evidence proving TOC assumptions)

29

Community Level NPIs

30

The Big Questions

What’s your impact on communities? 

Are communities better off as a result of Community Action intervention? 

What actually happened (what changed) to the community as a result of the services you provided?

Are you a part of “collective action” with partners in your community?  

What evidence do you have to prove your part in the community level success?

31

Community NPIs: National Reporting

The number of jobs within low‐income neighborhoods.

The number of safe and affordable housing units within low‐income neighborhoods.

The number of community facilities within low‐income neighborhoods.

The number accessible/affordable resources (ex. health, commercial, transportation, education, etc.) within low‐income neighborhoods.

32

Community NPIs: National Reporting, cont.

The number of community members working with Community Action to improve their delivery of service and/or conditions in the community. The number of hours donated. Of the above, the number who are Community Action program participants. The number of hours donated.

Of the above, the number of Community Action program participants who become engaged in a leadership role. The number of hours donated.

33

How Do We Get There?

Full implementation of the ROMA Cycle

Performance Management Principles Agency Level

State Level

It’s about enhanced DATA ANALYSIS

Its about investing in people as well as systemsHigh performance is primarily a function of people and culture, 

not data and technology. ~ Mario Marino, Leap of Reason

34

Performance Management Principles

Agencies implement the full ROMA Cycle (i.e. assessment, planning, implementation, achievement of results, and analysis/evaluation) on a continual basis

Agencies have protocols to manage the quality and effectiveness of their services

Agencies understand what’s working and what’s not working. They monitor their data and use it to make adjustments for improvement?.

35

Performance Management Principles, cont.

Agencies use the bundled services delivery model, where appropriate

Agencies connect critical data from disparate systems to meet diverse program needs and reporting requirements

Agencies connect data elements that can be aggregated, analyzed and reported quickly and efficiently

36

Performance Management Principles, cont.

Agencies achieve an unduplicated count without comprising program participant privacy

States monitor agency performance management

States use data to compare agency performance, statewide, and replicate best practices

37

Section E ‐ Use of CSBG Dollars

38

The first question

What does the CSBG $ buy?How do agencies (states) assign CSBG $ use to Service Categories? Direct tracking to expenses that are identified by service category Assignment of expenses to categories based on percents

39

Service Categories

Refined Listing and Definitions

Case Management/Family Development Education  Employment  Income Management  Housing   Health Nutrition Emergency and Stabilization Linkages Disaster Relief Organizational Capacity Building Admin

40

Case Mgmt./Family Development

Case Management

“Case management" meaning a "generic" case management rather than case management for one of the service categories, such as HUD Housing programs

If we are trying to benchmark our networks “bundled services” we need to clarify where these kinds are services are being reported.

41

Linkages

Linkages = Community Level Activities Money to support participation in planning and/or coordinating services for low‐income groups should be included in the Linkages service category

Statewide brokerage/advocacy projects, statewide study grants, and funding for statewide forums, conferences, or coalitions should be included

42

Organizational Capacity Building

Agency Capacity Cost of doing needs assessment

Cost of doing strategic planning

Support of tripartite board

Support of community engagement and input

Development of partnerships and coalitions

Development of community level strategies

43

Connecting CSBG $ to Outcomes

Suggestions:

Add (to the Guide/IS directions) suggestions about making connection between service categories and NPIs

Identify existing models already being used

Add a way to allow reporting of NPIs to include a question about use of CSBG $ in support of the outcome identified

44

For more information about the ROMA Next Generation Center of Excellence, visit: 

www.roma1.org