National Outcome Measures:Using Data to Show the Way

Forward

Second Canadian Roundtableon Child Welfare Outcomes

Fred Wulczyn, Ph.D.

slide 2

Objectives

You have the data - now what? Methods/analysis - getting a clearer picture

Risk or case mix adjustment

Better methods/better questions

Analysis - using the data to manage (reduce and induce) variation Linking outcomes and finance

Admission, duration, and unit cost strategies

slide 3

BD:Before there was data

Fact or Fiction - how much did it matter?

slide 4

AD:After we have data

slide 5

Develop a theory of change(Tell a story but no more fiction)

Look for differences This <fill in the outcome> happens more

often for this group

This <fill in the outcome> isn’t as common in this part of the province

Develop an intervention or interventions If we do this <fill in the intervention> this

will happen more often

Asking good questions is important

slide 6

What is this notion of risk or case mix adjustment?

Why does performance vary? Provinces

Providers

Children and families

Why do we care? Promotes better theories of etiology and

intervention

Targeting may improve effectiveness

slide 7

Incidence per 1000:The Likelihood of Placement

0.00

0.50

1.00

1.50

2.00

2001 2002 2003 2004 2005 2006 2007 2008

Year

Rate

slide 8

Variation in IncidencePlace and Age

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

16.00

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Single Year of Age

Rates per 1000

2001

2002

2003

2004

2005

2006

2007

2008

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

16.00

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Single Year of Age

Rates per 1000

2001

2002

2003

2004

2005

2006

2007

2008

slide 9

Placement Stability - Creating a better measure

Two views of stability Who is at risk?

When is the risk highest?

Moves per child addresses the first

Moves per day does a better job with the first but not the second.

slide 10

Ask ‘Richer’ Questions:3 view of stability

Percent Moved

44%

46%

48%

50%

52%

54%

56%

58%

0 to 1 1 to 5 6 to 12 13 to 17

Percent Moved

44%

46%

48%

50%

52%

54%

56%

58%

0 to 1 1 to 5 6 to 12 13 to 170

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

Percent MovedMoves/Day

0 . 0 0

0 . 1 0

0 . 2 0

0 . 3 0

0 . 4 0

0 . 5 0

0 . 6 0

0 - 6 7 - 1 2 1 3 - 1 8 1 9 - 2 4 2 5 - 3 0 3 1 - 3 6 3 7 - 4 2 4 3 - 4 8 4 9 - 5 4 5 5 - 6 0 6 1 - 6 6 6 7 - 7 2

M o n t h s i n C a r e

I n f a n t s

1 t o 5

6 t o 1 2

1 3 a n d o l d e r

Percent of children Moves per 100 days Moves per 100 days by time in care who moved at leastonce

slide 11

Monitoring

Looking for whether the changes you made induced the change you were looking for

Depends on a baseline

slide 12

Time to PermanencyMedian Duration

0.00

5.00

10.00

15.00

20.00

25.00

30.00

2001 2002 2003 2004 2005 2006 2007 2008

slide 13

Digging DeeperHow important is it?

A hypothetical:Two counties each working to

improve permanency.

The counties have selected different strategies.

After a period of time, the public agency pulls everyone together.

The question: with the resources we have, which intervention represents the better bet?

Vital statisticsChildren served: 350 in each

Success

50%

60%

70%

80%

90%

100%

County A County B

Permanency rate

slide 14

Conclusion:

Invest resources in the County B intervention because the permanency rate is higher.

Children will go home sooner and tax dollars will be spent more wisely

Everybody is a winner.

slide 15

What a minute:Are you sure?

What gives?Permanency rates in County A

are actually higher.

Why the difference? County B serves more Type A children.

Type A children have higher overall permanency rates than Type B children; thus the aggregate data show a different result.

New ConclusionCounty A’s program is probably

the better bet

50%

60%

70%

80%

90%

100%

Type A Type B

Permanency rate

County ACounty B

slide 16

Return on Investment -Shifting the dialogue

Spending vs. investment

‘Purchase’ more of what works, less of what does not

Underscores the importance of outcomes Know what and why you do what you do

Underscores the importance of data for looking back in time and for setting goals and expectations.

slide 17

Basic Fiscal Model

Revenue = Units * Unit Cost

slide 18

Link Outcomes and Funding

EXPENDITURE

$500,000,000

$1,000,000,000

$1,500,000,000

$2,000,000,000

$2,500,000,000

$3,000,000,000

$3,500,000,000

$4,000,000,000

$4,500,000,000

2008 2009 2010 2011 2012

CALENDAR FISCAL YEAR

BaselineDischarge StrategyAdmission Strategy

CHILDREN IN CARE

12,000

32,000

52,000

72,000

92,000

112,000

132,000

152,000

CFY 2006 CFY 2007 CFY 2008 CFY 2009 CFY 2010

CALENDAR FISCAL YEAR

BaselineDischarge StrategyAdmission Strategy

slide 19

Take Away

Gathering data is only the start Collecting data begs the question:

“What do you mean?” Data dramatically improves the

possibility of success Data is central to the rights of children

and families