Developing a Resource Guide for CANS Data Analysis and Reporting Vicki Sprague Effland, Ph.D.

Developing a Resource Guide for CANS Data Analysis and Reporting

Vicki Sprague Effland, Ph.D.

Youth Improved!

Youth St

rength

s

Life D

omain Fu

nctioning

Accultu

ration

Child Beh

avioral

/Emotional

Needs

Child Risk

Behav

iors

Caregiv

er Fu

nctioning

0

2

4

6

8

10

12

14

16

18

Dimension Scores

IntakeDischarge

Living Situa-tion

Permanence Placement Stability

Attendance Behavior Achievement0

2

4

6

8

10

12

14

16

18

Item Scores - Life Domain Functioning

IntakeDischarge

Impulsiv

ity/H

ypera

ctivit

y

Depres

sion

Anxiety

Oppositional

Conduct

Trauma

Anger C

ontrol

Attachmen

t

Judgmen

t

Socia

l Beh

avior

0

2

4

6

8

10

12

14

16

Item Scores - Child Behavioral and Emotional Needs

IntakeDischarge

Suici

de Risk

Self M

utilation

Other Se

lf Harm

Dange

r to O

thers

Sexu

al Agg

ressio

n

Runaway

Fire S

etting

0

1

2

3

4

5

6

Item Scores - Child Risk Behaviors Needs

IntakeDischarge

Family

Superv

ision

Involve

ment

Knowledge

Organiza

tion

Family

Stres

s

Socia

l Reso

urces

Safet

y0

2

4

6

8

10

12

14

16

Item Scores - Caregiver Functioning

IntakeDischarge

Youth St

rength

s

Life D

omain Fu

nctioning

Child Beh

avioral

/Emotional

Needs

Child Risk

Behav

iors

Caregiv

er Fu

nctioning

Youth Fu

nctioning

Youth O

veral

l0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Any Improvement

Living Situa-tion


Attendance Behavior Achievement0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Improved - Life Domain Functioning

Impulsiv

ity/H

ypera

ctivit

y

Depres

sion

Anxiety

Oppositional

Conduct

Trauma

Anger C

ontrol

Attachmen

t

Judgmen

t

Socia

l Beh

avior

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Improved - Child Behavioral and Emotional Needs

Suicide Risk Self Mutilation Other Self Harm Danger to Others Sexual Aggression0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Improved - Child Risk Behaviors

Family

Superv

ision

Involve

ment

Knowledge

Organiza

tion

Family

Stres

s

Socia

l Reso

urces

Safet

y0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Improved - Caregiver Functioning

Youth St

rength

s

Life D

omain Fu

nctioning

Child Beh

avioral

/Emotional

Needs

Child Risk

Behav

iors

Caregiv

er Fu

nctioning

Youth Fu

nctioning

Youth O

veral

l0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Reliable Change

Youth St

rength

s

Life D

omain Fu

nctioning

Accultu

ration

Child Beh

avioral

/Emotional

Needs

Child Risk

Behav

iors

Caregiv

er Fu

nctioning

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

Actionable Needs

IntakeDischarge

Youth St

rength

s

Life D

omain Fu

nctioning

Accultu

ration

Child Beh

avioral

/Emotional

Needs

Child Risk

Behav

iors

Caregiv

er Fu

nctioning

0.0

1.0

2.0

3.0

4.0

5.0

6.0

# Needs Met

Youth St

rength

s

Life D

omain Fu

nctioning

Accultu

ration

Child Beh

avioral

/Emotional

Needs

Child Risk

Behav

iors

Caregiv

er Fu

nctioning

0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%90.0%

100.0%

% Needs Met

Youth St

rength

s

Life D

omain Fu

nctioning

Accultu

ration

Child Beh

avioral

/Emotional

Needs

Child Risk

Behav

iors

Caregiv

er Fu

nctioning

0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%90.0%

100.0%

% Youth Met Need(s) in Dimension

Living Situa-tion



10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Met Needs - Life Domain Functioning

Impulsiv

ity/H

ypera

ctivit

y

Depres

sion

Anxiety

Oppositional

Conduct

Trauma

Anger C

ontrol

Attachmen

t

Judgmen

t

Socia

l Beh

avior

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Met Needs - Child Behavioral and Emotional Needs

Suicide Risk Self Mutilation Other Self Harm Danger to OthersSexual Aggression0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Met Needs - Child Risk Behaviors

Family

Superv

ision

Involve

ment

Knowledge

Organiza

tion

Family

Stres

s

Socia

l Reso

urces

Safet

y0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Met Needs - Caregiver Functioning

Youth Improved!

Did Youth Improve Enough?

Need for Resource Guide

• Standardize methodology for CANS data analysis

• Establish benchmarks for various data analysis methods

• Develop guidelines for reporting CANS results

Introduction to Choices

Choices, Inc.

• Non profit care management entity created in 1997

• Developed around a community need: “high cost youth”

• Blended system of care principles with wraparound values and managed care technology.

Choices Care Management •More than 220 employees

• $35 million annual budget

• More than 1300 youth served in child and family teams daily

• Working across ALL child serving systems – 60% child welfare

Indiana Choices – Since 1997Maryland Choices – Since 2005DC Choices – Since 2008Louisiana Choices – Since 2012

Choices, Inc.

• Adopted CANS in 2006– Comprehensive version– 12 Life Domains

• Outcomes Champion – Agency in 2007

Outcomes Monitoring

• Internal– Program effectiveness– Quality improvement

• External– Adherence to contract requirements– Marketing to new partners and communities

Successes

• Have lots of CANS data• Multiple resources to analyze and report

data– Outcomes and evaluation– Software development– Communications

• Ability to look at trends over time

Challenges

• Difficult to compare our performance to others– Multiple versions of the CANS– Variation in how CANS is analyzed– Multiple tools used across communities

• Need to establish meaningful performance expectations– Minimum levels of change– % youth expected to improve

Important Points about the CANS

Critical Elements of Communimetrics Measures

1. Partner Involvement2. Malleable to the Organization3. Just Enough Information Philosophy4. Meaningfulness to Decision Process5. Reliability at Item Level6. Utility of Measure Based on its

Communication Value

“Unlike psychometric measures in which clinical significance is a more rigorous standard than statistical significance, any change on the CANS is clinically significant.”

- Lyons (2009), Communimetrics: A Communication Theory of Measurement in Human Service Settings

Family & Youth Program System

Decision Support

Service Planning

Eligibility Resource Management

Quality Improvement

Case Management

& SupervisionAccreditation Transformation

Outcome Monitoring

Service Planning &

CelebrationsEvaluation Performance

Contracting

Total Clinical Outcomes Management

Methods for Analyzing the CANS

• Dimension-Level Analyses

• Item-Level Analyses

Dimension-Level Analyses

Change in Dimension Scores• Analysis Steps

1. Sum items in a specified dimensions2. Divide by the number of valid responses3. Multiply by 104. Conduct statistical analysis

Change in Dimension Scores• Reporting Results– Intake and discharge means– Results of statistical analysis– Statistically significant change in scores between

intake and discharge

• Benchmarks– Accepted statistical criteria– None available for clinical significance

Youth St

rength

s

Life D

omain Fu

nctioning

Accultu

ration

Child Beh

avioral

/Emotional

Needs

Child Risk

Behav

iors

Caregiv

er Fu

nctioning

0

2

4

6

8

10

12

14

16

18

Dimension Scores

IntakeDischarge

Change in Dimension Scores• Advantages– Uses well known statistical methods– Statistical significance has a commonly understood

meaning• Disadvantages– Statistical significance not always indicative of

clinical significance– Does not communicate results in terms of number

of youth showing improvement

Any Improvement in Functioning• Analysis Steps

1. Calculate intake and discharge mean scores2. Identify youth with lower scores at discharge• Intake Mean Score > Discharge Mean score

3. Divide by # youth in sample

Any Improvement in Functioning• Reporting Results– % of youth with any improvement in functioning

• Benchmarks– N/A

Youth St

rength

s

Life D

omain Fu

nctioning

Child Beh

avioral

/Emotional

Needs

Child Risk

Behav

iors

Caregiv

er Fu

nctioning

Youth Fu

nctioning

Youth O

veral

l0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Any Improvement

Any Improvement in Functioning• Advantages– Simple to analyze– Easy to explain methodology

• Challenges– Lack of established benchmarks– Difficult to communicate that change is clinically

meaningful

Reliable Change• Equation

– RCI = 1.28 * SD * SQRT(1 – Reliability)• Analysis Steps

1. Compute the RCI2. Calculate change in intake and discharge mean

scores3. Identify youth with change in scores >= RCI4. Divide by # youth in sample

Reliable Change• Reporting Results– % of youth with a reliable improvement in

functioning• Benchmarks– 60-80% of youth expected to improve in at least

one of the dimensions measured– 20-40% of youth expected to improve in a

specific dimension

Youth St

rength

s

Life D

omain Fu

nctioning

Child Beh

avioral

/Emotional

Needs

Child Risk

Behav

iors

Caregiv

er Fu

nctioning

Youth Fu

nctioning

Youth O

veral

l0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Reliable Change

Reliable Change• Advantages– Clearly defined method– Available benchmarks

• Challenges– Difficult for program staff to interpret and

communicate results– Results include youth with no needs at intake

Actionable Needs• Analysis Steps

1. Count the number of needs rated as a 2 or 3 within each dimension

2. Compare needs at Intake and Discharge

Actionable Needs• Reporting Results– Average number of needs at intake and discharge

across dimensions• Benchmarks– N/A

Youth St

rength

s

Life D

omain Fu

nctioning

Accultu

ration

Child Beh

avioral

/Emotional

Needs

Child Risk

Behav

iors

Caregiv

er Fu

nctioning

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

Actionable Needs

IntakeDischarge

Actionable Needs• Advantages– Easy to display graphically– Simple for audiences familiar with the CANS to

understand• Challenges– Requires additional explanation if audience

includes individuals not familiar with the CANS– Lack of established benchmarks

Met Needs• Analysis Steps

1. Identify youth with ratings of 2 or 3 on individual items at Intake

2. Determine whether item ratings decreased to a 0 or 1 by Discharge

3. Compute the number and percent of items met within each dimension

4. Calculate the percent of youth who met at least one (or more) needs within the dimension

Met Needs• Reporting Results– Average number of needs met by dimension– Percent of needs met– Percent of youth who met at least one need


Youth St

rength

s

Life D

omain Fu

nctioning

Accultu

ration

Child Beh

avioral

/Emotional

Needs

Child Risk

Behav

iors

Caregiv

er Fu

nctioning

0.0

1.0

2.0

3.0

4.0

5.0

6.0

# Needs Met

Youth St

rength

s

Life D

omain Fu

nctioning

Accultu

ration

Child Beh

avioral

/Emotional

Needs

Child Risk

Behav

iors

Caregiv

er Fu

nctioning

0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%90.0%

100.0%

% Needs Met

Youth St

rength

s

Life D

omain Fu

nctioning

Accultu

ration

Child Beh

avioral

/Emotional

Needs

Child Risk

Behav

iors

Caregiv

er Fu

nctioning

0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%90.0%

100.0%

% Youth Met Need(s) in Dimension

Met Needs• Advantages– Effective way to communicate improvement– Simple for audiences familiar with the CANS to

understand– Several options for reporting

• Challenges– Requires additional explanation if audience


Dimension-Level Analyses

• Questions?

• Additional Methods?

• Thoughts?

Item-Level Analyses

Item Score• Analysis Steps

1. Mean item score for all youth at Intake and at Discharge

2. Multiply by 10

Item Score• Reporting Results– Graph of intake and discharge scores


Living Situa-tion


Attendance Behavior Achievement0

2

4

6

8

10

12

14

16

18

Item Scores - Life Domain Functioning

IntakeDischarge

Impulsiv

ity/H

ypera

ctivit

y

Depres

sion

Anxiety

Oppositional

Conduct

Trauma

Anger C

ontrol

Attachmen

t

Judgmen

t

Socia

l Beh

avior

0

2

4

6

8

10

12

14

16

Item Scores - Child Behavioral and Emotional Needs

IntakeDischarge

Suici

de Risk

Self M

utilation

Other Se

lf Harm

Dange

r to O

thers

Sexu

al Agg

ressio

n

Runaway

Fire S

etting

0

1

2

3

4

5

6

Item Scores - Child Risk Behaviors Needs

IntakeDischarge

Family

Superv

ision

Involve

ment

Knowledge

Organiza

tion

Family

Stres

s

Socia

l Reso

urces

Safet

y0

2

4

6

8

10

12

14

16

Item Scores - Caregiver Functioning

IntakeDischarge

Item Score• Advantages– Easy to present graphically

• Disadvantages– Does not communicate results in terms of number

of youth showing improvement

Any Improvement• Analysis Steps

1. Identify youth with ratings of 2 or 3 at Intake2. Identify youth with lower scores at Discharge

• Intake Rating > Discharge Rating

3. Compute mean number of youth showing improvement • Note that need does not have to be met to count

in this analysis

Any Improvement• Reporting Results– % of youth with any improvement in functioning


Living Situa-tion



10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Improved - Life Domain Functioning

Impulsiv

ity/H

ypera

ctivit

y

Depres

sion

Anxiety

Oppositional

Conduct

Trauma

Anger C

ontrol

Attachmen

t

Judgmen

t

Socia

l Beh

avior

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Improved - Child Behavioral and Emotional Needs

Suicide Risk Self Mutilation Other Self Harm Danger to OthersSexual Aggression0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Improved - Child Risk Behaviors

Family

Superv

ision

Involve

ment

Knowledge

Organiza

tion

Family

Stres

s

Socia

l Reso

urces

Safet

y0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Improved - Caregiver Functioning

Any Improvement• Advantages– Simple to analyze– Allows for any improvement in functioning to be

reflected• Challenges– Lack of established benchmarks

Met Needs• Analysis Steps

1. Identify youth with ratings of 2 or 3 on individual items at Intake

2. Determine whether item ratings decreased to a 0 or 1 by Discharge

3. Calculate the percent of youth who met the item

Met Needs• Reporting Results– Percent of youth who met individual needs– Results for individual needs within a dimension


Living Situa-tion



10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Met Needs - Life Domain Functioning

Impulsiv

ity/H

ypera

ctivit

y

Depres

sion

Anxiety

Oppositional

Conduct

Trauma

Anger C

ontrol

Attachmen

t

Judgmen

t

Socia

l Beh

avior

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Met Needs - Child Behavioral and Emotional Needs

Suicide Risk Self Mutilation Other Self Harm Danger to Others Sexual Aggression0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Met Needs - Child Risk Behaviors

Family

Superv

ision

Involve

ment

Knowledge

Organiza

tion

Family

Stres

s

Socia

l Reso

urces

Safet

y0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

% Youth Met Needs - Caregiver Functioning

Met Needs• Advantages– Effective way to communicate improvement– Simple for audiences familiar with the CANS to

understand• Challenges– Requires additional explanation if audience


Item-Level Analyses

• Questions?

• Additional Methods?

• Thoughts?

Establishing Benchmarks

Grab those pens and pencils!

Establishing Benchmarks1. Youth2. Service models3. CANS versions4. Availability of data5. Analysis methods6. Reporting results

Establishing Benchmarks1. Youth

a. Ageb. Race/ethnicityc. Strengths and needs prior to intervention

Establishing Benchmarks2. Service models

a. Wraparoundb. Residential treatmentc. Crisis interventiond. Outpatient therapye. Detention

Establishing Benchmarks3. CANS versions

a. Comprehensiveb. Mental healthc. Juvenile justiced. Child welfaree. Educationf. Crisis

Establishing Benchmarks4. Availability of data

a. Number of youth served annuallyb. Method for completing CANSc. Data managementd. Willingness/ability to share data

Establishing Benchmarks5. Analysis methods

Dimension-Levela. Dimension scoresb. Any improvementc. Actionable Needsd. Met Needs

Establishing Benchmarks5. Analysis methods

Item-Levela. Item scoresb. Any improvementc. Met Needs

Establishing Benchmarks6. Reporting results

a. Youth demographicsb. Service contextc. Amount, frequency and/or duration of servicesd. Sample sizee. Length of stayf. CANS version usedg. Data analysis method used

Next Steps1. Compile your survey responses2. Share survey with other CANS users3. Form CANS Benchmarking Workgroup

a. John Lyonsb. Volunteers?c. Nominations

4. Develop action plan5. Provide updates on progress

Thank You!

Vicki Sprague Effland, Ph.D.Director, Outcomes and Evaluation

Choices, Inc.4701 N. Keystone Ave., #150

Indianapolis, IN [email protected]

mailto:[email protected]

Developing a Resource Guide for CANS Data Analysis and Reporting Vicki Sprague Effland, Ph.D.

Documents

Transcript of Developing a Resource Guide for CANS Data Analysis and Reporting Vicki Sprague Effland, Ph.D.