Operational Excellence: Outcomes Evaluation System Impact and Outcomes Breakout Session June 14 th,...
-
Upload
deborah-parker -
Category
Documents
-
view
216 -
download
0
Transcript of Operational Excellence: Outcomes Evaluation System Impact and Outcomes Breakout Session June 14 th,...
Operational Excellence: Outcomes Evaluation System
Impact and Outcomes Breakout SessionJune 14th, 2011
Sandra LaFleur, Senior Director of Youth Outcomes
2011 National Conference • Dallas, Texas • June 14 - 16
Purpose & Desired Outcomes
• Review critical components of BBBS’ Outcome Evaluation System (OES)
• Highlight latest data analyses / results
• Reinforce focus on and clarify understanding of the OES
• Others?
2011 National Conference • Dallas, Texas
Today’s framing…• Understand it!
– The rationale for it– The connection to both the past and the present
• Do it!– Are you using RR, SoR, YOS?– How are you tracking it?
• Use it!– Case for support– Indications for improvement
Big Brothers Big Sister’s Outcomes Evaluation System (OES)
Click to add text
2011 National Conference • Dallas, Texas
BBBS’ collective, evidenced-based approach to mentoring–a professional system that changes the lives of our children for the better, forever.
• Builds upon our two P/PV study findings that indicate specific areas where our work yields impact
• Builds upon BBBS’ ability to institute common service delivery methodologies and measures through AIM
• Gives us real-time, agency-specific information on our progress at achieving critical youth outcomes in addition to our nationwide reach
2011 National Conference • Dallas, Texas
Connecting Research to YOS Survey
Leveraging long-standing research to account for real-time progress in our youth across the Big Brothers Big Sisters network
2011 National Conference • Dallas, Texas
We didn’t get here overnight… BBBS has developed Unmatched Capabilities to Link Research & Process to Drive Outcomes on a National Scale
2011 National Conference • Dallas, Texas
Components of BBBS’ Outcomes Evaluation System (OES)
Eight (8)Measurable
Outcome Areas
Focused on:
Community Impact
Future:
Additional 3rd party research
Future: Targeted hard data collection
Youth Outcomes:YOS survey
Match QualityStrength (SOR survey)
Outcomes/AccountabilitiesEducational success
Avoidance of risky behaviorAspirations, confidence, relationships
Match QualityLength (RR=retention rate)
Community Impact
2011 National Conference • Dallas, Texas
Research…to Outcomes…to Accountabilities
2011 National Conference • Dallas, Texas
Reflect…
• Does your board know this information?• Does your agency leadership understand
this information?• Does your program staff recognize their
connection to this?
2011 National Conference • Dallas, Texas
2011 National Conference • Dallas, Texas
2011 National Conference • Dallas, Texas
MATCH SUPPORT Match Support Complete 2011 92.0% 96 % 95.5% 89.7% 6.3%
Match Support On Time 2011 71.0% 84.7 % 84.9% 66.5% 18.2%
% Completion SOR/SITE 3 MO 2011 70.0% 75% 100% 52.0% %
% Completion SOR/COMM 3 MO 2011 70.0% 64% 59.0% 46.0% %
% Completion of YOS/Baseline 2011 100% 97.9% 91.7% 92.2 % 6.2%
% Completion of YOS/Annual 2011 70% 52.9% 65.0% 0 % %
Average Match Length SITE 2011 13.0mo 9.3 9.3 mo 11.5 mo -19.1 mo
Average Match Length COMM 2011 21.5mo 16.7 mo 18.7 mo 17.3 mo -3.2 mo
12 Month Retention COMM 2011 61.1% 56.6 % 49.2% 15.1% 41.5%
12 Month Retention SITE 2011 35.0% 37.3 % 36.1% 11.8% 25.5%
TEAM
Total Youth Served 2011 16391084/680CB
248SB1028/727CB
301SB1,225/704 CB
310SB -10.5%
Closures 2011 40% 12.3% 8.6% 12.4% -1.1%
2011 National Conference • Dallas, Texas
SoR3 month vol compliance
YOSfollow-up
compliance
REGIONS 2010 Q1 2011 2010 Q1 2011
NE 45% 64% 45% 50%
SE 61% 75% 59% 66%
MW 55% 61% 63% 52%
SW 63% 78% 58% 60%
W 57% 70% 57% 53%
2011 National Conference • Dallas, Texas
Reflect…• Are you currently tracking your agency’s
performance on 12 month RR, SoR compliance, and YOS compliance?
• Do you share these numbers with the whole staff?
• Are these goals broken down into individual goals for appropriate staff?
• Are they held accountable for performance on those individual goals critical to OES?
2011 National Conference • Dallas, Texas
Developing Your Case for Support
2011 National Conference • Dallas, Texas
Outcomes Data – More Terms
Correlates to the three accountability areas identified in the BBBSA 2011-2014 Strategic Direction. Your local needs may differ or may require a focus in one particular area.
Strategic Outcomes Areas
Also called Outcome categories (8) and sub-categories on the YOS Score and Outcome reports. Outcomes Measures
Refers to the number of youth whose scores indicate improvement (NOT the percentage change from baseline score to follow-up score)
Percentage of Youth Improving
Percentage of Youth Staying the Same
Statistically Significant
Refers to the number of youth whose scores indicate no change. For example, staying the same in a measure that tends to decline as a child enters teen years could be meaningful.
A statistical significance test determines whether a change in scores is due to chance. For example, if the p value =.05, there is only a 5% likelihood that the change is due to chance
2011 National Conference • Dallas, Texas
Preliminary YOS Data: Early evidence of positive youth outcomes
YOS Scales CB SBSocial acceptance * *
Scholastic competency * *
Grades *
Educational expectations * *
Attitudes towards risk * (*)
Parental trust * *
Truancy
Special adult * *
2011 National Conference • Dallas, Texas
Preliminary YOS Data: Early evidence of positive youth outcomes SUB SCALES
YOS Scales CB SBFinish High School (Educ Exp) * *
Go to college *
Finish college *
Drugs *
Hitting *
Breaking rules in school *
Being late to school *
2011 National Conference • Dallas, Texas
Preliminary YOS Data: SPECIAL POPULATIONS
YOS Scales CBM SBM CBM Children w/ incarcerated
parent
SBM Hispanic Girls
SBM Hispanic
Boys
Social acceptance * * * * *
Scholastic competency
* * * * *
Grades * *
Educational expectations
* * * *
Attitudes towards risk
* (*) *
Parental trust * * *
Truancy
Special adult * * * * *
2011 National Conference • Dallas, Texas
Preliminary YOS Data: (Future Analysis) SPECIAL POPULATIONS
YOS ScalesFoster Care
Youth
Juvenile Justice
referrals
CBM African
American
SBM African
American
CBM Native
American
SBM Native
American
Social acceptance
Scholastic competency
Grades
Educational expectations
Attitudes towards risk
Parental trust
Truancy
Special adult
2011 National Conference • Dallas, Texas
Predicting match closures with SoR…• when controlling for background variables, both 3 month volunteer and child SoR scores are predictive of match closures• Vol SoR scale ranking:
•Confidence•Lack of Frustration•Closeness•Connectedness
•Child SoR scale ranking: •Closeness•Lack of Disappointment•Coping
2011 National Conference • Dallas, Texas
Revisiting SoR Cut Points
Volunteer 3 month SoR Score > 3.95
43% more likely to “survive” than matches below this level
Child 3 month SoR Score > 4.65
39% more likely to “survive” than matches below this level
2011 National Conference • Dallas, Texas
Reflect…• How successfully is your agency leveraging your
OES data in building your case for support?• Are your OES performance results used to
discuss improvements or enhancements to program operations?
2011 National Conference • Dallas, Texas
Next steps…• Assess your obstacles and your facilitators• Talk with your agency team – build your
action plan