davidr:davidr:

C-8 Moving Toward Excellence:C-8 Moving Toward Excellence:Integrating Performance and Integrating Performance and

Outcome MeasurementsOutcome Measurements

Jill Pfitzenmayer, Ph.D.Child & Family Services, Newport, RI David Robinson, Ed.D.Center for Evaluation and Research with Children and Adolescents, Massachusetts Society for the Prevention of Cruelty to Children, Boston, MA

Presentation ObjectivesPresentation Objectives

11.   .   Participants will learn how to identify program goals, objectives and specific program activities toward the accomplishment of program mission, vision and values.

2. 2.   Participants will obtain information about establishing an organizational structure that supports critical analysis of program performance and quality improvement.

3. 3.   Participants will identify specific outcome indicators, measurement tools and data collection methods relevant to their program plans.

Presentation ObjectivesPresentation Objectives

4. 4.   Participants will learn how to link staff training, employee retention, program performance and outcome measurement efforts.

5. 5.   Participants will be able to understand when internal outcome assessment can be enhanced by the introduction of an outside evaluator as consultant or primary investigator.

• Conceptual framework

•Description of the identified needs and problems

• Discussion of the methodology and approaches used to address the systems issues,

• Review of specific agency initiatives and reporting structures developed to enhance organizational quality, and

• Discussion by an outside evaluator familiar with the agency who was brought in for consultation

AgendaAgenda

DefinitionsDefinitions

Performance MeasurementPerformance Measurement – “Ongoing monitoring and reporting of program accomplishments, particularly progress towards preestablished goals.”*

• type or level of program activities (process)

• direct products and services delivered (outputs)

• results of products and services (outcomes)

Program EvaluationProgram Evaluation – “Individual systematic studies conducted periodically or on an ad hoc basis to assess how well a program is working.”*

• internal or external experts and program managers

• learn benefits of program or how to improve it

* GAO, April 1998

Integrating Performance Measurement and Integrating Performance Measurement and Program Evaluation: A Conceptual FrameworkProgram Evaluation: A Conceptual Framework

• LinkedLinked to institutional mission, goals, objectives, strategic plan, process and outcome

evaluation

• SMARTSMART - SSimple, MMinimum paper, AActively used, RRevised, and TTransferable to new programs

• Best PracticesBest Practices – continuously updated incorporating new findings, technology and resources

• ComprehensiveComprehensive - Quantity, Quality, Effort and Effect

Outcome Evaluation* Needs Assessment Formative and Process* *An understanding of Evaluation Mission, Problem/Needs and Goals is also needed

Logic Model, Evaluation Questions and Types of Evaluation

Goals Objectives Inputs Program Services + Activities

Outputs

Outcomes

Mission Problem Needs Immediate

Short-term

Long-term

Program EvaluationProgram Evaluation

Performance Measurement SystemPerformance Measurement System

1.1. Gilbert Performance Engineering Approach

• Develop a MODELMODEL of process, department, program

• Define INDICATORSINDICATORS of valued results and expectations

• Take MEASURESMEASURES to see how well process is working

• Implement METHODSMETHODS of improvement and evaluate results

Performance Measurement SystemPerformance Measurement System

2.2. Friedman Four Quadrant Approach

HOW MUCH SERVICE DID WE DELIVER?

HOW WELL DID WE DELIVER SERVICE?

HOW MUCH EFFECT/CHANGE DID WE

PRODUCE?

WHAT QUALITY OF EFFECT/CHANGE DID WE

PRODUCE?

INPUTINPUT

OUTPUTOUTPUT

QUANTITYQUANTITY QUALITYQUALITY

Effective Organizational Performance Effective Organizational Performance Measurement SystemsMeasurement Systems

• Commitment Commitment – Leadership team committed to measurement

• ClarityClarity – Degree of clear strategies and metrics

• MetricsMetrics – Indicators measure valuable processes and expectations

• AlignmentAlignment – Performance measures aligned with key people processes and structures

• InvolvementInvolvement – Key stakeholders involved in defining, tracking, and ongoing review of quality performance improvement system

• Multi-service organization over 136 years old. CEO been in office for 22 years; powerful Board of Directors

•Primary services:

•Nine child & adolescent residential sites•Two child care centers •Family counseling program•Home-based program•School-based programs •Transitional housing program

Information about CFSInformation about CFS

•Minimal COA requirements:

Chart reviewCustomer satisfactionUtilization review in select areasIndividual outcome dataIndication of systematic review of agency

Implementation of Continuous Quality Implementation of Continuous Quality Improvement (CQI) ProgramImprovement (CQI) Program

Problem 1:Problem 1: disconnect between accrediting body and agency leadership perceptions of need

Problem 2:Problem 2: global discomfort with/ suspicion of “science” and “data”

Problem 3:Problem 3: Continuous Quality Improvement is supposed to involve all levels of staff, not be a top-down process

Implementation Model: ChallengesImplementation Model: Challenges

1.1. Assemble committee representing key constituents of the agency:

Agency leadershipInformation systemsHuman ResourcesBusiness OfficeProgram staffParentConsumer

Steps Required to Develop an Outcome System Steps Required to Develop an Outcome System Year 1: Year 1:

2. Develop a conceptual model to help staff review program activities and outcomes (e.g., logic models)

3. Map out organizational structure and evaluation activities (e.g., department teams)

4. Identify strategy that makes sense to staff to collect, analyze and use information

Steps Required to Develop an Outcome System Steps Required to Develop an Outcome System Year 1: Year 1:

5. Develop protocol and process for all levels of staff participation

6. Develop tools for reporting program activities at least quarterly and identify survey instruments

7. Train staff on recommended model and process

Steps Required to Develop an Outcome System Steps Required to Develop an Outcome System

Year 1:Year 1:

Identify a Solution-Focused ProcessIdentify a Solution-Focused Process

• PLAN

• Decide what is the problem

• Brainstorm solutions

• DO

• Enact solutions

• Collect data on how solutions are working

• ACT• Revise action plan if needed• Discontinue plan• Continue to monitor

• CHECK

• Analyze data

• Discuss findings

CQI Activities Related to COA CQI Activities Related to COA StandardsStandards

C&FS develops long-term strategic plan (4 yrs)

(G2.3)Identified stakeholders

participate in all appropriate

activities (G2.2)

Each dept. develops annual plan of goals and objectives (G2.4)

Internal & externalaudits &

reviews (G2.10, G2.5)

Chart Reviews

(G2.6, G2.1)

Customer satisfaction

surveys(G2.8, G2.10)

OutcomesMeasures

(G2.7, G2.10)

Quarterly review of each dept.achievements

by agency CQI Team(G2.9)

Annual reportsof year-end progress(G2.9)

Reassessment,as necessary (G2.11)

Annual plan Quarterly Monitoring summary Summaries of client/referral satisfaction

forms Summaries of incidents that occurred

over the quarter Summary of external review activities Review department-specific and

aggregated findings

Flow of CQI ActivitiesFlow of CQI Activities

Year 2: Challenges to CQI SystemYear 2: Challenges to CQI System

• Many staff did not understand logic models

• Program Directors didn’t understand basic concepts (e.g., difference between benchmark and baseline data)

• Challenges existed around grasping conceptual material and managing task of putting information on computer!

Year 2: Challenges to CQI SystemYear 2: Challenges to CQI System

• Some programs never tracked outcome data at all

• State-funded programs required their own outcome tools that were either cumbersome or lacked validity

• Staff lacked buy-in, experienced turnover or couldn’t make use of findings

• Several programs had a hard time identifying individual outcome indicators

Year 2: Challenges to an Outcome SystemYear 2: Challenges to an Outcome System

• Mechanically tracking data and outcomes became a challenge—no standardized data base in the agency

• Changes implemented as a result of outcome data were often not documented

• Staff were “busy” but couldn’t document real client change (i.e., conflating outputs and outcomes)

Year 2: Small Successes for an Outcome SystemYear 2: Small Successes for an Outcome System

• Some programs used the process to study and implement real change

Residential Network: system for getting paperwork accomplished

Family Counseling: saw change in patient population via use of SCL-90-R

School-based: outcomes supported pilot program around prevention education

Creating Organizational ChangeCreating Organizational Change

• Year 2:Year 2: Agency/Board leadership in strategic planning meetings, which required a SWOT (Strengths, Weaknesses, Opportunities, Threats) analysis of programs

• Lack of careful investigation of program outcomes forced leadership to rely more on fiscal and output information than on outcome data

• Frustration Innovation

Creating Organizational ChangeCreating Organizational Change

• Continuous Quality Improvement

• Reactive• Minimal Board

involvement• Driven by COA

requirements• Voluntary

management participation

• Organizational Quality & Performance

• Proactive

• Board subcommittee

• Driven by strategic planning process

• Sr VP assigned to job

Coordination of OQP ActivitiesCoordination of OQP Activities

Agency Strategic Plan

Organizational Quality & PerformanceAssess performance and outcomes

to align agency goals and program activities

Learning InstituteLink staff needs with training and

learning activities

Employer of Choice InitiativesMaintain focus on performance

to attract and retain highest quality staff

Moving Toward Excellence: Moving Toward Excellence: Changes in Culture and SystemsChanges in Culture and Systems

• Staff are now required to identify steps in developing programs reflecting excellence and best practices

• Management are held accountable for process and program outcomes

• Systematic data collection and analysis requires new MIS system

• Agency-wide committees and initiatives brought into oversight process

Moving Toward Excellence: Moving Toward Excellence: Changes in Culture and SystemsChanges in Culture and Systems

• New OQP committee members include Corporate Communications and Development in order to ensure inter-departmental coordination

• Methodology for tracking process and outcomes in place

• OQP SVP meets provides technical assistance in program plan development

Moving Toward Excellence: Moving Toward Excellence: Goals of OQP ProcessGoals of OQP Process

• Department-specific process and outcome data will be collected, analyzed and acted upon

• Inter-departmental coordination will improve

– Corporate Com, MIS & Development included in program planning

– Programs work together on mutual or complimentary projects

– Inter-program conflicts will be quickly identified and resolved

Moving Toward Excellence: Moving Toward Excellence: Goals of OQP ProcessGoals of OQP Process

• Communication across departments and up, down chain of command will improve

• Agency leadership will have improved and speedier methods of determining where there are problems and strengths in programs

• Agency and program leaders have tools to continuously plan for the future

CFS Program Performance PlanCFS Program Performance Plan

Initial

Program Goals

(1)

Accomplishments

(2)

Current Status of Activities

(3)

Additional Activity Required

(4)

In order to accomplish our Program and

Performance Goals, we need to have the following

in place

This is how we define the components of Initial

Program Goals

These are the program activities we are currently

engaged in or have completed to reflect our

intended Accomplishments

These are the program activities that must be

completed to reflect our intended

Accomplishments with expected date (s) of

completion

•CFS has developed a framework for enabling staff to see patterns and interrelationships and we are able to learn from our own experiences by reviewing successes and failures systematically.

•Staff are taught and encouraged to employ the ability to deconstruct complex tasks while maintaining awareness of the connection to a larger whole.

•Staff meetings are held regularly to communicate to and training staff on current policies and practices.

•Staff are encouraged by supervisors to voice opinions about policies and procedures.

•Interdepartmental staff meetings should be held with regularity to ensure smooth operations and mutual problem-solving.

•Use of critical thinking skills should be modeled by supervisors.

CFS Program Outcome PlanCFS Program Outcome Plan

Outcomes

(5)

Outcomes Quantified

(6)

Date Sources

(7)

Measures

(8)

Baseline/

Benchmark Data

(9)

These are the observable,

measurable and quantifiable

changes we can expect if we meet

our Program Goals

This is how we will quantify our

expected program outcomes

Where we will obtain

information to measure Program

Outcomes

Instruments or measurement

tools we will use to assess Outcomes

Information to help assess where we are starting (baseline) or comparative markers

(benchmark)

•Staff will be skilled at acquiring knowledge

•75% of staff will report that they participated in at least one activity in which they acquired knowledge.

•Staff self-reports •Learning self-report

•None to report.

CFS Quarterly Program Performance ReportCFS Quarterly Program Performance Report

Initial

Program Goals

(10)

Additional Activity Required

(11)

Progress Made on Additional Activities

(12)

Comment/Analysis on Progress

(13)

As stated in the Program Performance plan from

column (1)

As stated in the Program Performance plan from

column (4)

Status of progress toward Additional Activities

Required as stated in the Performance Plan

Note any difficulties or challenges that are interfering with the

progress being made toward completion of

each task

•CFS has developed a framework for enabling staff to see patterns and interrelationships and we are able to learn from our own experiences by reviewing successes and failures systematically.

•Interdepartmental staff meetings should be held with regularity to ensure smooth operations and mutual problem-solving.

•Use of critical thinking skills should be modeled by supervisors.

•Five interdepartmental staff meetings have been held since Sept 03.

•Supervisors received critical thinking training in Aug 03 and have started modeling skills.

•Supervisors find challenges integrating multidisciplinary staff issues in meetings. Additional training required and will be completed by 1/1/04

CFS Quarterly Program Outcome ReportCFS Quarterly Program Outcome Report

Expected Outcomes

(14)

Baseline or Benchmark

Data

(15)

Results:

•Sample Size

•Findings

(16)

Analysis/

Conclusions

(17)

Future Actions:

•By Whom

•Activity

•Time Frame

(18)

As stated in the Program Outcome

plan from

Column (5)

As stated in Program Outcome plan from column

(9)

Sample size and data results from

measures and data sources

Discussion of findings;

comparison of this quarter’s findings to

baseline/

benchmarks

State what further action will be taken,

by whom, and expected date of

completion

•75% of staff will report that they participated in at least one activity in which they acquired knowledge.

None to report •100 staff surveyed•66% reported participation

•Some progress made toward staff participation in knowledge acquisition; staff report challenges in finding time to learn outside of job.

•By Director: provide two hours/month for staff development activities. Implement no later than 12/03.

Moving Toward Excellence: Moving Toward Excellence: How Do Program Staff Use OQP System?How Do Program Staff Use OQP System?

• Program Directors use process to bring their vision into program planning

• Staff are oriented to OQP performance and outcome plans upon hire as a learning tool

• Program leaders use plans to maintain an eye on program improvement activities

Moving Toward Excellence: Moving Toward Excellence: Lessons LearnedLessons Learned

• Appreciate what you bring to the process from your own discipline and theoretical orientation and how they inform your outcome assessment work

• Ask for help. Use volunteers, experts, etc.

• Get organized and don’t panic

• Don’t underestimate the value of communication

• Maintain a stance of being inquisitive, not judgmental

• Remind yourself and your staff that the process is developmental

• Aim for direction, not perfection

“I wanted to share with you my excitement of having completed the baseline ECERS and ITERS for child care today!…With this data we can now create our baseline and use it to establish our movement forward to improved overall quality in our Centers for the children, families and our staff! This is so exciting. Good things are in our future, along with much hard work…I am confident that I will lead the way to our success!”

Patty Shelley, Child Care Program Director

Genuine Unsolicited Testimony from Staff:Genuine Unsolicited Testimony from Staff:

• Commitment Commitment – Leadership team committed to measurement

• ClarityClarity – Degree of clear strategies and metrics

• MetricsMetrics – Indicators measure valuable processes and expectations

• AlignmentAlignment – Performance measures aligned with key people processes and structures

• InvolvementInvolvement – Key stakeholders involved in defining, tracking, and ongoing review of quality performance improvement system

ASSESSMENT: ASSESSMENT:

Case Study of CQI & Performance Measurement System Case Study of CQI & Performance Measurement System

SCORE = SCORE =

Rating Scale: 1 = A Little; 2 = Just Enough; 3 = SuccessRating Scale: 1 = A Little; 2 = Just Enough; 3 = Success

Primary Presenter: Jill Pfitzenmayer, Ph.D.Title: Senior Vice President, Organizational Quality and PerformanceAgency: Child & Family ServicesAddress: 24 School Street, Newport, RI 02840Phone: 401-848-4186Fax: 401-841-8841Email: jpfitzenmayer@cfsnewport.org Co-Presenter: David Robinson, Ed.D.Title: Director for the Center for Evaluation and Research with Children and Adolescents (CERCA)Agency: Massachusetts Society for the Prevention of Cruelty to ChildrenAddress: 399 Boylston Street, Boston, MA 02116Phone: 617-587-1594Fax: 617-587-1582Email: drobinson@mspcc.org or CERCA@mspcc.orgWeb: www.mspcc.org

CONTACT INFORMATIONCONTACT INFORMATION