Evaluation in Michigan’s Model

Steve Goodmansgoodman@oaisd.org

National PBIS Leadership Forum

October, 2011

http://miblsi.cenmi.org

Mission StatementTo develop support systems and sustained implementation of a data-driven, problem solving model in schools to help students become better readers with social skills necessary for success.

Nicole Matthews Data Entry

Nicole Matthews Data Entry

Anna HarmsEvaluation Coordinator

Anna HarmsEvaluation Coordinator

Ed HuthData Analyst

Ed HuthData Analyst

Jennifer RollenhagenMeasurement and

Evaluation Specialist

Jennifer RollenhagenMeasurement and

Evaluation Specialist

Evaluation contributes to Project by:

•Developing and providing resources to enhance local capacity related to measurement and evaluation, consistent with the implementation research. Evaluation supports the competencies and capacity necessary for implementation specialists and local districts to engage in effective data-based decision making as part of an integrated behavior and reading RtI model.

•Reporting on program activities and project outcomes to evaluate and improve effectiveness and efficiencies of the project to ensure value added to consumers and stakeholders. This is accomplished by implementing Plan-Do-Study-Act cycles.

MiBLSi Project Evaluation Team

Several Purposes of MiBLSi Assessments

• Audit

– for “taking stock” of current strengths/weaknesses and action planning

• Formative evaluation

– for improving program while it is in the process of being implemented

• Summative evaluation

– for improvement of future reiterations

Internal EvaluationInternal Evaluation(within the project)(within the project)

Feedback Feedback MiBLSi Value-Added Work System

Capital

Resources

People Materials Information

Work Systems: Providing the RtI practices and the supports for the these practices to take place successfully within schools and districts

Stakeholders/Funders

Stakeholders/FundersInvestments:

•Funding•Visibility•Political support

Returns:•Addressing critical issues (Discipline/Ethnicity)•Addressing program directives (State Performance Plan)

Consumers (schools, districts,

ISDs)

Consumers (schools, districts,

ISDs)

Valued RtI Products/ Services

FinancialEvaluationProfessional Learning

Technical Assistance

Worker

Organizational Level

Worker

Worker

Process Level

Evaluation

Tech. Assistance

Prof. Learning

Finance

Consumers

Stake Holders

Worker

Evaluation At Organizational, Process and Performer (worker) level: At each , measurement takes place at determined interval. This information is compared to established standards and provided as feedback for the system.

Levels of Internal Evaluation

Level Description Tools

Organizational Evaluating impact of the RtI project •Stakeholders

• Addressing critical issues (e.g., Discipline/Ethnicity)

• Addressing program directives (e.g., State Performance Plan)

•Consumers• Valued results (e.g., Improved

quality program, fidelity of implementation, successful student outcomes)

• Annual reports

• Measures of program quality

• Measures of fidelity of implementation

• Measures of student outcomes

• Survey of consumer satisfaction

Process Evaluating operational productivity and efficiency of each unit within organization

Project management tools•Fasttrack, MS Project

•Basecamp

Performer (worker)

Evaluating worker productivity and efficiency

• Job models

Job Model: Measurement and Evaluation Specialist

File Maker Pro Data Base

External EvaluationExternal Evaluation(outside the project)(outside the project)

Building StaffBuilding StaffBuilding StaffBuilding Staff

Building Building Leadership Leadership

TeamTeam

Building Building Leadership Leadership

TeamTeam

LEA District LEA District Leadership Leadership

TeamTeam

LEA District LEA District Leadership Leadership

TeamTeam

Collecting information to evaluate implementation effects and using this information for continuous improvement

•Fidelity of implementation (across schools)•Systems integrity (district-LEA)•Student success (district-wide)

•Fidelity of implementation (state)•Systems integrity (project)•Student success (project-wide)

•Fidelity of implementation (across grades)•Systems integrity (school)•Student success (school-wide)

•Student success/Intervention effectiveness

ISD ISD Leadership Leadership

TeamTeam

ISD ISD Leadership Leadership

TeamTeam

MiBLSi ProjectMiBLSi ProjectMiBLSi ProjectMiBLSi Project

•Fidelity of implementation (across districts)•Systems integrity (district-ISD)•Student success

Assessments

Elementary Schools• Major Discipline Referrals• PBIS Self-Assessment Survey• PBIS Team Implementation Checklist• Benchmarks of Quality (BOQ)• Schoolwide Evaluation Tool (SET)• Benchmarks for Advanced Tiers (BAT)• Dynamic Indicators of Basic Early Literacy

Skills (DIBELS)• Planning and Evaluation Tool (PET) for

Effective Schoolwide Reading Programs• Effective Reading Support Team

Implementation Checklist• Special Education Data Collection Form• Schoolwide Reading Analysis Support Page

Middle/Junior High Schools• Major Discipline Referrals• PBIS Self-Assessment Survey• PBIS Team Implementation Checklist• Benchmarks of Quality (BOQ)• Schoolwide Evaluation Tool (SET)• ORF/MAZE through AIMSWeb• School-Wide Evaluation and Planning

Tool for Middle School Literacy (SWEPT)

• Middle School Reading Team Implementation Checklist

• Special Education Data Collection Form

Building LevelBuilding Level

Assist Teams in Using Data for Decision-making

• First Year– Winter systems review

– Spring Data Review

• Second Year– Fall data review

– Winter data review

– Spring data review

• Third Year– Fall data review

– Winter data review

– Spring data review

Assessment Booklet

• Description of assessments

• Data collection schedule

• Data summary• Data forms and

assessment forms

Team Evaluation of Outcome, Process and Systems Data

Assessment Schedule (for Cohort 7 from MiBLSi website)

Video examples for completing and submitting PBIS assessments

Improving the accuracy and Consistency of Recording Office Discipline Referrals

Developing Fluency with Discipline Referral Categories

Example Exercise 2: Match the example situation below to the correct problem behavior on the discipline categories answer sheet. Write the letter in the column for Exercise 2.

District LevelDistrict Level

Increase 8%

Decrease 14.6%

Focus on Implementing with Fidelityusing Benchmarks of Quality (BoQ)/ODR ’06-’07 and ’07-’08

District Implementation Tracking Form

Leadership-Implementation Support Team Self-Assessment

Lesson Learned• Teams need to be taught how to

analyze and use data

• Emphasis on directing resources to need and removing competing activities

• As we grow, it is even more important to systematic gather data that is accurate and then act on the data for continuous improvement

• More work is needed in developing feedback cycles

“Even if you’re on the right track, you’ll get run over if you just sit there”

- Will Rogers