The Wisconsin RtI Center/Wisconsin PBIS Network (CFDA #84.027) acknowledges the support of the Wisconsin Department of Public Instruction in the development of this presentation and for the continued support of this federally-funded grant program. There are no copyright restrictions on this document; however, please credit the Wisconsin DPI and support of federal funds when copying all or part of this material.

A12: Roundtable/NetworkingEvaluation Tools

Presented by Peg Mazeika & Kao Moua Her WI PBIS & RtI Technical Assistance Coordinators Amy Tranel External Coach MMSD

Today’s Agenda• Overview of PBIS and Academic Surveys and Tools on the PBIS Apps

website and Wisconsinrticenter.org website

• Evaluation Action Plan – Generate discussions, decisions, and selections on best fit tools – Collaborate and develop district and/or school timelines

• Yearly Action Plan• 3-5 Year Plan

• Evaluation Schedule Examples

• Questions/Discussion

PBIS

Benchmarks of Quality (BOQ)

Self-Assessment Survey (SAS)

Academic

School-wide Implementation

Review (SIR)

All Staff Perception Survey

Why Evaluate?Synthesize Upon

• What is the current status of our implementation?

• To what extent was Academic RtI/SWPBS implemented as designed?

• What do we need to attain/maintain fidelity? Move up to next tier? What Training, Coaching, Staff Training? Family/Community support? What funding will be needed?

• Address the evaluation question: “Are we doing what we said we would do?”

Implementation Takes

Time: 2 – 4 Years

Explorati

on

Installa

tion

Initial

Implementation

Full

Implementati

on

/ Continuous

Improve

ment

Stages of Implementation

Levels of Implementation

*Exploration

Not in Place

Sustainability

*Installation

District Leadership TeamSelf-Assessment

• Use by district leadership team (for example, district administrators, principals, external coaches, etc)

• Use to self-assess RtI Academic & Behavior overall for annual and long term goals and needs

• Develop action plan for training, funding, Coaching, professional development, technical assistance for team and staff

“Evaluation is the process of collecting and using information for decision-making. A hallmark of School-wide Positive Behavior Support (SWPBS) is a commitment to formal evaluation.”

District Leadership Team Self AssessmentWisconsinpbisnetwork.org

Appendix A Blueprint For Success

What is Evaluation?

Current Tools for School PBIS Implementation Evaluation

Research Tool

Annual AssessmentTool

Progress Monitoring

Tool

UniversalSystem

System-wide Evaluation Tool

(SET)

Benchmarks of Quality (BoQ)

Early Childhood BoQ (EC BoQ)

PBIS Self Assessment Survey (SAS)

Tiered Fidelity Inventory (TFI)

Team Implementation Checklist (TIC)

Secondary &

TertiarySystems

(ISSET) Benchmarks of Advanced Tiers (BAT)

PBIS Self Assessment Survey (SAS)

Tiered Fidelity Inventory (TFI)

Monitoring Advanced Tiers

Tool (MATT)

Outcome Measures: School Safety Survey

Current Tools for Academic RtI Implementation Evaluation

Research Tool

Annual AssessmentTool

Progress Monitoring

Tool

UniversalLevel (Tier 1)

Selected Level (Tier II)

&Intensive Level

(Tier III)

Outcome Measures:

School-wide Implementation Review &

RtI All-Staff Perception Survey

Intensive

Selected

Universal

Current Tools for Academic RtI Implementation Evaluation

Scho

ol-w

ide

Impl

emen

tatio

n Re

view

&

Rt

I All-

Staff

Per

cepti

on S

urve

y

Current Tools for Academic Implementation Evaluation: School-wide

Implementation Review What Why Who When How

School-wide Implementation

Review

An online self assessment tool for Wisconsin schools to help them measure their RtI implementation as it relates to Wisconsin’s RtI framework.

To answer the following questions: What does RtI look like in action?

What does RtI look like in our school?

What are our RtI strengths?

Where have we improved? What do we still need to work on?

The SIR should be completed by school leadership teams that include a building administrator.

It should be taken annually as part of the school district’s action planning process.

The SIR is available through the Wisconsin RtI Center website at no cost. Username and password for access to the SIR and reports will be emailed to the building administrator.

Current Tools for Academic Implementation Evaluation: RtI All-Staff

Perception Survey What Why Who When How

RtI All-Staff Perception Survey

It is used by school staff for initial and annual assessment of perceptions of implementation of reading and mathematics multi-level support systems in their school.

The survey results are summarized and used for a variety of purposes including: decision making; annual action planning for each content area; and identifying staff awareness of specific RtI practices, etc.

All instructional staff in the school should be asked to complete the survey.

Initially, staff complete the RtI All-Staff Perceptions Survey after the school leadership team has completed the SIR. In subsequent years, and as an on-going assessment and planning tool, it should be completed before the SIR.

It is required that one school staff member be responsible for setting up and administering the survey to the instructional staff. The survey must be administered electronically in order for a report to be generated.

www.pbisapps.orgwisconsinrticenter.org

Decisions….Decisions!• Decisions about Evaluation Tools– Which tools answer your district/schools’ evaluation

questions/needs?

– How will you use data to action plan district/team training and support needs based on evaluation results?

• Decisions about Evaluation Timeline– When/how often do you administer the tools?

• Decisions about Evaluation Reporting– Who & How Often will you report outcomes to school staff

and to the District Leadership?

Evaluation & Action PlanningDecisions about Evaluation Tools

• Does your District/School need a Research tool?• PBIS: (SET/ISSET)

• Academic: School-wide Implementation Review (Reading/Math) & RtI All-Staff Perception Survey (Reading/Math)

• Which Annual Assessment Tool(s) will you use?• PBIS: BoQ, ECBoQ SAS, Safety Survey, BAT,

• Academic: School-wide Implementation Review (Reading/Math) & RtI All-Staff Perception Survey (Reading/Math)

• Which Screening/Progressing Monitoring Tool(s) will/do you use?• PBIS: TIC, MATT

• Academic: School-wide Implementation Review (Reading/Math) & RtI All-Staff Perception Survey (Reading/Math)

• Does your District/School plan include evaluation of all levels of implementation? Tier I (Universal Level), Tier II (Selected Level) -Tier III (Intensive Level)?

Example:The District wants to bring RtI Behavior and Academics together and create

one evaluation plan.

• In what specific PBIS & Academic RtI areas are we strong, and in which areas do we need improvement or training?

• Does our current evaluation plan meet the needs of our school(s) in both Academic and Behavior?

• How can we schedule our evaluations so that we are not overwhelming our schools?

• What Funding/Coaching /Training /Technical needs will we have in the next few years to scale up our implementation in both areas?

Attendance • Attendance/tardies• Chronic absenteeism

• Mobility• Engagement• Participation

BehaviorSocial-Emotional

• Office referrals• Suspensions• Behavioral screening• Internalizing behaviors

• Developmental assets• At-risk support• Behavioral plan• Family stressors

Content• Academic screening• Common assessments• Standardized testing• Grades

• Retention• Accelerated learning• Interventions, Title I• Head Start, Summer School

Use your ABCs: Existing Data

Behavior & Academic Assessment Timeline

Assessment Instrument

Fall (August/Sept/Oct)

Winter (Dec/Jan)

Spring(April/May)

Purpose

Team Implementation Checklist (TIC)

X X X until 80%Progress monitor

Implementation Tier 1

Self-Assessment Survey (SAS)

X Annually

Fidelity of implementation

Staff perception T1,2,3RtI All-Staff Perception Survey

X Annually (Before or After the SIR)

OR X Annually (Before or After the SIR)

Assess needs

School-wide Implementation Review

X (Progress Monitor?) X Annually Progress monitor

BoQ

X Annually

Fidelity of Implementation Tier 1

MATTX X (until 80%)

Progress monitor implementation Tier2/3

BATTX Annually

Fidelity of Implementation Tier2/3

SSBDX

X X Universal Screener

Early ID Internalizer/Externalizer

ABC’s (Existing Data) or Academic Screening Data

X X XScreening

Progress Monitor

Blue Print Self Assessment Dist Leadership Team

XTraining and

implementation action planning and support.

Adapted from Nieves Flores, 2012

• What is our baseline, and what is our progress through the year in our Systems development and implementation?

• Are we improving on our implementation compared to baseline? Moving toward attaining Fidelity? Are student outcome data improving?

• How do our individual staff members rate our implementation in each area?

• Are our self-assessments consistent with data from outside evaluators? (SET, Walk Through)

• Are we meeting our annual Action Plan Goals?

ExampleDistrict/School Evaluation Questions

Newly implementing/not yet at fidelity; Questions to be answered:

• What specific areas in RtI are we strong in, and in which areas do we need training or support to improve?

• Are we maintaining our implementation of RtI Academic/PBIS at Tier 1 as we move up in the Tiers ?

• Do our staff members continue to ‘buy in’ to our Academic/PBIS implementation?

• Have we assessed our implementation of in the areas of Culturally Responsive Practices and Family Engagement?

ExampleDistrict/School Evaluation Questions

Implementing PBIS for three years: We have been using the SAS, BoQ, SIR and have started Tier 2.

Task: Developing your PlanQuestions to Consider:• Which surveys are your schools/districts currently using?• Do they answer your evaluation questions:

– Are we implementing RtI/SWPBIS with fidelity? (SAS, BoQ, BAT, SIR)– Is there more information that we need? (TIC,MATT, SSS, CRP Interview, Family

Engagement Interview)– How does our staff rate our implementation (SAS, RtI All-Staff Perception

Survey)• Do your surveys assess ALL the Tiers you are implementing? (TIC, ECBoQ, BoQ, MATT, BATT, TFI)• Do you need a research tool? (SET, ISSET)• Are you using a Universal Screener for Behavior/Social Skills? • Are you Action Planning from your Assessments?

Task: Developing your PlanOther questions to consider

• Are you “topping out” of the surveys you are currently taking?• Do you need to customize your evaluation plan for your

school, or will one evaluation plan fit for all schools in your district?

• Are you assessing both Academic and Behavior RtI implementation?

• Do you use Universal Screening?• Is your Evaluation Plan manageable for your school(s)?• Do you have a District Leadership Team that plans for PBIS/RtI

Implementation and support? • Does your District Leadership Team Self Assess annually?

Task: Complete this Plan for your District/SchoolMeasure/Tool

Year One Year Two Year Three

Fall Winter Spring Fall Winter Spring Fall Winter Spring

Action Plan from your Evaluations!Action Plan Resources: Templates on PBISassessment.org for TIC, SAS & BoQ etc. action plans

Use Action Plans available in resources area Wisconsinpbisnetwork.org

TIPS format meeting minutes action planState specifically when you will review your goals in your action plan!

Identify components you are Working on Now!

What RtI components are already underway?AREA: ________

Strengths Working on now!

BoQ/SIR Indicators of Success

Work on next!

Universal

Selected

Intensive

Leadership & Organizational Structures

School name_________________________________ Date______________

Implementation Action Plan:

Item/Section Evidence/Data Roadblocks Strategies and resources

Next StepsAction: Who? When? Goal:

Universal Screeners

For info on Selection of Universal ScreenersGo to Wisconsinpbisnetwork.org enter Universal Screeners in Search box

Resources:

• pbisapps.org: website to connect to pbis assessment materials , videos, resources

• Evaluation Blueprint for School-Wide Positive Behavior Interventions & Support: Appendix A District Leadership Assessment Tool (wisconsinpbisnetwork.org)

• Bob Algozzine, Robert H. Horner, George Sugai, et.al

The Role of Fidelity Measures in Implementing Evidence-based Practices: Lessons Learned from School-wide PBIS

• Tary J. Tobin , Claudia G. Vincent, et.al