Evaluating School Psychologists Within a Multi-tiered System of

Supports Delivery Model: A New Era of Accountability

SSPEM

2

FASP Annual Conference

David WheelerGeorge Batsche

Student Support Services Project

3

OverviewStudent Success Act (S.B. 736): Setting the state for professional personnel evaluations in FloridaMulti-Tiered System of Supports: Common Language/Common UnderstandingAlignment Between the MTSS Model and Student Services Delivery in FloridaWhy MTSS? Why Now?School Psychologists Skills/Role in a Multi-Tiered Support SystemFlorida’s New Evaluation SystemOverview of Student Services Personnel Evaluation Model (SSPEM)

4

Florida’s New Evaluation System

The Student Success Act (SB 736)1012.34, F.S.

5

Purpose of Student Success Act

For the purpose of increasing student learning growth by

improving the quality of instructional, administrative, and

supervisory services in public schools of the state.

1012.34(1), F.S.

6

Evaluation System Requirements

Designed to support effective instruction and student learning growth & must be used in developing School Improvement Plans.Provide appropriate instruments, procedures, and criteria for continuous quality improvement of professional skills & results must be used when identifying professional development.

Include a mechanism to examine performance data from multiple sources including parents when appropriate.

Identify teaching fields for which special evaluation procedures and criteria are necessary.

Non-classroom Instructional Personnel (Student Services)

Student learning growth data (50%) assigned over three years ORCombination of student learning growth data (30%) & measureable student outcomes specific to the assigned positionInstructional practice based on FEAPs and specific job expectationsProfessional & job responsibilities

1012.01(3)(a)1.b, F.S.

8

Non-classroom Instructional Personnel (Student Services)

- 1012.01(3)(a)1.b, F.SStudent performance (50%)

Student learning growth as assessed by statewide or district assessments ORCombination of student learning growth data (30%) & other measureable student outcomes specific to the assigned position

Instructional practice (non-classroom instructional personnel)

Florida Educator Accomplished Practices (FEAPs)May include specific job expectations related to student supportProfessional and job responsibilities

9

FEAPs do not adequately reflect the job responsibilities & practices of student services personnel

Impact on student performance is indirect

Student Services personnel typically assigned to multiple schools

Measuring student outcomes related to job

Challenges for Student Services Personnel

Evaluations

10

What Informs Instructional

Practice?NASP (and other student services) Professional Practice Standards

Florida Educator Accomplished Practices

School Psychologist Competency Areas

Multi-Tiered System of Supports Delivery System in Florida

Domains of Practice that incorporate professional standards and Florida Multi-tiered System of Supports

11

Domains of PracticeData-based Decision Making and EvaluationInstruction/Intervention Planning & DesignInstruction/Intervention Delivery & FacilitationLearning EnvironmentProfessional Learning, Responsibility, & Ethics

12

13

14

Florida Educator Accomplished Practices

(FEAPs)Quality of Instruction

Instructional Design and Lesson PlanningThe Learning EnvironmentInstructional Delivery and FacilitationAssessment

Continuous Improvement, Responsibility and Ethics

Continuous Professional ImprovementProfessional Responsibility and Ethical Conduct

15

School Psychologist Competencies: Florida

Selected

Data-Based Decision-Making and Accountability

Knowledge of Curricula and Instruction

Knowledge of Evidence-Based Interventions

Consultation, Collaboration and Problem-Solving

Professional School Psychology and Ethical Decision-Making

16

MTSS: Common Language/Common Understanding

17

MTSSA Multi-Tiered System of Supports (MTSS) is a term used to describe an evidence-based model of schooling that uses data-based problem-solving to integrate academic and behavioral instruction and intervention.

The integrated instruction and intervention is delivered to students in varying intensities (multiple tiers) based on student need.

“Need-driven” decision-making seeks to ensure that district resources reach the appropriate students (schools) at the appropriate levels to accelerate the performance of ALL students to achieve and/or exceed proficiency .

18

Why Organize an Evaluation System Around an MTSS

Model?Research supports that an integrated (academic/behavior/social emotional) service delivery system has greater impact on student performance than separate systems

Services and personnel in schools already are organized by levels of intensity of service delivery

Tier 1—What everybody gets—typically general education teacher ledTier 2—What “some” get—typically more intensive, smaller groupsTier 3—What “few” get—typically most intensive, specialized

19

Why Organize an Evaluation System Around an MTSS

Model?Existing and proposed statutes, regulations and practices support a multi-tiered system

IDEIANCLBLearn ActAchievement Through Prevention (PBIS) Act (SB 541)Florida Educator Accomplished Practices (FEAPs) and School Psychologist CompetenciesNASP Model

Evaluation systems require clear responsibility for levels of service delivery and “stakeholders” who are one focus of the evaluation process

20

Why Organize an Evaluation System Around an MTSS

Model?Instructional support staff of all types typically provide instruction/intervention at all levels (Tiers 1,2 and 3) in a school and/or district

School-based research that identifies evidence-based practices is conducted at levels aligned with the Tiers

School-wide (e.g., PBIS, Crisis Prevention)Classroom level (e.g., Group Procedures, Instructional StrategiesGroup level (e.g., academic instruction, social skills training, group work)Very Small Group/Individual (e.g., therapeutic, intense psychological skills training, academic skills)

21

MTSS: Critical Elements

The Four Corners of the “Frame”

Parts of the “Frame”• 3 Tiers of service delivery into which all

academic and behavioral instruction/intervention “fit.”– Content is not been defined by the model

• A structured Problem-Solving Process used to develop, implement, and monitor instruction/interventions

Parts of the “Frame”

Instruction/interventions are modified, intensified and or dropped based on student performance data

Instruction is integrated and systematically planned across the tiers

24

MTSS & the Problem-Solving Process

ACADEMIC and BEHAVIOR SYSTEMS

Tier 3: Intensive, Individualized Interventions & Supports.

The most intense (increased time, narrowed focus, reduced group size) instruction and intervention based upon individual student need provided in addition to and aligned with Tier 1 & 2 academic

and behavior instruction and supports.

Tier 2: Targeted, Supplemental Interventions & Supports.

More targeted instruction/intervention and supplemental support in addition to and aligned with the core academic and behavior curriculum.

Tier 1: Core, Universal Instruction & Supports.

General academic and behavior instruction and support provided to all students in all settings.

Revised 12/7/09

Problem Solving Process

EvaluateResponse to

Intervention (RtI)

Problem AnalysisWHY are they not

doing it?Identify Variables that Contribute to the Lack of Desired Outcomes

Identify the GoalWhat Do We Want Students to Know and

Be Able to Do?

Implement PlanImplement As Intended

Progress MonitorModify as Necessary

Steps in the Problem-Solving Process1. Problem Identification

Identify replacement behaviorData- current level of performanceData- benchmark level(s)Data- peer performanceData- GAP analysis

2. Problem AnalysisDevelop hypotheses (brainstorming)Develop predictions/assessment

3. Intervention DevelopmentDevelop interventions in those areas for which data are available and hypotheses verifiedProximal/DistalImplementation support

4. Response to Intervention (RtI)Frequently collected dataType of Response- good, questionable, poor

27

Why MTSS? Why Now?

MTSS: Integrating Two Evidence-Based Models to Improve the Academic and Behavior Outcomes for ALL Students

• Challenging Times In Which to Educate America’s Children and Youth– Performance Evaluations Tied to Student

Growth– Economic Crises resulting in reduction of

resources– Alternatives to Public K-12 Education– AYP Projections and Expectations– Recruitment and Retention of Qualified

Professionals– Common Language/Common Understanding

with Educators, Parents and the Community

Strategies for Successfully Addressing these Challenges

Align and allocate effective resources with student needs-Return on Investment Model (ROI)

Anticipate the Future-prevention is cost-effective

Use of Highly Effective Practices-identify them, reward them

Efficient Delivery of those Practices

Data to Evidence Effectiveness of Practices

Strong Professional Development and Support to Sustain Effective Practices aligned with district priorities

Communicating Clearly and Frequently with Stakeholders

Use of professional personnel evaluation models that demonstrate impact of evidence-based practices aligned with district mission

Highly Effective Practices: Research

High quality academic instruction (e.g., content matched to student success level, frequent opportunity to respond, frequent feedback) by itself can reduce problem behavior (Filter & Horner, 2009; Preciado, Horner, Scott, & Baker, 2009, Sanford, 2006)Implementation of school-wide positive behavior support leads to increased academic engaged time and enhanced academic outcomes (Algozzine & Algozzine, 2007; Horner et al., 2009; Lassen, Steele, & Sailor, 2006)“Viewed as outcomes, achievement and behavior are related; viewed as causes of the other, achievement and behavior are unrelated. (Algozzine, et al., 2011)Children who fall behind academically will be more likely to find academic work aversive and also find escape-maintained problem behaviors reinforcing (McIntosh, 2008; McIntosh, Sadler, & Brown, 2010)

30

31

School-wide Behavior & Reading Support

The integration/combination of the two:Are critical for school success

Utilize the three-tiered prevention model

Incorporate a team approach at school level, grade level, and individual level

Share the critical feature of data-based decision making

Produce larger gains in literacy skills than the reading-only model

(Stewart, Benner, Martella, & Marchand-Martella, 2007)

32

School Psychologists Role in a Multi-tiered

Support System

33

Emerging Leadership Themes

Multi-tiered Systems of SupportEvidence-based practicesImplementation science

Rob Horner, Futures of School Psychology Conference 2012

Professional Development:Core Skill Areas for ALL Staff

• Data-Based Decision Making Process

• Coaching/Consultation

• Problem-Solving Process

• Collection, Management and Use of Integrated Data Systems

• Instruction/Intervention Development, Support and Evaluation

• Instruction/Intervention Fidelity

• Staff Training

• Effective Interpersonal Skills

35

Student Services Role in an MTSS System

Academic Performance of students (educator appraisal factor) is influenced significantly by social, emotional and behavior factors—the professional practices of student services personnel

Combining evidence-based instructional strategies with evidence-based strategies to enhance student engagement results in the most dramatic student gains (LESSON STUDY)

Enhancing student engagement (at all levels) is a primary role of students services personnel

36

Student Services Role in an MTSS System

The continued viability and importance of student services personnel is influenced strongly by the impact of their practices on student performance-particularly academic performance

Services provided by student services personnel have a strong, evidence-based relationship with student academic performance

A blueprint for a clear, explicit relationship between the provision of evidence-based student services practices and positive student outcomes is critical in the context of school accountability

Student services personnel must PLAN in such a way as to demonstrate ACCOUTABILITY and COMMUNICATE those outcomes.

37

Multi-tier System of Student Supports (MTSSS):Response to Instruction/Intervention (RtI)

An Overview of Data-based Problem-solving within a Multi-tier System of Student Supports in Florida’s Public Schools

Intensive, Individualized Supports•Intensive interventions based on individual student needs•Students receiving prolonged interventions at this level may be several grade levels behind or above the one in which they are enrolled•Progress monitoring occurs most often to ensure maximum acceleration of student progress•If more than approximately 5% of students are receiving support at this level, engage in Tier 1 and Tier 2 level, systemic problem-solving

Targeted, Supplemental Supports•Interventions are based on data revealing that students need more than core, universal instruction•Interventions and progress monitoring are targeted to specific skills to remediate or enrich, as appropriate•Progress monitoring occurs more frequently than at the core, universal level to ensure that the intervention is working•If more than approximately 15% of students are receiving support at this level, engage in Tier 1 level, systemic problem-solving

Core, Universal Supports•Research-based, high-quality, general education instruction and support•Screening and benchmark assessments for all students•Assessments occur for all students •Data collection continues to inform instruction•If less than approximately 80% of students are successful given core, universal instruction, engage in Tier 1 level problem-solving

38

Critical Role in Addressing Barriers to Learning

Engage in collaborative problem-solving at district, school, and individual levels.Provide culturally competent services (academic, social-emotional, behavioral) to students, schools, and families within a multi-tier model of service delivery.Develop and implement evidence-based interventions at each tier.Conduct assessments that inform instruction (screening, progress monitoring, diagnostic). Assess fidelity and effectiveness of instruction and intervention.

39

Critical Role in Addressing Barriers to Learning

Assist in the design and use of data systems (data collection, display, and interpretation).Provide leadership implementing policies and practices that result in effective and equitable outcomes.Provide services and supports to reengage disconnected students.Engage familiesAdvocate for for evidence-based and culturally competent practices.

40

Florida’s New Evaluation System

The Student Success Act

41

Purpose for Personnel Evaluations

As set forth in the Student Success Act and Race to the Top, teacher evaluations are:

Designed to support effective instruction and student learning growth.

Used when developing district and school level improvement plans.

Used to identify professional development.

42

Measure sound educational principles and research in effective practice in three major areas:

Performance of studentsInstructional Practice (FEAPs)Professional & job responsibilities

Evaluations must differentiate among 4 levels of performance:

Highly EffectiveEffectiveNeeds Improvement or Developing (1st 3 years)Unsatisfactory

Purpose for Personnel Evaluations

(cont.)

43

Major Components of the Evaluation System

Instructional Practice measured by the District’s Instructional Practice Framework

Student performance measured by student learning growth

Instructional Practice

(50%)

Performance of

Students(50%)

Instructional PracticeSection 1012.34, F. S., requires that

instructional practice evaluate the following:For Classroom teachers, excluding substitutes:

Florida Educator Accomplished Practices (FEAPs)For Instructional personnel, not classroom teachers:

FEAPsMay include specific job expectations related to student support

Instructional Framework goal: An expectation that all teachers can increase their expertise from year to year which produces gains in student achievement from year to year with a powerful cumulative effect

44

Performance of StudentsAt least 50% of a performance evaluation must be based upon data and indicators of student learning growth assessed annually and measured by statewide assessments or, for subjects and grade levels not measured by statewide assessments, by district assessments as provided in s. 1008.22(8), F.S.

Section 1012.34(3)(a)1., Florida Statutes45

47

Performance of Students

The performance of students represents 50% of a teacher’s evaluation, with performance based on student learning growth– Growth data for 3 years of students assigned

to the teacher.– If less than 3 years of data are available,

years for which data are available must be used, and percentage of evaluation based on growth may be reduced to not less than 40%.

To meet the above requirement, the development of a fair and transparent measure of student growth is essential.

Florida’s Value-Added Model

A value-added model measures the impact of a teacher on student learning, by accounting for other factors that may impact the learning process.

These models do not:Evaluate teachers based on a single year of student performance or proficiency (status model).Evaluate teachers based on simple comparison of growth from one year to the next (simple growth).

48

49

Advantages of Value-Added Models

Teacher teach classes of students who enter with different levels of proficiency and possibly different student characteristics.

Value-added models “level the playing field” by accounting for differences in the proficiency and characteristics of students assigned to teachers.

Value-added models are designed to mitigate the influence of differences among the entering classes so that schools and teachers do not have advantages or disadvantages simply as a result of the students who attend a school or are assigned to a class.

Value-Added Example

50

0

100

200

300

400

500

Student E

Teacher X

Prior Performance Current Performance Predicted Performance

The difference between the predicted performance and the actual performance represents the value-added by the teacher’s instruction.

The predicted performance represents the level of performance the student is expected to demonstrate after statistically accounting for factors through a value-added model.

51

Florida’s Value-Added Model

Begins by establishing expected growth for each student based on historical data each year.Represents the typical growth seen among students who have earned similar test scores the past two years, and share the other characteristics identified by the committee.Accounts for student, classroom, and school characteristics (factors outside the control of the teacher)

52

Factors Identified to “Level the Playing Field”

Student CharacteristicsUp to two prior years of achievement scores (the strongest predictor of student growth)The number of subject-relevant courses in which the student is enrolledStudents with Disabilities (SWD) statusEnglish Language Learner (ELL) statusGifted statusAttendanceMobility (number of transitions)Difference from modal age in grade (as an indicator of retention)

Classroom CharacteristicsClass sizeHomogeneity of students’ entering test scores in the class

Factors Identified by the SGIC to

“Level the Playing Field” The model recognizes that there is an independent factor related to the school that impacts student learning – a school component. 

Statistically is simply the factors already controlled for in the model measured at the school level by grade and subject.May represent the impact of the school’s leadership, the culture of the school, or the environment of the school on student learning.Acts as another covariate, just like all other factors.

53

54

Overview of SSPEM Developing a State Model for

Student Support Services Personnel Evaluations

55

56

57

Student Support Services

58

Fundamental Principles

Fundamental Purpose: Improve academic & behavioral outcomes for students

Reflect a Multi-tiered System of Support (MTSS) framework.

Align with evidence and research-based practices and professional standards linked to positive student outcomes.

Integrate common practice standards across student services professions.

Support professional growth and continuous improvement.

59

Fundamental Principles (cont.)

Offer a state-approved evaluation framework that is dynamic (flexible

& fluid) and complies with the Student Success Act for districts to

adopt, adapt, or use as a guide.

60

Focus on “practices” component

Crosswalk Professional Practice Standards with FEAPs, Professional Competencies, & Teacher/Principal models

Identify Domains of practice; Practices; Indicators for each practice (levels of performance/proficiency)Research/evidence supporting practice

Develop an evaluation rubric

Vet model rubric with key stakeholders

Developing the Model

61

Relevance of Professional Standards

PurposeEstablishes foundationAuthenticatesLends credibility and agreement

IntegratesResearch linked to positive student outcomesEvidence-based strategies Best practices

62

Bureau of Educator Recruitment & Student Support Services (BEESS)

Task: Develop Process and Model

Core Development Workgroup (District Coordinators from each of the Student Service disciplines)

Task: Develop Draft Model

Focus Group (Student Services Directors; Coordinators from Student Service disciplines; Administrators; Other stakeholders)

Task: Feedback on Draft Model

Contributors/Partners

63

Conceptual Model

Domains (5 Domains) – broad categories used to organize professional practices and help structure the evaluation.

Practices (25 Practices) – standards of practice within a a domain related to a specific area of professional skill.

Indicators – continuum of descriptive statements that assist in differentiating levels of performance for each practice (Highly Effective, Effective, Emerging, Ineffective).

64

Domains of PracticeData-based Decision Making and EvaluationInstruction/Intervention Planning & DesignInstruction/Intervention Delivery & FacilitationLearning EnvironmentProfessional Learning, Responsibility, & Ethics

65

Scoring Rubric for Indicators

Highly Effective – practice has broader, systemic impact (school-wide/district-wide) OR facilitates effective practice of others through mentoring and/or trainingEffective – demonstrates the essential elements of the practice competently and independently with individual students and groupsEmerging – developing practice competence but requires additional supervision, support or training Ineffective – does not demonstrate practice or demonstrates practice poorly

66

Evaluation Rubric

67

Evaluation Rubric

68

Evaluation Rubric

69

Evaluation Rubric

70

Evaluation Rubric

71

Resources/Tools in Guide

Research Support for Model (Appendix B)Crosswalks

Professional Practice Standards (Table 2)FEAPs, Marzano, & Danielson (Table 3)

Methods and Sources of Evidence for Evaluating Professional Practice (Table 1)Scoring Protocols

Evaluation Rubric Scoring Protocol (Form 1)Student Growth ProtocolSummative Evaluation (Form 3)

72

73

74

75

76

Recommendations for District Use

The Evaluation Cycle ProcessSSPEM and the District FrameworkStudent Growth Component

77

SSPEM is designed toEstablish practices and expectations that are linked to student outcomes (academic & behavioral) and based on research.

Develop evaluation procedures that align with professional standards and accomplished educator practices (FEAPs).

Provide feedback to the professional that recognizes effective performance, identifies areas for improvement, and directs professional growth activities.

Provide support to supervisees/practitioners not meeting performance expectations.

78

Evaluation Cycle Process

79

Student Growth Component

Student learning growth component must account for 50% of the evaluation (modified if less than three years of data).

Must be based on students assigned to the professional.

Measurable student outcomes – up to 20% of the student learning growth component may be based on measurable student outcomes specific to the position.

80

81

Problem-Solving Process:Step 1

Desired Outcome: What Do We Want the Student(s) to Know and Be Able to Do? (Tier 1 Goal-Impact on Academic Growth?)

Current Level of Performance (Prior/History)Metrics

Desired Level of Performance (Expected)MetricsPeer Comparisons, Other Data

Rate of Growth

Actual (Value Added?)

82

Table Top Activity

Setting Service ExpectedOutcome

Personnel Involved

Data Collection

Multiple? BehaviorAcademic

Tier 1

Tier 2

Tier 3