Implementing Evidence-based Practices at Scales of Social Significance Rob Horner University of...

Implementing Implementing Evidence-based Practices at Evidence-based Practices at Scales of Social SignificanceScales of Social Significance

Rob HornerUniversity of Oregon

www.pbis.org

GoalsGoalsConsiderations for the emergence of

“evidence-based practices”

Six features for taking EBP technology to scale

Define “implementation” as a unique technology.

Use School-wide Positive Behavior Support as one example

Considerations in the Considerations in the Emergence of Evidence-Emergence of Evidence-based Practicesbased Practices

Defining “practices”

Defining “evidence-based”

Why is it useful to define “Evidence-based Practices?”

Considerations in the Emergence Considerations in the Emergence of Evidence-based Practicesof Evidence-based Practices

Defining a “practice”

◦ A “practice” is a procedure, or set of procedures, designed for use in a specific context, by individuals with certain skills/features, to produce specific outcomes for specific individuals.

◦ A practice can be at any of an array of sizes: Individual technique Intervention package Intervention program

Are these “practices?”Are these “practices?”For whom, in what context, to what For whom, in what context, to what outcome?outcome?

Phonics-based early reading instructionWhole Language Reading InstructionPositive reinforcementInclusionNCLBDiscrete trial trainingGeneralizationPositive behavior supportFunctional analysisApplied behavior analysis

Why should we be invested in Why should we be invested in the emergence of Evidence-the emergence of Evidence-based Practices?based Practices?

Researchers should continually debate and discuss the definition and details.

Implementers need to be able to identify practices that are validated.

Take great care in distinguishing between◦ Documentation that a practice is evidence-

based, and ◦ Documentation that one practice is better

than some other practice.

Taking Evidence-based Practices Taking Evidence-based Practices to Scales of Social Significanceto Scales of Social Significance

The value of a science of human behavior◦ Basic principles that help us describe,

interpret and establish effective patterns of behavior

While behavior analysis is among the most powerful approaches for achieving social change, too often ABA is viewed as relevant only within a narrow range of applications

(we are a niche or boutique technology)

Key Features to AchieveKey Features to AchieveLarge-scale Implementation of Evidence-based Large-scale Implementation of Evidence-based PracticesPractices

1. Focus on comprehensive outcomes defined by the values of the social system

2. Expand the unit of analysis

3. Measure process as well as outcome

4. Use ABA principles to build effective and accessible practices

5. Establish a technology for implementation

6. Define practices for scaling up practices that are evidence-based.

What isWhat is School-wide Positive Behavior School-wide Positive Behavior Support?Support?

School-wide PBS is: A systems approach for establishing the social

culture and individualized behavioral supports needed for schools to achieve both social and academic success while preventing problem behavior

Evidence-based features of SW-PBS Prevention Define and teach positive social expectations Acknowledge (reward) positive behavior Arrange consistent consequences for problem behavior On-going collection and use of data for decision-making Continuum of intensive, individual interventions. Administrative leadership – Team-based

implementation (Systems that support effective practices)

School-wide Positive Behavior Support:School-wide Positive Behavior Support:Current ImplementationCurrent Implementation

School-wide Positive Behavior Support7500 schools in 44 states

Team Coach Curriculum emphasizing prevention: Define

and teach appropriate social behavior to all students

Formal system for rewarding appropriate behavior

Intensive, individual interventions based on behavioral function

On-going data collection and use of data for active decision-making

7500 Schools across 44 states implementing school-wide positive behavior support

Primary Prevention:School-/Classroom-Wide Systems for

All Students,Staff, & Settings

Secondary Prevention:Specialized Group

Systems for Students with At-Risk Behavior

Tertiary Prevention:Specialized

IndividualizedSystems for Students

with High-Risk Behavior

~80% of Students

~15%

~5%

SCHOOL-WIDE POSITIVE BEHAVIOR

SUPPORT

SWIS summary 07-08 SWIS summary 07-08 (Majors Only)(Majors Only)2,532 schools; 1,300,140 students; 1,139,119 2,532 schools; 1,300,140 students; 1,139,119 ODRsODRs

Grade Range Number of Schools

Mean Enrollment per school

Mean ODRs per 100 per school day

K-6 1762 444 .35 (sd=.46)

(1 /300 / day)

6-9 482 653 .92 (sd=1.42)

(1/ 110 / day)

9-12 176 914 1.06 (sd=.1.57)

(1/105 / day)

K-(8-12) 312 401 1.00 (sd=.1.86)

(1/ 155 / day

N = 1679 443 163 246 Elementary Middle High K (8-12)

10%

N = 1679 443 163 246

Elementary Middle High K (8-12)

School-wide PBSSchool-wide PBS1. Focus on comprehensive valued

outcomes Social competence Academic achievement Safety

2. Expand the unit of analysis◦ Whole school

Classroom Groups of “at-risk” students

Individual students needing intensive support

School-wide PBSSchool-wide PBS3. Measurement/ Evaluation

◦ Include both process and outcome measures Outcomes: Office Discipline

Referrals/Academics Process (implementation): Team Checklist Research: System-wide Evaluation Tool (SET)

SWIS

TIC data

SET data

Comprehensive Measures

Progress Monitoring Measures

FidelityMeasures

OutcomesMeasures

XX (ABA)

Comprehensive Measures

Progress Monitoring Measures

FidelityMeasures

XX XX

OutcomesMeasures

XX XX (ABA)

Iowa Checklist 01-05, PK-6 % Fully & Partially Implemented

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

05-A

ug-0

3

05-N

ov-0

3

23-F

eb-0

4

22-J

an-0

4

01-F

eb-0

5

02-J

un-0

5

12-A

ug-0

4

24-N

ov-0

4

01-M

ar-0

5

12-S

ep-0

2

31-O

ct-0

2

28-F

eb-0

3

21-A

pr-0

3

01-S

ep-0

3

05-N

ov-0

3

05-A

ug-0

3

11-S

ep-0

3

07-N

ov-0

3

06-F

eb-0

4

01-S

ep-0

3

01-N

ov-0

3

01-M

ar-0

4

03-A

ug-0

4

08-N

ov-0

4

08-M

ar-0

5

03-J

un-0

5

1 1 1 2 2 2 3 3 3 4 4 4 4 4 4 5 5 5 5 6 6 6 7 7 7 7

Start Up Full Implementation Start Up Part Implementation

School-wide PBSSchool-wide PBS

4. Use ABA principles to Establish Accessible Evidence-based Practices◦ Use the language of the implementation context

◦ Combine technologies needed to achieve valued outcomes. ABA + Person-centered planning + Organizational

Systems + Bio-Medical

◦ Collaborate with other disciplines Mental Health, Juvenile Justice, School Psychology,

Sociology

◦ Use Single-case Designs to Document Evidence-based Practices

Academic, Behavioral, and Academic, Behavioral, and Functional Predictors of Functional Predictors of Chronic Problem Behavior in Chronic Problem Behavior in Elementary GradesElementary Grades

Kent McIntosh University of Oregon

40

5th Grade ORF Trajectories by Function (n = 47)Peer Attention vs. Escape Task

80

90

100

110

120

130

140

150

Fall 03-04 Winter 03-04 Spring 03-04

Mean

Co

rrect

Wo

rds p

er

Min

ute

or 1 ODRs 0

Non-target

Students

5th Grade ORF Trajectories by Function (n = 47)Peer Attention vs Escape Task

80

90

100

110

120

130

140

150

160


Mean

Co

rrect

Wo

rds p

er

Min

ute

.Peer Attn

or 1 ODRs 0

ORF Trajectories by Function (n = 47)

80

90

100

110

120

130

140

150

160


Mean

Co

rrect

Wo

rds p

er

Min

ute

.Peer Attn

Esc. Task

or 1 ODRs 0






Sociology


Embedding Bully-Proofing Embedding Bully-Proofing in School-wide PBSin School-wide PBS

Scott RossRob Horner

University of Oregonwww.pbis.org

A Functional Approach to Bully A Functional Approach to Bully Prevention:Prevention:Remove the social reinforcersRemove the social reinforcers

Define, teach and reward school-wide behavior expectations.

Teach all children to identify and label inappropriate behavior.

Not respectful, not responsible, not safe

Teach all students a “stop signal” to give when they experience problem behavior.

What to do if you experience problem behavior (victim, recipient)

What to do if you see someone else in a problem situation (bystander)

Teach all students what to do if someone delivers the “stop signal”

29

0

2

4

6

8

10

0

2

4

6

8

10

Baseline Acquisition Full BP-PBS Implementation

0

2

4

6

8

10

0

2

4

6

8

10

0

2

4

6

8

10

Num

ber of Incidents of Bullying B

ehavior

School Days

0

2

4

6

8

10

School 1

Rob

Bruce

Cindy

Scott

Anne

Ken

School 2

School 3

3.14 1.88 .88 72%

BP-PBS, Scott Ross 30

Conditional Probabilities of Victim Responses to Problem Behavior

0%

10%

20%

30%

40%

50%

"Sto

p"

"Wa

lk"

Po

sitiv

e R

esp

on

se(l

au

gh

ing

/ch

ee

rin

g)

Ne

ga

tive

Re

spo

nse

(cry

ing

/fig

htin

gb

ack

)

No

Re

spo

nse

Pro

bab

ilit

y o

f R

esp

on

seBaseline

BP-PBS

28% increase 19% decrease

BP-PBS, Scott Ross 31

Conditional Probabilities of Bystander Responses to Problem Behavior

0%

10%

20%

30%

40%

50%

"Sto

p"

"Wa

lk"

Po

sitiv

e R

esp

on

se(l

au

gh

ing

/ch

ee

rin

g)

Ne

ga

tive

Re

spo

nse

(cry

ing

/fig

htin

gb

ack

)

No

Re

spo

nse

Pro

bab

ilit

y o

f R

esp

on

seBaseline

BP-PBS

21% increase

22% decrease






Sociology


Use single-case research to document evidence-based practices.

Define protocol for measuring “effect size”

Define professional standards for defining a practice as “evidence-based” using single case research.

School-wide PBSSchool-wide PBS5. Build a functional technology of Implementation

◦Define conditions for implementation◦Define conditions for high fidelity

Coaching, Policies, Administrative Contingencies

◦Establish implementation with low cost◦Establish procedures for sustainability

and continuous regeneration right from the beginning.

The Role of Behavior The Role of Behavior Specialists in the Development Specialists in the Development of Function-based Behavior of Function-based Behavior Support PlansSupport Plans

Leah BenazziUniversity of Oregon

Three knowledge areas needed Three knowledge areas needed for a behavior support planfor a behavior support plan

Behavior Support Plan

Knowledge about The Student

Knowledge aboutThe Setting

Knowledge about Behavioral Theory

DesignDesignThree conditions in which

behavior support plan recommendations were built from simulated cases (descriptive information, functional assessment information)

5 counterbalanced simulations

◦Team alone 12 plans◦Specialist alone 12 plans◦Team with Specialist 12 plans

MeasuresMeasures

Technical Adequacy: Are elements of behavior support plan consistent with functional assessment hypothesis? (1-6) (range 3-18)◦ ABA Experts (published studies employing

functional analysis)

Contextual Fit: Are elements of plan consistent with values, skills, resources, administrative support?◦ 16 questions (8 factors): ◦ Scored on 1-6 scale: Total (16-96)

Team member Ranking of Plans based on preference for implementation (1,2,3)

Results: Results: Technical AdequacyTechnical AdequacyMean Expert Rating (6-18)Mean Expert Rating (6-18)

Mean Expert Ratings of Technical Soundness

0

2

4

6

8

10

12

14

16

Team Alone Team + Spec Spec Alone

Me

an

Te

ch

nic

al S

ou

nd

ne

ss

* Team alone plans were statistically different from plans that included behavior specialist.*Team + Specialist and Specialist Alone were not statistically significantly different.

Table 3Repeated Measures Analysis of Variance Summary Table for the Effects of Plan Developer on Technical Adequacy Scores

Source df SS MS F

Plan DeveloperBeh Spec Invol Unpredicted

211

285.88279.276.62

142.94279.276.62

32.89*64.26* 1.52

Team 11 45.51 4.14

Error 22 95.61 4.35

Total 35 427.00

*p < .01.

Post-hoc analysis of Post-hoc analysis of Technical AdequacyTechnical Adequacy

Behavior Team Team +

F Specialist Alone Specialist

Problem Behavior 3.38 0.95 0.68 0.78

Antecedents identified 4.40 1 .83 1

Identified Function 14.14** 1 0.7 1

Prevention Strategies 16.2** 0.98 0.58 0.98

Teaching Strategies 5.51 0.92 0.68 0.92

Extinction Strategies 55.3** 0.93 0.28 0.84

Positive Reinforcement 5.57 0.94 0.78 0.98

Person Responsible 134.16** 0.93 0.11 0.83

Assess Fidelity 1.5 0.03 0 0.05

Assess Impact 163.83** 0.93 0.13 0.84

** p< Bonferroni family-wise alpha .05

Results: Results: Contextual FitContextual FitMean Team Rating (0-100)Mean Team Rating (0-100)

Mean Team Ratings of Contextual Fit

70

72

74

76

78

80

82

84

86

88

90

Team Alone Team +Spec Spec Alone

Me

an

Co

nte

xtu

al F

it

* Specialist Alone plans were statistically different from plans that included team members.* Team Alone and Team + Specialist plans were not statistically significantly different

Post-hoc Analysis of Contextual Post-hoc Analysis of Contextual FitFit

Contextual Fit Domain F Behavior Team Team +

Specialist Alone Specialist

Knowledge of BSP Elements 15.99** 5.08 5.76 5.67

Skills to perform BSP 9.72 5.35 5.62 5.56

Values consistent with BSP 52.62** 4.69 5.85 5.76

Resources to implement 2.59 4.62 4.89 4.9

Administrative Support 10.68 4.93 5.23 5.32

BSP expected to be effective 29.78** 4.29 5.25 5.4

BSP in best interest of student 30.21** 4.78 5.74 5.77

BSP Efficient to Implement 13.10** 4.32 5.04 4.97

** p< Bonferroni family-wise alpha .05

Implementation Implementation TechnologyTechnologyRole of Coaching

Importance of Policy

Training Outcomes Related to Training Components

Training Outcomes

Training Components

Knowledge of Content

Skill Implementation

ClassroomApplication

Presentation/ Lecture

PlusDemonstration

Plus Practice

Plus Coaching/ Admin SupportData Feedback

10% 5% 0%

30% 20% 0%

60% 60% 5%

95% 95% 95%

Joyce & Showers, 2002

Discipline Foundation Policy: Discipline Foundation Policy: LAUSDLAUSD

School-Wide Positive Behavior Support NUMBER: BUL-3638.0 ISSUER: Donnalyn Jaque-Antón, Executive Officer, Educational Services DATE: March 27, 2007

POLICY: Every student, pre-school through adult, has the right to be

educated in a safe, respectful and welcoming environment. Every educator has the right to teach in an atmosphere free from disruption and obstacles that impede learning. This will be achieved through the adoption and implementation of a consistent school-wide positive behavior support and discipline plan for every school in LAUSD.

All school level discipline plans will be consistent with the Culture of Discipline: Guiding Principles for the School Community (Attachment A) and Culture of Discipline: Student Expectations (Attachment B). This will include: teaching school rules and social-emotional skills; reinforcing appropriate student behavior; using effective classroom management and positive behavior support strategies by providing early intervention for misconduct and appropriate use of consequences.


6. Define practices for scaling up

Efficacy Effectiveness

Dean Fixsen and Karen Blase

Scale Model

Fixsen et al

Leadership Team

FundingVisibility Political

Support

Training Coaching Evaluation

Local Demonstration Schools

Active Coordination

BehavioralExpertise

School-wide PBSSchool-wide PBS Document effects in multiple formats

◦ Single case, Descriptive, RCT, etc.

Documentation via randomized control-group Design◦ Provide research outcomes that address

multiple audiences Families Administrators Teachers Scientist from all disciplines

The Effects of School-wide The Effects of School-wide PBS within a Randomized PBS within a Randomized Control Effectiveness TrialControl Effectiveness Trial

Rob Horner, George Sugai, Keith Smolkowski, Lucille Eber, Jean Nakasato, Anne Todd, Jody Esperansa

OSEP TA Center on Positive Behavior Support

www.pbis.orgIn press in the Journal of Positive Behavior Intervention

Research QuestionsResearch QuestionsCan SWPBS be implemented to

criterion by typical state trainers?

If SWPBS is implemented are schools perceived as safer settings?

If SWPBS is implemented do students benefit academically?

MethodMethodRandomized Control Trial

◦ 30 Elementary Schools in Illinois, and 30 Elementary Schools in Hawaii

◦ Random assignment of schools to (Initial SWPBS training; and Delayed SWPBS

training)◦ Replacement (7 schools) randomly assigned

Data collected across three years◦ Time 1: No SWPBS training for any schools◦ Time 2: Initial Treatment Schools get training◦ Time 3: Delay Schools get training (problems…)

MeasuresMeasures Office Discipline Referrals (ODR)

Implementation of SWPBS◦ School-wide Evaluation Tool (SET)

Sugai et al. Perceived School Safety

◦ School Safety Survey (SSS) Sprague, Colvin & Irvin

Academic Success◦ Proportion of Students Meeting State Reading

Standards (SAT – 9 in Hawaii; ISAT in Illinois)

Randomized Waitlist Controlled Randomized Waitlist Controlled TrialTrial

Assessment Time Period

Group T 1 T 2 T 3

Treatment (N = 30) O X O O

Control/Delay (N = 30) O O X O

(T = time (by year), O = observation, X = implementation of SWPBS training)

Finding #1: Finding #1: Implementation by regular Implementation by regular personnelpersonnelMean SET score (Total) [Mean SET score (Total) [Bold indicates post interventionBold indicates post intervention]]

T1 T2 T3

Treatment Group

.381N = 33

.785N = 30

.823N = 30

Control/Delay Group

.388N= 26

.459N= 27

.640N= 23

Random coefficients analysis (Murray, 1998; Singer & Willett, 2003):

Time X Condition p < . 0001; r = .67; d = 1.78

Random coefficients analysis (Murray, 1998; Singer & Willett, 2003): Time X Condition p < . 0001; r = .67; d = 1.78

ResultsResults: With training by regular state : With training by regular state trainers, schools are able to implement trainers, schools are able to implement SWPBS to criterionSWPBS to criterion..

Implementation of SWPBS

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

T1 T2 T3

Mea

n S

ET

Sco

res

Initial (N = 33) Delayed (N = 28)

Random coefficients analysis: p <.0001; d = 1.78

Initial

Training

Delay

Training

* *

Finding #2: Finding #2: SWPBS is associated with SWPBS is associated with increased perception of safety: increased perception of safety: School Safety School Safety Survey: Risk FactorSurvey: Risk Factor

T1 T2 T3

Treatment Group

.370N= 24

.344N = 29

.343N = 25

Control/Delay Group

.387N = 19

.415N= 24

.358N = 20

Time X Condition p = .0154 r = - .40 d = - .86

ResultsResults: Perceived Social : Perceived Social Risk Factors decreasedRisk Factors decreased when when SWPBS was implemented with fidelity. SWPBS was implemented with fidelity.

Perceived Risk Factor Score from School Safety Survey

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

T1 T2 T3

Mea

n S

cho

ol

Saf

ety

Su

rvey

Ris

k S

core

s

Initial (N = 24) Delay (N = 19)

Random coefficients analysis p = .0154; d = -.86

* *

Finding #3: Finding #3: SWPBS associated with SWPBS associated with increase in proportion of students meeting state increase in proportion of students meeting state reading standardreading standard

T1 T2 T3

Treatment Group

.455N = 33

.529N= 31

.536N = 31

Control/Delay Group

.38N = 28

.402N= 27

.436N= 23

T2 Treatment vs. Control: p = .032 r = .28 d = .58

ResultsResults: The percentage of 3: The percentage of 3rdrd graders graders meeting the state reading standard increased meeting the state reading standard increased with SWPBS implementationwith SWPBS implementation

Percentage of 3rd Graders meeting State Reading Standard

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

T1 T2 T3Per

cen

tag

e o

f 3r

d G

rad

ers

mee

tin

g s

tate

rea

din

g

stan

dar

d

Initial (N = 33) Delay ( N= 28)

N.S. p = .032; d = .58

* *

SummarySummary

◦ Never stop development of the rigorous, precise science of human behavior.

◦ Expand the unit of analysis to address socially relevant outcomes Address the full set of outcomes

defined as important for a context/ community

◦ Expand the research methods/questions to address socially important concerns. Sustainability Scalability

SummarySummaryInvest in a technology of “implementation” that will focus on taking evidence-based practices to scale.

Implementing Evidence-based Practices at Scales of Social Significance Rob Horner University of...

Documents

Transcript of Implementing Evidence-based Practices at Scales of Social Significance Rob Horner University of...