Tools That Work: Improving Child Welfare Services Through Research, Performance Measurement, and

Information TechnologyNovember 12, 2003

Best Practice Outcomes in Behavior Support and Intervention:

A Multi-site Evaluation

Lloyd Bullard

Katie Johnson

Background

Physical Risks– Hartford Courant series documents 142

deaths related to the use of restraint and seclusion

• Harvard Center for Risk Analysis estimates 50-150 deaths per year are related to the use of restraint and seclusion

Background Psychological Risks

– Consumers describe restraint and seclusion as dehumanizing and humiliating (Binder and McCoy, 1983)

– Children and adolescents who have been restrained in psychiatric hospitals report nightmares, intrusive thoughts, avoidance responses, and marked startle responses (General Accounting Office, 1999).

– Restraint and seclusion have been identified as having the negative neuro-biological effects associated with trauma and re-traumatization

Background

Charles Curie and the Pennsylvania Initiative

Children’s Health Act of 2000

SAMHSA develops RFP

– Three-year grant beginning October, 2001

– Selects five demonstration sites and one coordinating center

The Coordinating Center Partnership:

CWLA and FFCMH

Child Welfare League of America– Establish an Advisory Committee– Provide Technical Assistance– Conduct Evaluation– Disseminate Findings– Develop Tools and Resources

The Coordinating Center Partnership:

CWLA and FFCMH Federation of Families for Children’s

Mental Health– Provide Technical assistance to sites

– Participate on the National Advisory Committee

– Conduct key informant groups and provide comprehensive report of focus group activities

– Establish Family Advocate National Advisory Committee

Demonstration Sites The Methodist Home for Children and Youth: Macon, GA Girls and Boys Town National Resource and Training

Center: Boys Town, NE– A.B. and Jessie Polinsky Children’s Center: San Diego, CA

Lakeside Treatment and Learning Center: Kalamazoo, MI University of Alabama Brewer Porch Children's Center:

Tuscaloosa, AL Connecticut Collaboration for Training Excellence

– Klingberg Family Centers: New Britain, CT– Devereux Glenholme School: Washington, CT– Riverview Hospital: Middletown, CT

Project Goals

Year One: Identify best practices for reducing restraint and seclusion use

Year Two: Reinforce staff training with organizational support

Year Three: Disseminate findings

Evaluation Methodology Individual Incident Data

– Individual variables (e.g. race, gender, time of day, incident precipitant, etc.) recorded for each incident of emergency physical intervention

Standardized Measures – Measures (e.g. Organizational Climate, Family

Centeredness Assessments, etc.) evaluated at specific time periods

Qualitative Measures– Reports to SAMHSA, significant events

timelines & process meetings

Challenges to EvaluationDemonstration Sites Data collection worthiness -

for the purposes of SAMHSA research

– Incident tracking

– Isolating intervention effects

Over exposure

– Risk management

Administratively burdensome

– Program rigidity

Coordinating Center Uniformity

– Common definitions– Common data points– Common measures

Complex design– Multiple sites– Multiple baselines– Multiple interventions– Multiple crises training

modules (e.g., TCI, CPI, etc.)

Isolating intervention effects– Provide context to outcomes

Evaluation Tools Key Informant Focus Groups Organizational Climate Assessment

– 14 dimensions that identify organizational factors affecting outcomes

Assessment of Policy and Procedure– Based upon Best Practice Guidelines For Behavior

Management

Family Centeredness Assessment– Family-Professional Partnership Scale developed by

University of Kansas– Trieschman Carolinas Project Instrument (TCPI)

Process Assessment and Feedback Data Collection

Current Evaluation Outcomes Key Informant Focus Group Findings

– The seclusion and restraint process needs to be explained more thoroughly.

– The training of staff to handle seclusion and restraint incidents is critical to avoiding injury.

– Family members need to be a part of the service planning process.

– Family members need to be notified in a timely way of any incidents.

– Youth in facilities need to be able to file grievances.

Current Evaluation Outcomes Organizational Climate Assessment

– Staff from two sites successfully reducing restraint and seclusion were rated highly in the following categories:

• Ability to get things done, meet the needs of their clientele, and use funds efficiently.

• Ability to respond to change, flow of communication, compensation, competency, morale, personnel evaluation process, training, organizational values, and mission to serve youth.

• Quality of relationship with supervisor and perceived standards of services rendered.

Current Evaluation Outcomes

Assessment of Policy and Procedure– Overall Policy Findings

• The findings revealed that there was little difference between the site’s staff and the external site consultant’s assessment ratings.

– What Sites are Doing Best• Consultants and site representatives rated the sites the highest

on policies governing Legal and Ethical concerns and Professional Development.

– What needs Improving• Representatives and consultants indicated complete agreement

that the sites were relatively weakest on their written policies governing Medical Issues.

Current Evaluation Outcomes Family Centeredness Assessment

– Families indicate that it is “very important” to them that they have good working relationships with their service provider.

– Families indicate that they are “satisfied” with their relationship with their child’s primary service provider.

Current Evaluation Outcomes Process Assessment and Feedback

– Sites attributed trends in physical intervention data to the following:

• Temporal Factors• Policy and Practice Factors• Staff Factors• Child Factors• Training Factors

Current Evaluation Outcomes

All Sites: Raw Number of Restraint and Seclusion Incidents

0100200300400500600700800900

1000

Jan-01 Apr-01 Jul-01 Oct-01 Jan-02 Apr-02 Jul-02 Oct-02 Jan-03 Apr-03 Jul-03Mechanical Restraint Physical Restraint I Physical Restraint II Seclusion

Log. (Mechanical Restraint) Log. (Physical Restraint II) Log. (Seclusion)

Note: Six sites submitted data regarding physical restraint between J an. 2001 and J une 2002. The seventh site began submitting data in J uly 2002, which accounts for the break in the physical restraint line during that month.Note: The dashed line is a logarithmic trend line denoting general trends in the numbers of restraints and seclusions since data collection began.

Current Evaluation OutcomesNon-Medical Community-Based Residential Facilities:

Raw Numbers of Restraint and Seclusion Incidents

0

100

200

300

400

J an-02 Apr-02 J ul-02 Oct-02 J an-03 Apr-03 J ul-03

Physical Restraint SeclusionLog. (Physical Restraint) Log. (Seclusion)

Note: The dashed line is a logarithmic trend line denoting general trends in the numbers of restraints and seclusions since data collection began.

Current Evaluation Outcomes Overall Practice Findings

– The frequency of incidents largely coincides with seasonal events. – The use of seclusion and mechanical restraint has decreased. – The number of physical restraint incidents has increased sharply over the

past six months. – Overall, injuries to children and staff have increased.

What Sites are Doing Best– Multiple preemptive behavior management techniques are attempted with

the youngest children.– There has been no increase in medication management as reported by

most sites.– The duration of seclusion and mechanical incidents is decreasing.

What Needs Improving– Property damage accounts for 20% of all behavior management

incidents.– Injuries to staff and child occur more often in restraints that require

moving the child.– Ethnic disparities exist in the use of seclusion and physical restraint.

Program Resources CWLA Best Practice in Behavior Support and

Intervention Assessment Instrument Trieschman Carolinas Project Instrument: A

Questionnaire For Family Centered Group Care Practice

Effective Supervisory Practice: Behavior Support and Intervention for Children and Youth

National Task Force on Behavior Support and Intervention Training Guidelines

Reducing the Use of Restraint and Seclusion: Promising Practices and Successful Strategies

Other Resources

Project Resources CWLA Best Practice in Behavior Support and Intervention

Assessment Instrument– Aims to help agencies improve their behavior support and intervention

policies, procedures and practices through careful self-assessment

Example: The agency has a philosophy governing the use of restrictive procedures.

• Any behavior management intervention is first based on the premise of “first, do no harm.”• Nonphysical interventions are always the first choice, unless safety issues demand an immediate physical

response.• Caregivers are involved in developing and/or selecting a behavior intervention model.• Caregivers support the philosophy governing the use of restrictive procedures as indicated by their

knowledge of that philosophy, by their adherence to the philosophy in daily practice, and by their completion of all relevant documentation.

A A written philosophy governing the use of restrictive procedures is present, and it is consistently reflected in all areas of agency practice.

B A written philosophy governing the use of restrictive procedures is present, but only partially reflected in agency practice.

C A written philosophy governing the use of restrictive procedures is present, but poorly reflected in agency practice.

D A written philosophy governing the use of restrictive procedures is absent or clearly inadequate.E Not applicable.

Program Resources Trieschman Carolinas Project Instrument:

A Questionnaire For Family Centered Group Care Practice– A staff survey to assess an agency’s

commitment to family-centered group care Example: To protect the interests of children they must remain connected

to their parents regardless of circumstances.

Strongly agree Strongly disagree (1) (2) (3) (4) (5)

Program Resources Effective Supervisory Practice: Behavior Support

and Intervention for Children and Youth– Synthesizes the best practices that supervisors can

employ in order to reduce the use of restraint and seclusion

Competencies:• Supervisors understand that positive organizational climate

is necessary to an effective behavior management program.

• Supervisors know how to support staff in developing the skills and attitudes that are essential to successful implementation of an effective behavior management program.

• Supervisors understand their role and essential skills and attitudes in successful implementation of an effective behavior management program.

Program Resources National Task Force on Behavior Support and

Intervention Training Guidelines• Creation of guidelines that will assess behavior

support and intervention training programs. Proposed Chapters:

1. Organizational Culture and Leadership

2. Behavior Support Training Programs

3. Risk Factors

4. Emergency Physical Interventions

5. Training Process

Program Resources Reducing the Use of Restraint and

Seclusion: Promising Practices and Successful Strategies

– An issue brief that annotates policies and practices that successfully reduce the use of restraint and seclusion

Chapters• Leadership• Organizational Culture• Agencies’ policies, procedures and practices • Staff Training and Professional Development• Treatment Milieu • Continuous Quality Improvement

Program Resources Website Annotated Bibliography Standards/Definitions for Restraint and Seclusion Report Card Vols. 1 to Vol. 5 CWLA Best Practice Guidelines for Behavior

Management

Recommendations and Next Steps

Creation of a national resource center that• Acts as a clearinghouse that creates/disseminates

evidenced-based practices• Compiles research and data on restraint and

seclusion issues• Conducts trainings and provides technical

assistance to states, providers, and consumers, and

• Provides national database of restraint and seclusion incidents, program outcomes, etc.

For Further Information

Lloyd Bullard

Project Director

(202) 942-0280

Coordinating Center for Behavior Support and Intervention

behaviormanagementtraining@cwla.orgor

http://www.cwla.org/programs/behavior/

Darren Fulmore

Evaluation Coordinator

(202) 639-4904

Katie Johnson

Research Assistant

(202)639-4910