Conceptual framework and a research agenda in the transition

from efficacy to effectivenessIrwin Sandler

ASU Prevention Research Center

P30 MH068685-01

Overview

• What we learn from efficacy – New Beginnings Program

• Effectiveness conceptualized • Expanded conceptualization of effectiveness• Research agenda on transition from efficacy to

effectiveness: NBP• Research studies to bridge from efficacy to

effectiveness– Marketing study– Court factors study

Efficacy: New Beginnings Program

• Efficacy trial tests whether a program does more good than harm when delivered under optimal conditions (Flay, 1986)

New Beginnings Program Efficacy Trial (NBP) Selection process• Extensive recruitment methods• Participants agreed to accept randomization and assessment• Residential mothers meeting 9 eligibility criteria

Implementation process• 11 session manualized (scripted) program• Delivered at ASU Prevention Center• Implemented by 13 selected group leaders• Training (30 hours, plus 1.5 hours prior to each session)• Supervision (videotaped sessions, 1 hour supervision per wk)

Selection and Implementation

Selection• 36% of eligible families enrolled in trial• Higher SES; 88% non Hispanic white• Low drop out rate from intervention (11%)Implementation• Very high (90% +) level of program

implementation• Very low variability in implementation

Randomized Efficacy Trial: Reduce Multiple Problem Outcomes Six-Years Later

Effects on outcomes over 6 years:– 36% reduction in 1-year prevalence of diagnosed

mental disorder on C-DISC – Reduce total psychiatric symptoms– Reduce Marijuana, Alcohol and Drug Use for

higher risk group– Improve Grade Point Average– Reduce number of sexual partners

Effects are mediated by improved parentingEffects strongest for high risk groupDemonstrates viability of concept but not

delivery in natural settings

Effectiveness Trial(Flay, 1986)

• “…concerned with whether program does more harm than good when delivered via a real world program”.

• Effects of trial may be due to:– Selection (acceptance): rate of acceptance; who

accepts (program x person interaction)– Implementation - fidelity, adaptation

• Both acceptance and implementation are multi-level concepts that apply to the program and the organization that delivers the program

Transition from Efficacy to Effectiveness

• Research agenda for making the transition involves developing systems to study real world conditions in which it will be delivered and design program to optimize successful implementation

• Prevention programs often don’t have pre-existing delivery systems – but host institutions with which they share an interest and through which they are delivered

• Identify Domestic Relations Court as the Organizational Context– Access to the population– History of innovative services– Overlapping (though not identical) interests with prevention

Research Agenda• Do the courts want this collaboration?

Current practices and factors influencing adoption

• Identify factors influencing selection and implementation of NBP using quality management concepts and methods – research with consumer Court factors Cultural factors Provider agency factors

• Design program and system for robust delivery through courts Optimizing court, culture, provider factors that influence implementation Optimize participant selection processes Pilot test selection and redesigned program and receive feedback on

viability

• Effectiveness trial to test whether the program have the desired effects Design of multi-court effectiveness trial

Market Study: National Survey of Courts (Cookston, Braver, Sandler & Genalo, 2002)

Question: What is the current practice and readiness of county courts to adopt “evidence-based” parenting program in the court?

Background: o 1556 Counties provide divorce parenting

services in 1998o 11 states require divorcing families to participate

in parenting programso Programs are generally short (4 hours) and

evaluations of efficacy are limited

Method for Survey of Courts

• Stratified random sample drawn of 154 courts with parenting program

• Interview with key informant who is most knowledgeable about the program

• Questions addressed:Description of current programs offeredPlanned changes in programs Support for parenting program like NBPBarriers to implementation

Support and Opposition for Lengthier

Parent Education Programs Influence Group Support* Oppose^

Judges 77% (N = 115) 11% ( N = 16)

Members of the bar 73% (N = 111) 14% (N = 21)

County supervisors 71% (N = 102) 17% (N = 25)

State legislators 63% (N = 89) 22% (N = 31)

Child advocacy groups

96% (N = 148) 3% (N = 4)

Local child and family agencies

97% (N = 147) 1% (N = 1)

State Supreme Court 87% (N = 111) 7% (N = 9)

* - percent of respondents who indicated the influence group would support lengthier parent education programs^ - percent of respondents who indicated the influence group would not support lengthier parent education programs and would, in fact, oppose such programs

0% 20% 40% 60% 80%

Funding

Finding Presenters

Allocating Space

Cooperation from Judges

Support from Court

Adequate Staff

Lack of Community Interest

Parent Attendance

Showing Value of Program

Attorney Support

Ba

rrie

r

Percentage of counties

Barriers To The Implementation Of Lengthier Divorcing Parent Education

Program

Court Factors Affecting Adoption and Implementation

• National Advisory Group of Court Leaders – judges, court administrators, mediators, court service providers, executive director of national professional organization (AFCC)

• Identified by core working group of program advocates – program developers, director of mediation services

National Advisory Board Role

• Provide advice and guidance to “translate the NBP ‘experimental prototype’ into a service that fits as a service for divorcing families through the Family Court.”

• Three stage processa) Understand issues in translation into a

real-world serviceb) Advise on specific aims of effectiveness

trial and assessment of court readinessc) Advise on protocol for court recruitment

Advisory Board Input on Court Level Selection and Implementation Issues (2/03)

• Respond to program selection scenario

“You have just received an RFP from your state government for a parenting program to be delivered through your court. Two proposals for parenting education have been submitted and you must choose between them. What factors are likely to make you select one program over another?”

Factors that Influence Program Selection

• Members independently listed 65 factors

• Group discussion of factors – reduced to 24 factors

• Factors grouped into six categories by research team

• Categories rank ordered by the members

Rank Order of Program Selection Factors

1. Compatibility and resource investment (compatible with court priorities, cost of implementation)

2. Program efficacy (broad support from stakeholders and meet needs of parents)

3. Perceived program credibility (e.g., Reputation from other courts)

4. Program structure and content (e.g., number of sessions; program content)

5. Program delivery quality (e.g., mode of presentation; accompanying material)

6. Program accessibility (avg. rank = 4.7) (e.g., customer access; multicultura/multilingual)

Factors that Influence Quality of Implementation

• Respond to quality of implementation scenario.

“When you receive your state funding award one of the stipulations for continuation of future funding was assuring high quality of implementation of this parenting program. What factors are most likely to influence quality of implementation in your court?”

Rank Order of Quality of Implementation Factors

• Staffing (on-site coordinator; administrator – audits, details of administration)

• Stakeholder/consumer feedback (ongoing; evaluation of impact on consumer and the court)

• Training (Training and motivating providers) • Expanded reach of program (expand range of

providers and participants)• Program visibility and acceptance (Advisory

committee; publicity)• Program facilities (appropriate space)

Summary

• Research for transition to effectiveness trial involves study of court, provider, cultural, selection factors and design system to optimize selection and implementation

• Braver will present studies to evaluate alternative systems to recruit participants

• MacKinnon will discuss issues in the effectiveness trial – focusing on role of mediational analysis

Service Providers

NewBeginningsProgram

Domestic Relations Court

Families

System for Implementation of New Beginnings Program Through Domestic Relations Courts

Families

Training, technical assistanceand monitoring

SelectionMethods

Stakeholder input

Families