Understanding the Move Toward Evidence-Based Programs: Considerations for Early Childhood Leaders...

Understanding the Move Toward Evidence-Based Programs:

Considerations for Early Childhood Leaders

Michelle Hughes, MA, MSWProject Director, Benchmarks

September 28, 2011

Today’s Presentation

• Why evidence-based programs?– Why all the fuss about EBPs?– Changing Landscape of Practice– What does evidence-based actually mean?

• Getting to outcomes: More than choosing an evidence-based program off a list!!– NIRN Framework for Assessing EBPs – Implementation, Implementation, Implementation

(Webinar #2)• Resources and Questions

Evidence-based programs….

What’s all the fuss?

Some Contributing Factors• Increased focus on accountability in multiple

sectors. • Increased knowledge about affecting change.

– More information about “what works” and what doesn’t

– Increased focus on measuring community-level impact (are we moving the “community needle”)

• Stewardship of public/philanthropic dollars: limited resources need to be used strategically.

The Changing Landscape

• Focus on EBPs seen across all fields, in public and private sectors, at all levels of government

• This is a learning process – changing systems and practice is a long-term (and challenging) endeavor.

At the National Level• Increased focus/guidance/policy change with regard to

evidence in federal agencies (SAMSHA, CBCAP, etc.)

• Nonprofits and Academic Institutions: Coalition for Evidence-Based Policy, Child Trends, Center for Early Literacy Learning, National Implementation Network

• “What Works” Sites: Promising Practices Network, California Clearinghouse (child maltreatment); Blueprints for Violence Prevention; National Registry of Effective Evidence-Based Programs and Practices (substance abuse, mental health)…

And in North Carolina…• Most public agencies and many private

philanthropic agencies are increasingly focused on supporting EBPs– NC Division of Social Services, CBCAP and IV-B, part 2

funding for prevention and family support; – NC Division of Public Health, Evidence-Based Home

Visiting Initiative– The Duke Endowment, Kate B. Reynolds Charitable

Trust– Alliance for Evidence-Based Family Strengthening

Programs launched Nurse-Family Partnership, Incredible Years, and Strengthening Families Program

In Smart Start…• NCPC has been encouraging local partnerships to move

toward the use of ebp’s for several years. • In 2011 NC General Assembly passed legislation.

“State funding for local partnerships shall also be used for evidence-based or evidence-informed programs for children from birth to five years of age that: – Increase child’s literacy– Increase the parents’ ability to raise healthy, successful

children– Improve children’s health– Assist four and five star rated facilities in improving and

maintaining quality”

Smart Start DefinitionsNCPC Board of Directors adopted definitions:•Evidence-based programs or practices are those that have repeatedly and consistently demonstrated desirable outcomes through application of scientific research methods (replicated experimental, experimental, or quasi experimental)•Evidence-informed practice is one that is guided by child development theory, practitioner wisdom, qualitative studies, and findings from basic research and has written guidelines, a strong logic model, and a history of demonstrating positive results. They may be rated “Promising” or “Emerging” by at least one source that rates evidence-based programs.

Why the focus on “rigorous” scientific research methods???

• I don’t get it…if I’m getting good pre-test, post-test results, doesn’t that prove what I’m doing is working?

It is a great first step, but we know from research that some programs that show good preliminary results may not be as effective as we think when we do more rigorous evaluation.

How Do You Know If Something Works?

• A Quick Review of Evaluation Methods…bear with me for a few minutes!!

• A program or practice “works” if it produces the intended positive results (outcomes).

• We can determine if something works through evaluation methods…but some methods (like experimental studies) are better at determining “cause-effect” than others. Let’s talk about why….

Levels of Evidence

• Continuum of evaluation methods: Non-experimental, quasi-experimental, experimental.

• Non-experimental: anecdotes, qualitative interviews, case studies, surveys, pre-post assessments without comparison/control groups.

• These are IMPORTANT methods that contribute to our knowledge base about program effectiveness but they are insufficient for determining if a program is effective. Why???

Here’s why….• A non-experimental design cannot determine if the

positive effects seen in a program are the result of the program, or some other cause. – Might occur anyways without the intervention; – Might occur because of some other characteristics about

the group participating in the intervention;– Might occur because of chance.

• How can we determine cause and effect with confidence? – We need a group to compare with (i.e., a “control” group). – Helps you determine if outcomes are caused by other

factors besides the intervention

Experimental Research Methods• Quasi-experimental: Use a non-randomized comparison

group in which intervention and comparison group are closely matched (e.g., demographics, SES, etc.). • Differences in outcomes between groups can be attributed to

the intervention with some degree of confidence, depending on strength of study design.

• Experimental: Considered the “gold standard” in determining cause and effect. Randomly assign a target population to an experimental or a control group. • Differences in outcomes can be attributed to the intervention

with high degree of confidence if strong study design.

FRIENDS Diagram of Evidence

Cited from FRIENDS Website, http://friendsnrc.org/cbcap-priority-areas/evidence-base-practice-in-cbcap

• Used with child abuse prevention programs funded through Federal Community-Based Child Abuse Prevention Program

• NC Division of Social Services uses for prevention funding

Some Examples of Why RCT ImportantMedicine: •Hormone replacement therapy for post-menopausal women (increases risk of stroke and heart disease for many women)•Dietary fiber to prevent colon cancer (shown ineffective)•Stents to open clogged arteries (shown no better than drugs for most heart patients).

Cited from Jon Baron, NC Family Impact Seminar, 2009.

More Examples…Social Welfare:•Job Training Partnership Act

– Pre-post scores of the men who went through the job training program indicated an increase in earnings (a positive finding), but when program participants were compared to a control group who did not go through the training, program participants demonstrated smaller increase in earnings than control group.

Prevention : : •DARE

– ENORMOUS support for DARE. At one point, DARE taught in 80% of the school districts in the United States, in 54 other countries around the world, and was taught to 36,000,000 students each year (Hanson, D J., cited from www.alcoholfacts.org)

– Pre-post assessments indicate increased student knowledge of risks, and improved attitude changes toward using alcohol & drug use, however, RCTs revealed program has no effect in reducing alcohol/drug use.

Cited from Jon Baron, NC Family Impact Seminar, 2009.

Assessing Research on Programs• Assessing the research on different programs

can be overwhelming.

• Who has time to do literature reviews?

• Effect size; sample size; attrition; validated measures….OH MY!!

Some Steps to Take….• Use existing resources and registries

(resources at end of presentation) who have already done this work

• Use NCPC state office staff and other resources (Sarah Currier/Emmy Marshall at Prevent Child Abuse NC; Catherine Joyner at DPH; Center for Child and Family Health; Duke Center for Child and Family Policy)

• Educate yourself to become a smart consumer of research

Getting to Outcomes…

• So, once I find a program that is evidence-based or evidence-informed I am ready to go right??

• Let’s take a step back and use a framework developed by the National Implementation Research Network to talk about assessing EBPs for your community.

Not exactly…but you are headed in the right direction!

EBP: 5 Point Rating Scale: High = 5; Medium = 3; Low = 1. Midpoints can be used and scored as a 2 or 4.

High Medium Low

Need

Fit

Resources Availability

Evidence

Readiness for Replication

Capacity to Implement

Total Score:

Need in Agency, SettingSocially Significant IssuesParent & Community Perceptions of NeedData indicating Need

Need

Fit

Fit with current - •Initiatives• State and Local Priorities• Organizational structures• Community Values

ResourceAvailability

Resource AvailabilityITStaffingTrainingData SystemsCoaching & SupervisionAdministrative & system supports needed

EvidenceOutcomes – Is it worth it?Fidelity dataCost – effectiveness data Number of studiesPopulation similaritiesDiverse cultural groupsEfficacy or Effectiveness

Evidence

Assessing Evidence-Based Programs

and Practices

Intervention Readiness for ReplicationQualified purveyor Expert or TA availableMature sites to observe# of replicationsHow well is it operationalized?Are Imp Drivers operationalized?

Intervention Readiness

for Replication

Capacity to ImplementStaff meet minimum qualificationsAble to sustain Imp Drivers • Financially• Structurally• Buy-in process operationalized• Practitioners• Families• Agency


© National Implementation Research Network 2009 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland


Need

Fit





Evidence


and Practices



for Replication




Evidence


Evidence


Need

Fit





Evidence


and Practices



for Replication




Need



Need

Fit





Evidence


and Practices



for Replication




Evidence

• What is the need in my community?

• What are we trying to change?

• What program might address those needs and help us affect those changes (outcomes)?

• What is the evidence that this program will indeed meet those needs and result in those desired outcomes?

Need


Need

Fit





Evidence


and Practices



for Replication




• Is that program a good “fit” for my agency and community?

• Does it fit with:

• Current initiatives?• State and local

priorities?• Organizational

mission and values?• Organizational

structure?• Community values?

Evidence

Need

Fit

Now Let’s Stop for a Moment….

Implementation, Implementation, Implementation….and….IMPLEMENTATION!!!

Getting Positive Outcomes Means..

•Recognize that “evidence” is necessary but insufficient for achieving positive outcomes for children and families. •We must focus on supporting community-based agencies and practitioners in…..

Effective Implementation

Implementation MATTERS! • Successfully replicating evidence-based programs

requires….

Proven practice + fidelity/quality implementation

= Better Outcomes

• Fidelity: adherence to core elements which contribute to program effectiveness

• A poorly implemented program will yield poor outcomes (and results in a poor investment)


Need

Fit





Evidence


and Practices



for Replication




Evidence

Need

Fit

• Is the program we’ve chosen ready for replication?

• What does the program we’ve chosen require for successful implementation?

• What capacity do we have to implement?

Stay tuned….• Next Webinar on October 12th at 10:00 am.

Getting Results with Evidence-Based Programs: The Importance of Effective Implementation. – Focused the science of implementation – what do we

know about effective implementation and how to support it?

– What are the key “drivers” for successfully implementing a program or practice?

– How can these key “drivers” not only help us implement a specific EBP – but build our agencies’ capacity to offer an array of services to support children and families in our communities?

In Summary…• The trend toward evidence-based programs is

here to stay. • You are not alone in trying to figure this out! • Moving toward EBPs is a process of capacity

building for everyone! (staff, agency leaders, Boards of Directors, your community partners, NCPC state office)– Becoming a good consumer of research – Using state and national resources– Building organizational capacity to implement EBPs

(which strengthens the entire agency!)

Summary continued…

• Getting good outcomes is NOT about choosing a program off a list. It IS about building the capacity of your organization and your partnerships with the community to implement effectively.

• There are a lot of resources to help you with this work.

ResourcesFRIENDS National Resource Center for Community-Based Child Abuse Prevention. http://www.friendsnrc.org/Provides information, online training, toolkits on evidence-based programs, evaluation, logic models and continuous quality improvement. Resources would be useful for any agency focused on family support programs and practices.

Promising Practices Network www.promisingpractices.netProvides evidence-based information about what works to improve the lives of children, youth, and families. Features summaries of programs and practices proven to improve outcomes for children.

National Implementation Research Network (NIRN)http://www.fpg.unc.edu/~nirn/Provides extensive research and information on the science of implementation, including several frameworks for understanding effective implementation and examples from the field.

Several resources listed cited from 2009 Duke University Family Impact Seminar

http://www.friendsnrc.org/

http://www.friendsnrc.org/

http://www.fpg.unc.edu/~nirn/

ResourcesSocial Programs that Work www.evidencebasedprograms.org/Summarizes the results of well-designed randomized controlled trials in social policy that have particularly important policy implications. Organized into the following substantive areas: early childhood, K-12 education, youth development, crime/violence prevention, substance abuse prevention and treatment, mental health, employment and welfare, and international development.

National Registry of Evidenced-based Programs and Practices, SAMHSA nrepp.samhsa.govOffers a searchable database of interventions for the prevention and treatment of mental and substance use disorders. Meant to help people, agencies, and organizations implement effective programs and practices in their communities.

Blueprints for Violence Prevention, Center for the Study and Prevention of Violence www.colorado.edu/cspv/blueprintsIdentifies outstanding violence and drug prevention programs that meet a high scientific standard of effectiveness. Serves as a resource for governments, foundations, businesses, and other organizations trying to make informed judgments about their investments in violence and drug prevention programs.

RESOURCESNorth Carolina Family Impact Seminar, Duke Universityhttp://familyimpactseminars.org/index.asp?p=1&page=seminar&seminarid=181&siteid=34This family impact seminar held in 2009 at Duke University provides several powerpoints presentations on evidence-based policy and programs nationally and in North Carolina, and a publication with good overview of evidence-based policy and program development.

Washington State Institute for Public Policy, Olympia, WA www.wsipp.wa.gov/topic.asp?cat=19&subcat=0&dteSlct=0Summarizes evidence-based programs and cost-benefit analyses for interventions in the following areas: developmental disabilities, child welfare/child maltreatment prevention, children’s mental health, criminal justice, juvenile justice, and K-12 education.

NCPC Resources• Program Staff

– Ann Ward, 919-821-9556, [email protected]– Cynthia Turner, 919-821-9565, [email protected]– Gale Wilson, 919-821-9563, [email protected]– Lois Slade, 919-821-9577, [email protected]

• Evaluation Staff– Leigh Poole, 919-821-9580, [email protected]– Meshell Reynolds, Counts & reporting, 919-821-9567,

[email protected]

• Smart Net Resources– Program & Evaluation tab> folders for Evidence-Based/Informed

programs, & for Family Support, Health, & Early Care & Education

Presenter Contact Information

Michelle Hughes, MA, MSWProject Director, [email protected](919) 357-7361www.benchmarksnc.org

Understanding the Move Toward Evidence-Based Programs: Considerations for Early Childhood Leaders...

Documents

Transcript of Understanding the Move Toward Evidence-Based Programs: Considerations for Early Childhood Leaders...