No Place Like Home Cross-Site Evaluation Training.

No Place Like HomeCross-Site Evaluation Training

General Staff Survey

• Please open [enter link] on your computers

• Read the consent page, and if you agree, please click “Next” to begin the survey

• It should take approximately 15 minutes to complete

NPLH Summary

• Evaluation of three child welfare agencies’ use of family meetings with families receiving in-home services– Process

• What does practice look like?• Tools: focus groups, observations, surveys

– Outcomes• Placements (number and type), re-reports,

recurrence of maltreatment, etc.• Satisfaction and service provision• Tools: administrative data, surveys

Your role…

• Is critical!• To provide data and feedback about your

experience working in child welfare and with family meetings

• To recruit families to participate, so that they can provide data and feedback about their experience

• To participate in the feedback loop – ongoing communication

Who What is the value?

…the field •Research is mixed and evidence-base of FGDM is moderate—more research is needed•Additional research can support the national spread of family meetings or suggest the need for alternative practices

…families •Families receive interventions with known effectiveness•Opportunity for families to provide input about their experiences with CPS

What’s in it for…?

Who What is the value?

…CPS Staff and Agencies (You!)

•Accountability. •Opportunity to get input from staff who know the most about the families (You!)•Knowledge of effectiveness of family meetings can increase investment in the practice•Visibility and respect for advancing the field and systematically examining internal practices•Increased knowledge about the use and potential of evaluation•Results can impact system and practice change

What’s in it for…?

Human Subjects

• What is the CITI program?• History and Ethical Principles• The Belmont Report

Human Subjects

• Defining Research with Human Subjects• Federal Regulations • Assessing Risks • Basic Institutional Review Board (IRB)

Regulations and Review Process

Human Subjects

• Informed Consent– Risks to you personally are minimal– There are potential benefits to children, families, and

to staff in the agency – Your participation is voluntary

• Privacy and Confidentiality– All survey information is confidential and used only

for the study– Survey data is maintained by Kempe Center NPLH

evaluation team– Individually identifiable information will NOT be

shared outside of the Kempe Center evaluation team

Human Subjects

• Records-Based Research• Research with Protected Populations –

Vulnerable Subjects• Unanticipated Problems and Reporting

Requirements

Evaluation Basics

• Evidence-Based Practice• Confidentiality vs. Anonymity• Bias• Design Types

– Experimental– Quasi-Experimental

• Process vs. Outcome evaluation– How vs. What

• Qualitative vs. Quantitative methods– Depth vs. Breadth

Evaluation Design - TX

Staff Consent; General Staff

Survey

Families meet eligibility criteria and are consented into the study; Caregiver

Survey

Random Group Assignment

Meeting participants

consented into the study; Fidelity

Survey

Follow Up Survey Follow Up Survey

Case-Specific Questionnaire

Right now

FBSS worker makes FGC referral

FGC

1-2 months post-FGC

FBSS Case closure/transfer to CVS

Evaluation Design - SD


Survey


Survey

Random Group Assignment



Survey



Right now

IFA worker makes FGC or FGC/TDM referral

FGC/TDM; IFA and Ongoing worker attend

1-2 months post-FGC

Ongoing case closure/transfer to out-of-home services

Evaluation Design - CO


Survey


Survey

Group Assignment – Propensity Score Match



Survey


Needs-Services Assessment; Case-

Specific Questionnaire

Right now

Intake or FAR worker makes 1st FSRT or FUM referral

2nd FUM/FGC

1-2 months post-FUM/FGC

Ongoing case closure/transfer to out-of-home services

NPLH Surveys

1. General Staff Survey

2. Caregiver Survey

3. Meeting Fidelity Surveya) Participant

b) Facilitator/Coordinator

4. Case-Specific Questionnaire

5. Follow-Up Surveya) Caregiver – Intervention

b) Caregiver – Control

c) Meeting Participant

Surveys that are distributed TO you

• General Staff Survey–All staff who have a role in the

evaluation (case workers, coordinators/facilitators, supervisors)

• Coordinator-Facilitator Survey–Coordinators/Facilitators only

• Case-Specific Questionnaire–Case-workers only


• Purpose:–To understand more about:

• Who utilizes family meetings in CPS• How much experience staff have with

family meetings• How staff perceive family meetings• The context in which meetings are

happening.–We can use this information to inform

our process and outcome evaluations


• Who? You!– Case workers (FBSS, Ongoing…) providing

in-home services– Family meeting facilitators/coordinators– Supervisors of both

• When? Now

• How? Via Survey Monkey

Coordinator-Facilitator Fidelity Survey

• Purpose:– To understand more about:

• How, why, and when family meetings are used• Who is invited to and who attends family

meetings• Other characteristics of family meetings

–Fidelity–Logistics

–Meeting fidelity is part of our process evaluation

Coordinator-Facilitator Fidelity Survey

• Who? Meeting coordinator/facilitator

• When? After the family meeting (FUM/FGC/TDM)

• How? Via Survey Monkey


• Purpose:–To understand more about:

• What services in-home families receive• Are services provided impacted by family

meetings?–Services provided is one of our outcome

evaluation variables


• Who? Case workers• When? At case closure OR transfer from

in-home services to out-of-home services (foster care, kinship care, etc.)– Transfer does NOT mean voluntary

placements– It does mean a formal or legal change in

custody• How? Via Survey Monkey

– We will send you a ‘tickler’ or reminder

Surveys that are distributed BY you

• Caregiver Survey– By case workers– At face-to-face meetings with family– At point of referral for a family meeting

• Participant Fidelity Survey– By coordinators/facilitators– At family meetings

What you ARE responsible for

1. Informed Consent• Participation is voluntary• Will families agree to be part of the pilot?• If so, they have rights which are outlined in the

Information Sheet, including the right to change their mind at any time.

• If not, no penalty will come to them.

2. Distribute• Paper and pencil surveys with

• Caregiver Survey – self-addressed and stamped envelope with a flyer with a phone number to call for assistance

• Meeting Participant – large self-addressed, stamped envelope for all surveys in the center of the room/table

What you are NOT responsible for

• Explaining the content of surveys to families• Helping families/meeting participants fill out

surveys• People with questions can call the number

provided on the Information Sheet (have extra copies available for them to keep)

• Other household members or family supports, etc. can help.

• In order to maintain confidentiality, it is important that you merely handout surveys and provide no more information than what is included on the information sheets.

Caregiver Survey


• The characteristics of families receiving in-home child welfare services (e.g. demographics)

• The protective capacities of parents • The parent-child relationship and child behavior

– This information can be linked to outcomes

Caregiver Survey

• Who? Parents or legal guardians• When?

– Face-to-face meeting– At time of referral for family meeting

• How?– Paper and pencil– Self-addressed and stamped envelope– Distributed by IFA/FBSS/FAR/Intake

Caseworker

Participant Fidelity Survey


• How, why, and when family meetings are used• Who is invited to and who attends family meetings• Other characteristics of family meetings

–Fidelity–Logistics

– Meeting fidelity is part of our process evaluation

– We want to hear from families and other participants as well as staff.

Participant Fidelity Survey

• Who? Any meeting participant who agrees to fill it out

• When? AT the family meeting– Section 1: Before the meeting starts– Section 2: After the meeting ends

• How? Paper and Pencil– Meeting participants will fill out paper-pencil

surveys and place them in a large envelope– The coordinator/facilitator will mail them to the

evaluation team

Surveys administered by evaluation team

• How? Paper and pencil; via mail• 3 follow-up survey versions with the

following components:– Meeting Follow-Up

• Who? All meeting participants who filled out the initial Meeting Participant Fidelity Survey

– Caregiver Follow-Up• Who? All (intervention and control group) parents

– Family Satisfaction• Who? All (intervention and control group) parents

AND meeting participants

NPLH Survey Summary

• Case workers:– Complete:

• General Staff Survey (1 time)• Case Specific-Questionnaire (for every study case)

– Distribute:• Caregiver Survey (to every study case)

• Coordinators/Facilitators:– Complete:

• General Staff Survey (1 time)• Coordinator-Facilitator Fidelity Survey (for every study

family meeting)

– Distribute:• Meeting Participant Fidelity Survey (for every study family

meeting)

Coordinators/Facilitators Meeting

Log

• [placeholder]

THANK YOU!

If, throughout the life of this

project, you have any questions

please contact:

Heather Allan, NPLH Project Coordinator

Email: [email protected]

Phone: 720.336.8283

mailto:[email protected]

No Place Like Home Cross-Site Evaluation Training.

Documents

Transcript of No Place Like Home Cross-Site Evaluation Training.