TOOLKIT | December 2018 Formative Evaluation ... TOOLKIT | December 2018 Formative Evaluation...

download TOOLKIT | December 2018 Formative Evaluation ... TOOLKIT | December 2018 Formative Evaluation Toolkit

of 70

  • date post

    18-Mar-2020
  • Category

    Documents

  • view

    2
  • download

    0

Embed Size (px)

Transcript of TOOLKIT | December 2018 Formative Evaluation ... TOOLKIT | December 2018 Formative Evaluation...

  • TOOLKIT | December 2018

    Formative Evaluation Toolkit A Step-by-Step Guide and Resources for Evaluating Program Implementation and Early Outcomes

  • TOOLKIT | December 2018

    Formative Evaluation Toolkit A Step-by-Step Guide and Resources for Evaluating Program Implementation and Early Outcomes

    Submitted to

    Chereese Phillips, Ph.D. Child Welfare Program Specialist Children’s Bureau Administration for Children and Families U.S. Department of Health and Human Services

    Prepared by

    James Bell Associates 3033 Wilson Blvd., Suite 650 Arlington, VA 22201 (703) 528-3230 www.jbassoc.com

    Elliott Graham, Ph.D. Project Director

    This document is in the public domain. Permission to reproduce is not necessary. Suggested citation: James Bell Associates. (2018). Formative evaluation toolkit: A step-by-step guide and resources for evaluating program implementation and early outcomes. Washington, DC: Children’s Bureau, Administration for Children and Families, U.S. Department of Health and Human Services.

    Disclaimer

    This publication was developed by James Bell Associates on behalf of the Children’s Bureau, Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS), under contract number HHSP233201500133I, order number HHSP23337001T. Its contents are the sole responsibility of the author and do not necessarily represent the official views of the Children’s Bureau, ACF, or HHS.

    For more information, please visit the Children’s Bureau website at https://www.acf.hhs.gov/cb.

    http://www.jbassoc.com/ https://www.acf.hhs.gov/cb

  • Formative Evaluation Toolkit ii

    Acknowledgments The authors wish to thank several individuals for their support in the production of this toolkit. Catherine Heath of the Children’s Bureau, U.S. Department of Health and Human Services, was instrumental in spearheading this effort and helped inform its development through her work with Children’s Bureau discretionary grantees that received Planning Grants to Develop a Model Intervention for Youth and Young Adults with Child Welfare Involvement At-Risk of Homelessness (HHS-2013-ACF-ACYF-CA-0636). We are also grateful to Serena L. Williams, Ph.D., Matthew McGuire, and Melinda Baldwin, Ph.D., of the Children’s Bureau; to Jan P. Shafer, Director of the Children’s Bureau’s Division of Program Innovation; as well as to John Denniston, formerly with ICF/Child Welfare Information Gateway, for their insights and guidance. We would also like to express our appreciation to the technical assistance and evaluation teams of the Permanency Innovations Initiative; their work was influential in developing the content of many sections of the toolkit.

  • Formative Evaluation Toolkit ii

    ................................................................................................................

    ..................................................................

    .............................................................................................................................. ...............................................................................................................................

    .............................................................................................................. ................................................................................................................

    ................................................

    .................................................................... ...........................................................................................

    ......................................................... .

    ..................... ..........................................

    ............................ ....................................................

    .................................................................................................................... ................................................................................

    ......................................................................................... ..........................................................................

    .....................................................................

    .....................................................................

    ...................................................................... ...........................................

    .......................................................... .......................................................................

    ................................................................................... ............................................................................

    ................................................................................................ ..........................................................................................................................

    ............................................................................................ ..................................................................................................... ....................................................................................................

    Contents Introduction 1

    I. Understanding Formative Evaluation 2

    1. Exploration 4 2. Installation 4 3. Initial Implementation 5 4. Full Implementation 5

    II. Meeting Preconditions of Formative Evaluation 7

    1. Do you have an implementation team in place? 7 2. Have you set realistic time lines? 8 3. Do you have a clear theory of change based on data? 8 4. Have you selected a program that fits your needs and aligns with your theory of change? 9 5. Have you developed a logic model and evaluated it for plausibility and gaps? 9 6. Have you operationalized the program and installed supports? 10 7. Have you identified needed data elements and installed data systems? 11 8. Have you conducted usability testing to address “bugs”? 12 Section II Examples 14

    Washington County Theory of Change 15 Washington County Logic Model 16 Washington County Usability Testing Plan 17 Washington County Usability Testing Results 19

    III. Conducting Formative Evaluation 21

    1. Identify Key Formative Evaluation Questions 22 Is the program reaching the intended number of participants? 22 How are inputs contributing to program functioning? 22 Is the program being delivered as intended? 23 Are short-term outcomes promising? 24

    2. Select Indicators of Program Functioning 24 3. Choose Evaluation Methods 25 4. Collect Data 26

    What are your response rates? 27 5. Analyze and Report Data 27 6. Decide What Comes Next 27

    .

  • Formative Evaluation Toolkit iv

    .......................................................... ...................................................................................................................

    .............................................................. .....................................................................

    ............................................................ ............

    ..................................

    ................................................................................................................

    ....................................................................................................

    ........................................................... ........................................................................

    .................................................................................. ..................................................................................................

    ............................................................................ ....................................................................................

    .................................................................. ...................................

    ..............................................................................................

    ................

    ...........................................................................

    ..................................................................

    .....................................

    .......................................................

    ......................................................

    ........................

    ..............................................

    Section III Examples 29 Washington County Formative Evaluation Time Line 30 Washington County Program Enrollment Tracking 31 Washington County Response Rate Tracking 32 Washington County Short-Term Outcome Tracking 34 Washington County Formative Evaluation Questions and Indicators/Benchmarks 35 Washington County Presentation of Formative Evaluation Findings 38

    References 44

    Appendix A. Tools 46

    Tool II.1. Readiness for Formative Evaluation Checklist 47 Tool II.2. Refining Your Theory of Change Tool 54 Tool II.3. Refining Your Logic Model Tool 55 Tool II.4. Usability Testing Tool 57 Tool III.1. Program Enrollment Tracking Tool 58 Tool III.2. Response Rate Tracking Tool 59 Tool III.3. Short-Term Outcome Tracking Template 61 Tool III.4. Formative Evaluation Questions and Indicators/Benchmarks 62

    Appendix B. Glossary 63

    Exhibits Exhibit 1. Key Differences Between Formative and Summative Evaluation 2