The Policing Evaluation Toolkit 2018-10-31¢  The Policing Evaluation Toolkit Page 6 of 33...

download The Policing Evaluation Toolkit 2018-10-31¢  The Policing Evaluation Toolkit Page 6 of 33 Version 1.0

of 33

  • date post

    08-Jun-2020
  • Category

    Documents

  • view

    0
  • download

    0

Embed Size (px)

Transcript of The Policing Evaluation Toolkit 2018-10-31¢  The Policing Evaluation Toolkit Page 6 of 33...

  • The Policing Evaluation Toolkit Professor Stuart Kime, Director of Education, Evidence Based Education

    Levin Wheller, Research and Analysis Standards Manager, College of Policing

    Version 1.0

    November 2018

  • The Policing Evaluation Toolkit

    Page 2 of 33 Version 1.0

    © – College of Policing Limited (2018)

    This publication is licensed under the terms of the Non-Commercial College Licence v1.1

    except where otherwise stated. To view this licence visit

    http://www.college.police.uk/Legal/Documents/Non_Commercial_College_Licence.pdf

    Where we have identified any third-party copyright information, you will need to obtain

    permission from the copyright holders concerned. This publication may contain public sector

    information licensed under the Open Government Licence v3.0 at

    www.nationalarchives.gov.uk/doc/open-government-licence/version/3/

    This publication is available for download at:

    http://whatworks.college.police.uk/Support/Pages/Evaluation-Toolkit.aspx

    Any enquiries regarding this publication please contact us at the College on 0800 4963322

    or email contactus@college.pnn.police.uk

    This document has been created with the intention of making the content accessible to the

    widest range of people regardless of disability or impairment. To enquire about having this

    document provided in an alternative format please contact us at the College on

    0800 4963322 or email contactus@college.pnn.police.uk

    http://www.college.police.uk/Legal/Documents/Non_Commercial_College_Licence.pdf http://www.nationalarchives.gov.uk/doc/open-government-licence/version/3/ http://whatworks.college.police.uk/Support/Pages/Evaluation-Toolkit.aspx mailto:contactus@college.pnn.police.uk mailto:contactus@college.pnn.police.uk

  • The Policing Evaluation Toolkit

    Version 1.0 Page 3 of 33

    Contents

    Introduction ......................................................................................................................... 5

    About this toolkit ................................................................................................................ 5

    Evaluation and Evidence-Based Policing ........................................................................... 5

    What is evaluation? ........................................................................................................... 5

    How long does an evaluation last? .................................................................................... 6

    What do we lose without effective evaluation? ................................................................... 6

    How can the Evaluation Toolkit help me? .......................................................................... 7

    How to use this toolkit ........................................................................................................ 7

    Stage 1: Preparation (before the evaluation begins) .......................................................... 7

    Stage 2: Implementation (during the evaluation) ................................................................ 7

    Stage 3: Analysis and Reporting (after the evaluation ends) .............................................. 8

    System change projects ..................................................................................................... 9

    Stage 1: Preparation ......................................................................................................... 10

    Preparation Stage 1.1: Frame your evaluation question................................................... 10

    Examples of evaluation questions ................................................................................ 10

    Logic models ................................................................................................................ 10

    Tips on framing evaluation questions ............................................................................ 11

    Preparation Stage 1.2: Identify the measures you will use ............................................... 11

    Validity and reliability .................................................................................................... 11

    Seven tips for developing good outcome measures: ..................................................... 12

    Administrative data ....................................................................................................... 12

    Questions you should be able to answer about your evaluation .................................... 12

    Case study: Selecting measurements for a crime reduction intervention ...................... 13

    Preparation Stage 1.3: Identifying and recruiting participants/selecting locations ............. 14

    Recruiting participants .................................................................................................. 14

    Selecting locations ........................................................................................................ 14

    Reducing bias ............................................................................................................... 14

    Involving stakeholders .................................................................................................. 15

  • The Policing Evaluation Toolkit

    Page 4 of 33 Version 1.0

    Preparation Stage 1.4: Generating a comparison group .................................................. 15

    Evaluation design A: Random allocation ....................................................................... 17

    Evaluation design B: Matched comparison groups and locations .................................. 22

    Questions to be answered before moving to Stage 2 (Implementation) ........................ 23

    Stage 2: Implementation ................................................................................................... 24

    Think ‘EMMIE’ ................................................................................................................. 24

    Implementation Stage 2.1: Conduct a baseline test ......................................................... 25

    How baseline data are useful ....................................................................................... 25

    Implementation Stage 2.2: Implement the intervention..................................................... 26

    Implementation Stage 2.3: Process evaluation ................................................................ 26

    Process evaluation case study ..................................................................................... 27

    Implementation Stage 2.4: Conduct a post-test ............................................................... 28

    Stage 3: Analysis and Reporting ..................................................................................... 30

    Analysis and Reporting Stage 3.1: Analyse your data ...................................................... 30

    Effect sizes ................................................................................................................... 30

    Interpretation ................................................................................................................ 30

    Potential difficulties with interpretation .......................................................................... 31

    Analysis and Reporting Stage 3.2: Reporting your results ............................................... 31

    What should you report? ............................................................................................... 31

    Adapt your practice ....................................................................................................... 33

    Share your findings ....................................................................................................... 33

  • The Policing Evaluation Toolkit

    Version 1.0 Page 5 of 33

    Introduction

    About this toolkit

    This is a practical toolkit which brings together evaluation design and implementation

    strategies. It can be used by practitioners and researchers to ensure evaluations are

    designed in such a way that strong causal statements about effectiveness can be made.

    We are keen to receive feedback from users of the toolkit to help us develop the content and

    ensure it is as useful as possible. If you have any feedback on the toolkit, please get in touch

    via email: research@college.pnn.police.uk

    Evaluation and Evidence-Based Policing

    Evaluation is a component of an Evidence-Based Policing (EBP) approach in which “police

    officers and staff create, review and use the best available evidence to inform and challenge

    policies, practices and decisions”.1 The College provides a range of resources to support

    people across policing and crime reduction to use evidence-based approaches.

    What is evaluation?

    Evaluation is a set of methods and tools which enable officers and staff to answer the

    questions ‘Did it work?’, ‘To what extent did it work?’, and ‘Why and how did it work?’

    However, not all evaluations are the same: the quality of design and the execution matter.

    Rigorous planning and execution are hallmarks of effective evaluation. In this toolkit, you will

    learn