Slide 1

The golden thread of evaluation in effective programs and practice Alicia McCoy, Research & Evaluation Manager Jo Cavanagh OAM, CEO Slide 2 About Family Life Slide 3 Research & Evaluation at Family Life o Research and Evaluation Unit since 2009 o Developed organically for first 3 years o Evaluation Capacity Building approach o Program Planning and Evaluation Framework o Training, consultation and support o Family Life Knowledge Hub o Research partnerships o Outcomes Framework (currently being developed) Where are we now? Slide 4 The facts Creating an organisational environment for successful evaluation and social impact measurement requires consideration of: Culture Leadership Communication Systems and structures References: Preskill, H. & Torres, R. (1999) Evaluative Inquiry for Learning in Organisations, SAGE McCoy, A., Rose, D. & Connolly, M. (2014) Approaches to evaluation in Australian child and family welfare organizations, Evaluation and Program Planning, 44, 68-74 McCoy, A., Rose, D. & Connolly, M. (2013) Developing evaluation culture in human service organisations, Evaluation Journal of Australasia, 13, 15-20 Slide 5 How to we get to here! An initial thought Qualitative evidence is the bones, qualitative evidence is the flesh, and evaluative reasoning is the vital organs that brings them both to life -- E. Jane Davidson (2014) Davidson, E.J. (2014) How beauty can bring truth and justice to life. In J.C. Griffith B. Montrosse-Moorhead (Eds.), Revisiting truth, beauty and justice: Evaluating with validity in the 21 st century. New Directions for Evaluation, 142, 31-43. Slide 6 Environment Influencing factors: increased scrutiny articulate impact of services on reducing/influencing social issues in an austerity environment Federal and state governments reviews interest in new ways of funding services; outcome-based funding Evidence-informed programs and implementation vs social innovation Impact measurement e.g. Social Return on Investment Collective Impact Strategic philanthropy Impact investing reliant on evidence of effectiveness; Social Benefit (Impact) Bonds Slide 7 Untangling the golden thread Evidence- informed practice Individual learning Professional ethics Professional standards Organisational ethics Organisational learning Leadership Culture Processes Evaluative inquiry Communicatio n Slide 8 Levels of evaluation use McCoy, A., Rose, D. & Connolly, M. (2014). Approaches to evaluation in Australian child and family welfare organizations. Evaluation and Program Planning, 44, 68-74 [Adapted from Patton 2008] Entirely external Minimal ad hoc internal evaluation Occasional internal evaluation Part-time internal evaluator Full-time internal evaluator Routine internal evaluation Fully integrated and highly valued internal evaluation Slide 9 How organisations use evaluation Slide 10 Considerations for organisations Meeting organisational needs Cost Utilisation Strategy Expertise Organisational understanding Relationship-building and management Bridging the research-practice gap Objectivity and credibility Slide 11 Family Lifes journey Entirely external Minimal ad hoc internal evaluation Occasional internal evaluation Part-time internal evaluator Full-time internal evaluator Routine internal evaluation Fully integrated and highly valued internal evaluation Slide 12 Family Lifes journey (contd) Internal research and evaluation position appointed 2011 Program Planning and Evaluation Framework introduced 20092012 2013 Research and Evaluation Manager joins Executive Managemen t Team 2 questions in agency- wide staff engagemen t survey First research and evaluation climate survey undertaken Outcomes Framework developed Research and evaluation consultation days commence at all service centres Regular presentation s at team meetings, all agency days External research and evaluation partnership s with universities Second research and evaluation climate survey undertaken 2014 Slide 13 Family Lifes approach Evaluation Capacity Building is a context-dependent, intentional action system of guided processes and practices for bringing about and sustaining a state of affairs in which quality program evaluation and its appropriate uses are ordinary and ongoing practices within and/or between one or more organisations/programs/sites -- Stockdill et al (2002) Stockdill, S.H., Baizerman, M. & Compton, D.W. (2002) Toward a definition of the ECB process: a conversation with the ECB literature, New Directions for Evaluation, 93, 7-25 Slide 14 Staff climate survey Slide 15 Challenges Mainstreaming evaluation meaningfully e.g. anything is not good enough Organisational growth impact. Convincing practitioners evaluation is of programs and services, not of individuals Subcultures may undermine the broader effort to create an organisational-wide culture (critics and problem generators Vs curious questioning) Research and evaluation is work for all not a job title Slide 16 What we have learnt There is a tipping point. Meet practitioners where they are at. Different disciplines, evidence-informed practice and evaluation. A sense of safety in learning - constructive feedback is welcome and beneficial Know your evaluation champions. Slide 17 Trust in practitioners knowledge. Provide a hook make evaluation interesting and relevant Be comfortable with diverse views and focus efforts strategically What is paid attention to, and publicly valued is what will get done. What we have learnt (contd) Slide 18 Personal relationships are important Attitudes toward evaluation are not binary. Market evaluation with this in mind Make the evaluation message noticeable but not overwhelming Make messages about staff roles and responsibilities clear and consistent Leaders are the ones that drive culture Communication is key Slide 19 A final reflection Evaluation must become part of the culture, building a common language. If these things happen, evaluation behaviour will follow. Changes in values lead to changes in behaviour We are asking people to mainstream a frame of mind, to ask good questions, be sceptical, use answers to bring about change, to continue to move toward excellence in practice -- James Sanders (2001) Sanders, J.R. Reflections on mainstreaming evaluation. Comments presented during Session 790 at the annual meeting of the American Evaluation Association, St Louis, November 10 2001 Slide 20 Thank you Alicia McCoy Research and Evaluation Manager amccoy@familylife.com.au Jo Cavanagh OAM CEO www.familylife.com.au