RMIT Midterm Evaluation of Independent Family Advocacy and ...€¦ · Web viewThe evaluation...

Midterm Evaluation of Independent Family Advocacy and Support (IFAS) pilot

Midterm Report

March 2020

Dr Chris Maylea, Sherie Thomas, Lucy Bashfield, Associate Professor Bawa Kuyini, Professor Susie Costello, Meena Singh

Social and Global Studies Centre, RMIT University

© Social and Global Studies Centre, RMIT

This work may be reproduced in whole or in part for study or training purposes subject to acknowledgement of the source and is not for commercial use or sale. Reproduction for other purposes or by other organisations requires the written permission of the authors.

Suggested citation: Maylea, Chris; Thomas, Sherie; Bashfield, Lucy; Kuyini, Bawa; Costello, Susie; Singh, Meena (2020) Midterm Evaluation of Independent Family Advocacy and Support (IFAS) pilot, Melbourne: Social and Global Studies Centre, RMIT University.

ii

ContentsExecutive Summary...............................................................................................................................................vi

1. Introduction...................................................................................................................................................8

1.1. Background............................................................................................................................................8

1.1.1. Establishment of IFAS....................................................................................................................8

1.1.2. IFAS service model.........................................................................................................................9

1.2. Evaluation Methodology......................................................................................................................10

1.2.1. Participant driven.........................................................................................................................10

1.2.2. Co-location...................................................................................................................................10

1.2.3. Expert reference group................................................................................................................10

1.2.4. Aims and scope............................................................................................................................11

1.2.5. Key evaluation questions.............................................................................................................11

1.3. Data Collection.....................................................................................................................................12

1.3.1. Review of IFAS documentation....................................................................................................12

1.3.2. Annotated bibliography and legislative review............................................................................12

1.3.3. Case bank review.........................................................................................................................13

1.3.4. Interviews with people who have used or are eligible for IFAS...................................................13

1.3.5. Professional stakeholder interviews and focus groups................................................................13

1.3.6. Quantitative data.........................................................................................................................14

1.4. Data analysis........................................................................................................................................15

1.5. Limitations...........................................................................................................................................15

1.5.1. IFAS database...............................................................................................................................15

1.5.1. Measures of performance success...............................................................................................15

1.5.2. Priority group referrals................................................................................................................16

1.5.1. DHHS data....................................................................................................................................16

1.6. Ethics...................................................................................................................................................16

2. Preliminary evaluation findings...................................................................................................................17

2.1. The IFAS experience.............................................................................................................................17

2.2. IFAS program measures.......................................................................................................................18

2.3. Context................................................................................................................................................21

2.4. Elements of the IFAS Model.................................................................................................................22

2.4.1. Representational advocacy..........................................................................................................22

2.4.2. Rights-based advocacy.................................................................................................................24

iii

2.4.3. Coaching for self-advocacy..........................................................................................................24

2.4.4. Systemic advocacy.......................................................................................................................25

2.4.5. Support beyond advocacy............................................................................................................26

2.4.6. Diverting from court....................................................................................................................26

2.4.7. Mediation and facilitation............................................................................................................28

2.4.8. Professional collaboration and colocation...................................................................................30

2.5. Other Specific Findings........................................................................................................................31

2.5.1. Cost benefits................................................................................................................................31

2.5.2. Co-design and lived experience...................................................................................................31

2.5.3. Access to legal services................................................................................................................32

2.5.4. Assisting with compliance............................................................................................................33

2.5.5. Advocacy targets..........................................................................................................................33

2.5.6. Children........................................................................................................................................34

3. Conclusion...................................................................................................................................................35

4. Interim recommendations...........................................................................................................................36

5. Appendices..................................................................................................................................................38

Appendix 1. IFAS Logic Model......................................................................................................................38

Appendix 2. List of IFAS documents reviewed.............................................................................................39

Appendix 3. Question Matrix.......................................................................................................................40

Appendix 4. IFAS Monitoring and Evaluation Matrix...................................................................................41

Appendix 5. Cost-benefit analysis................................................................................................................47

5.1. Cost-benefit analysis methodology......................................................................................................47

5.2. Cost savings from pre-court diversion.................................................................................................50

5.1. Qualitative data on court diversion.....................................................................................................52

5.2. Quantitative data on court diversion...................................................................................................52

5.3. Priority group court diversion..............................................................................................................55

5.4. Cost savings from shorter DHHS Child Protection investigations.........................................................56

Appendix 6. Mapping against key evaluation questions..............................................................................57

iv

Table of figuresFigure 1 - Level 1 services provided.....................................................................................................................18

Figure 2 – Increase/decrease in number of open files IFAS Levels 2-5...............................................................19

Figure 3 - Ages range of people using IFAS..........................................................................................................20

Figure 4 - Priority group representation..............................................................................................................20

Figure 5 - Other priority group representation....................................................................................................20

Figure 6 – System level impact logic....................................................................................................................49

Figure 7 - Case expenditure by pilot site with linear trend..................................................................................53

Figure 8 - Interpolated and actual case expenditure, Pilot LGAs.........................................................................54

Figure 9 – Case expenditure, Pilot LGAs, Comparison LGAs and Rest of Victoria................................................54

Table of tablesTable 1 - Incoming referrals.................................................................................................................................19

Table 2 - Costs of child protection court involvement.........................................................................................51

Table 3 - Return on investment (ROI) at 4% success (whole of government)......................................................51

Table 4 - Return on investment (ROI) at 4% success (VLA only)..........................................................................51

Table 5 - Minimum return on investment...........................................................................................................52

Table 6 - Return on investment (ROI) at 5% and 10% success.............................................................................55

v

https://rmiteduau-my.sharepoint.com/personal/chris_maylea_rmit_edu_au/Documents/IFAS/Interim%20report/Midterm%20Evaluation%20of%20Independent%20Family%20Advocacy%20and%20Support.docx#_Toc43041356









Executive SummaryThis report presents the findings from the midterm review of the Independent Family Advocacy and Support (IFAS) pilot run by Victoria Legal Aid (VLA). IFAS is a pilot service that provides non-legal advocacy and support to parents and primary carers who are involved in the early stages of the child protection system. Two main priority groups for IFAS are Aboriginal and Torres Strait Islander families and families where one or both of the parents has an intellectual disability. IFAS aims to divert families away from the child protection system and to increase access to legal services for the priority groups. The two IFAS pilot areas are Greater Bendigo and Darebin/Moreland.

The evaluation is in two parts, a midterm review and a final evaluation. This report presents the findings from the midterm review. Qualitative data has been collected using interviews and focus groups with people who have used IFAS, IFAS staff and professional stakeholders. Quantitative data has been provided by IFAS and VLA and sourced from other publicly available data.

Overall, the IFAS pilot has been very successful. People who used IFAS highly valued the service. People who had used IFAS reported finding the child protection system difficult to navigate, unpleasant and sometimes traumatic. Advocates performed a mediation and facilitation role, assisting people to navigate the system, improving communication and resulting in better outcomes. Stakeholders agreed on the potential for IFAS to make an impact on the broader child protection system but agreed that no actual systemic change was evident.

There are opportunities to improve stakeholder understanding of the service and to refine the model to suit the context of child protection. Advocates have an excellent understanding of the child protection context and system but require further support to embed rights-based approaches and systemic reform into the IFAS model. The lived experience aspects of the model require more support and integration.

While only limited data are available at this stage, the available data indicates that IFAS has a positive return on investment. More time is required to established this with confidence. DHHS data will also help address this question in the final evaluation.

The evaluation was limited by the quality and availability of data. DHHS could not produce the data requested within the contracted timeframes and so this data will be included in the final stage of the evaluation. Inconsistent or inaccessible IFAS data limited some aspects of the evaluation. People from IFAS’s priority groups, Aboriginal and Torres Strait Islander families and families where a parent has an intellectual disability were underrepresented in the group of potential participants referred to the evaluation team. The evaluation team will work with IFAS to address this prior to commencing the final stage of the evaluation.

IFAS has an Evaluation and Monitoring Framework, however this has no set targets as it was developed prior to the implementation of the service. The evaluation team will work with IFAS to develop a robust evaluation framework with measurable targets prior to commencing the final stage of the evaluation.

The final stage of the evaluation will focus on collecting qualitative and quantitative data from the Department of Health and Human Services (DHHS) and consulting with people who have used IFAS and belong to the priority groups. People who are or were eligible for but who have not used IFAS will also be consulted.

vi

AcknowledgmentsThis evaluation was conducted on Aboriginal land, which was never ceded. The Social and Global Studies Centre at RMIT University acknowledges the Australian Aboriginal and Torres Strait Islander peoples of the nations of Victoria, the custodians of this land. We pay our respects to ancestors and Elders, past and present. We are committed to honouring Australian Aboriginal and Torres Strait Islander peoples’ unique cultural and spiritual relationships to the land, waters and seas and their rich contribution to society.

The evaluation team would like to thank all participants who gave considered feedback, and Victoria Legal Aid (VLA) and IFAS for committing to an independent and transparent evaluation process. The evaluation team also acknowledges the work of previous consultants, VLA and IFAS staff who developed the program logic and objectives used as the basis of this evaluation.

Abbreviations and TermsA# Advocate

ACCO Aboriginal Community Controlled Organisations

BDAC Bendigo and District Aboriginal Co-operative

CB Case Bank

DHHS Department of Health and Human Services (Victoria)

FIN Vic Family Inclusion Network Victoria

IFAS Independent Family Advocacy and Support

IM IFAS Manager

IMHA Independent Mental Health Advocacy

Law# Lawyer

LGA Local Government Area

p.a. per annum

PS# Professional Stakeholder

RMIT Royal Melbourne Institute of Technology University

SEAS Shared Experience and Support

SU# Service User

VACCA Victorian Aboriginal Child Care Agency

VALS Victorian Aboriginal Legal Service

VALID Victorian Advocacy League for Individuals with Disability

VLA Victoria Legal Aid

vii

1.IntroductionThis report presents the preliminary findings from the evaluation of the Independent Family Advocacy and Support (IFAS) service run by Victoria Legal Aid (VLA). IFAS is a pilot, fully funded by VLA, implemented to demonstrate the efficacy of non-legal advocacy to families who are involved with DHHS Child Protection but not yet involved in the court system. It is a three-year pilot, running from 2018 to 2021, in the local government areas (LGAs) of Darebin and Moreland in suburban Melbourne and Bendigo in regional Victoria.

In 2019, VLA conducted a formal competitive tendering process to evaluate IFAS. RMIT University’s Social and Global Studies Centre was contracted to evaluate IFAS in two stages: a midterm review from November 2019 to March 2020 and a final review from March 2021 to May 2021. This report presents the findings from the midterm review. This section outlines the background and establishment of IFAS, the evaluation methodology, data collection and analysis undertaken by the evaluation team.

1.1. BackgroundIFAS began on 30 October 2018 in Melbourne and November 2018 in Bendigo. The IFAS team consists of three advocates, one of whom is the team leader, a part time Lived Experience Advisor, administration support and a manager. One advocate is based at the Bendigo VLA office with the rest of the team based at the Melbourne VLA office. IFAS has a lived experience reference group called Shared Experience and Support (SEAS), coordinated by the Lived Experience Advisor.

1.1.1.Establishment of IFASIFAS was established in response to Action 7 and Action 8 of the Child Protection Legal Aid Services Review completed in 2017, which committed VLA to piloting an early intervention non-legal advocacy service for client groups over-represented in the child protection system, and linking them to legal advice where needed. 1

Action 7

Victoria Legal Aid will pilot an Early Intervention Unit2 featuring non-legal advocates, where the following are considered a priority:

cases for which a Protection Application by Notice would ordinarily proceed; families where one or both primary carers have an intellectual disability; Aboriginal and Torres Strait Islander families; Aboriginal Family-led Decision Making Conferences.

Action 8

Legal assistance will be available to clients of the Early Intervention Unit to obtain legal advice about their child protection case.

1 Victoria Legal Aid, Child Protection Legal Aid Services Review (2019).

2 Early Intervention Unit/EIU was the original working title for the project. Independent Family and Advocacy Support (IFAS) was decided on by the Reference Group, and confirmed by the project sponsor (Nicole Rich) in May 2018.

8

1.1.2. IFAS service modelIFAS service is provided on five distinct levels:

Level 1 (General Information) – open to anyone for information, including services and workers; Level 2 (Information and Referrals) – for any parent, family member or worker, who has involvement with

Child Protection; Level 3 (Coaching for Self-Advocacy) – parent or parents who identify as:

≈ Aboriginal/Torres Strait Islander (or are parents to an Aboriginal/Torres Strait Islander child);≈ Having a disability;≈ Having an intellectual disability;≈ Having a mental health issue;≈ In a family situation where family violence is present;≈ Culturally and/or linguistically diverse;≈ LGBTQI+;

Level 4 (Direct Advocacy and Coaching for Self-Advocacy) – parent or parents who identify as:≈ Aboriginal/Torres Strait Islander (or are parents to an Aboriginal/Torres Strait Islander child); ≈ Having an intellectual disability;

Level 5 (Complex Direct Advocacy and Coaching for Self-Advocacy) – parent or parents who identify as:≈ Aboriginal/Torres Strait Islander (or are parents to an Aboriginal/Torres Strait Islander child);≈ Having an intellectual disability.

IFAS defines ‘parent’ as someone who is a biological parent, stepparent and/or acting as a child’s primary carer. Levels 2-5 are intended for parents who:

Have a substantiated report with Child Protection; Are subject to a Protection Application by Notice, Temporary Assessment Order OR have ongoing

engagement with DHHS; Have not been to court related to the substantiation; Are the subject of an unborn report.

While in the establishment phase, where capacity exists, IFAS has provided support to parents who do not meet all of the above criteria to ensure the best use of resources.

The IFAS service model incorporates a representational advocacy model. IFAS policy documentation defines representational advocacy as:

… presenting the client’s views and preferences to others in order for their voice to be amplified in settings where they are in a position where they have less power to impact on decisions that directly affect their rights and lives.

This can be viewed as distinct from best-interests advocacy, where the advocate determines what is ‘best’ for the person or family and then advocates for that. IFAS advocates do not make value-based decisions about a person’s goals, share information without the person’s explicit consent, or take any action without the specific direction of the person. This model is based on the model of advocacy used by the Independent Mental Health Advocacy

9

service, also run by VLA.3 The IFAS principles and values are Independence, Empowerment, Impartiality, Confidentiality, Free and Voluntary, Inclusive and Accessible and Strength-Based.

1.2. Evaluation MethodologyThis evaluation employed a mixed-methods approach, combining qualitative thematic analysis of primary and secondary data with descriptive and analytic statistical data to help address the range of evaluation questions. This methodology was developed from the methodology used for the successful evaluation of IMHA, also conducted by Dr Maylea.4 The methodology used for the cost-benefit analysis is detailed in Appendix 5.

1.2.1.Participant drivenThe evaluation has been implemented using principles of participatory co-design and co-production, with a focus on valuing and responding to the lived experience of the people who are currently or may in the future use IFAS’s service. This approach is based on principles of equality, diversity, accessibility, reciprocity and mutuality. Shared Experience and Support (SEAS) (the IFAS lived experience reference group), and the IFAS Lived Experience Consultant were consulted throughout the evaluation. The evaluation team includes an evaluator with lived experience of the child protection system.

1.2.2.Co-locationThe lead consultant (Dr Maylea) was co-located on-site at IFAS from November 2019 to March 2020. This embedded model facilitated a deeper level of engagement with the IFAS team. Dr Maylea and the other team members were able to observe organisational processes and provide ongoing feedback during the project, in consultation with IFAS.

1.2.3.Expert reference groupThe evaluation is guided by an expert reference group including representatives from stakeholder organisations and recognised experts in the field. These are:

Alexander di Giorgio and Kylie Ponchard (VACCA); Carol Clark (Northern Area Mental Health Service FaPMI (Families where a Parent has a Mental Illness);

coordinator and Family Inclusion Network Vic committee member); Cassie Cox – (Anglicare); Mary Kyrios (DHHS – Manager Child Protection Policy Unit); and Sarah Forbes (Advocacy Manager VALID).

1.2.4. Aims and scope The overarching aim of this evaluation is to detail the extent to which IFAS is providing effective, efficient, sustainable independent advocacy services. Efficacy, efficiency and sustainability are understood as interlinked and interdependent, with service quality being achieved by balancing these factors.

3 Chris Maylea et al, Evaluation of the Independent Mental Health Advocacy Service (IMHA) (Social and Global Studies Centre, RMIT University, 2019) 56 <sway.office.com/GZJrEJJcVJZlGGvY>.

4 Ibid.

10

bookmark://_Toc531958569


Both IFAS delivery sites are within the scope of this evaluation, covering the Darebin/Moreland and Bendigo regions. All people who have used or who are eligible for IFAS, IFAS staff and other stakeholders are within the scope of this evaluation. People who have accessed Level 1 services were not considered IFAS service users for this evaluation, although Level 1 services are within scope. The evaluation team considered data relating to Level 1 service provision but did not interview any Level 1 service users.

1.2.5. Key evaluation questionsThe program logic model and these key evaluation questions informed the evaluation approach for the midterm review (see Appendix 1 and Appendix 2). The midterm review sought to begin to answer all of the key evaluation questions and to identify what additional or alternative data is necessary to collect prior to the final evaluation.

1) Has IFAS successfully achieved its aims? a) To establish and trial a pilot independent child protection non-legal advocacy service, providing

representational advocacy services to parents involved in the child protection system and develop evidence through robust monitoring and evaluation to determine impact and outcomes; and,

b) To develop and deliver the service working closely with key stakeholders including families and individuals with lived experience and other key stakeholders.

2) Has IFAS successfully achieved its objectives? a) To enhance clients’ capacity to self-advocate, understand and enact their rights within the child

protection system, better understand their current involvement with child protection, and make informed decisions about their family;

b) To assist clients to access support services they identify they require; c) To ensure that clients receive high-quality services and have a positive experience of the advocacy

service; d) To enable clients to obtain legal advice about their child protection case from qualified legal

practitioners; and e) To support a reduction in the proportion of substantiated cases in the pilot area proceeding to

court, proceeding to court as emergency care applications or proceeding to contested interim proceedings.

3) Is IFAS reducing the number of matters which proceed to court? 4) Is IFAS better supporting families through the child protection process than a family’s previous

experience of child protection without an advocate? 5) Is IFAS delivering a better experience for children in the child protection process? 6) Is IFAS operating consistently with its values and principles? 7) Is IFAS delivering advocacy consistent with its advocacy model? 8) Is IFAS operating consistently with its service model? Is this service model the most appropriate for

IFAS? 9) Is IFAS operating consistently with its policy and procedures? 10) Is IFAS having a systemic impact?

See Appendix 6 for a detailed mapping of IFAS performance against each of the key evaluation questions. These questions have also been mapped against each participant group to inform data collection (see Appendix 3). All evaluation questions will be reviewed in the final evaluation plan in reference to the revised Monitoring and Evaluation framework.

11


1.3. Data CollectionThe evaluation employed a suite of data collection methods including:

A review of IFAS documentation; Annotated bibliography and legislative review; Case bank review; Interviews with people who have used or are eligible for IFAS; Professional stakeholder interviews and focus groups; and Quantitative data.

In total, the evaluation team consulted with 41 participants, 30 via interview and 11 via three focus groups. Data collection generally proceeded as planned.

1.3.1.Review of IFAS documentationThe evaluation team reviewed all available IFAS documentation, including the initial Child Protection Legal Aid Services Review which led to IFAS, the Child Protection Early Intervention Unit Project Plan which established IFAS and all IFAS policies and procedures. See Appendix 2 for a complete list of documents reviewed.

1.3.2.Annotated bibliography and legislative review An annotated bibliography and legislative review were conducted. The key findings from the annotated bibliography were that advocacy services in child protection are not widespread and research on their effectiveness is limited but promising.5 Models varied between legal preventative advocacy, non-legal professional advocacy and lived experience advocacy.6 There is insufficient literature to indicate that any particular model is preferred and to date no systematic literature reviews or meta analyses have been conducted. The legislative review highlighted the predominance of the principle of the best interests of the child and some attempts by legislators to protect parents’ rights but offered little to the early intervention context in which IFAS operates. These findings influenced the design of the evaluation and the preliminary recommendations and will be revisited in the final evaluation report.

1.3.3.Case bank reviewIFAS advocates each developed advocacy examples based on successful outcomes, which are stored in the IFAS case bank. These advocacy examples are intended to highlight IFAS best practice and are not necessarily representative of all IFAS practice or common IFAS outcomes. They have been used in this evaluation to construct a model of best practice to be tested against data from other service users and stakeholders.

5 Susan Collings et al, ‘“She Was There If I Needed to Talk or to Try and Get My Point across”: Specialist Advocacy for Parents with Intellectual Disability in the Australian Child Protection System’ (2018) 24(2) Australian Journal of Human Rights 162; Cora Peterson et al, ‘Cost-Benefit Analysis of Two Child Abuse and Neglect Primary Prevention Programs for US States’ (2018) 19(6) Prevention science : the official journal of the Society for Prevention Research 705.

6 H Douglas and T Walsh, ‘Mothers and the Child Protection System’ (2009) 23(2) International Journal of Law, Policy and the Family 211; Vivek Sankaran, ‘Using Preventive Legal Advocacy to Keep Children from Entering Foster Care A Critical Look at Child Protection’ [2013] (3) William Mitchell Law Review 1036; Beth Tarleton, ‘Expanding the Engagement Model: The Role of the Specialist Advocate in Supporting Parents With Learning Disabilities in Child Protection Proceedings’ (2013) 7(5) Journal of Public Child Welfare 675.

12


1.3.4. Interviews with people who have used or are eligible for IFASTo recruit participants who had used IFAS, VLA approached past IFAS users on behalf of the evaluation team. This was done in two ways, with IFAS advocates directly inviting people to participate when they ended their engagement, and via a randomised process whereby the IFAS administration support staff or an IFAS social work student directly contacted past users of IFAS. This approach was adopted to both maximise referral numbers and minimise referral bias.

Of a total possible 85 eligible service users, 19 people were referred to the evaluation team, 11 who were interviewed. Eight participants referred by IFAS were not able to participate in an interview or not able to be contacted. Of the 19 referred by IFAS to the evaluation team, only one person identified as Aboriginal and one identified as having an intellectual disability. Engaging these two priority groups will be the focus of the next stage of the evaluation.

Interviews for participants who had used IFAS were all led by a lived experience evaluator with the support of an academic evaluator.

People who are eligible for IFAS but have not used it will be consulted in the next phase of the evaluation.

1.3.5.Professional stakeholder interviews and focus groupsIn total, 24 professional stakeholders were consulted, 13 via interview and 11 via three focus groups. An additional focus group was held with the IFAS staff team. Professional stakeholders can be grouped as follows:

6 IFAS staff (2 advocates, 1 team leader, 1 Lived Experience Advisor, 1 student, 1 manager); 2 stakeholders from Family Inclusion Network Victoria (FIN Vic); 1 stakeholder from First Peoples’ Health Service; 1 stakeholder from Berry Street; 1 stakeholder from Elizabeth Morgan House; 6 stakeholders from the Victorian Aboriginal Child Care Agency (VACCA); 4 stakeholders from Shared Experience and Support (SEAS) (IFAS reference group); 5 stakeholders from the Bendigo and District Aboriginal Co-operative (BDAC); and 4 stakeholders from VLA Child Protection team.

Professional stakeholders were recruited both by referral from IFAS staff and by the evaluation team directly contacting identified stakeholder organisations. This approach was adopted to ensure that key stakeholders were able to feed into the evaluation and to reduce referral bias.

One stakeholder referred by IFAS was not able to participate in an interview.

Due to DHHS approval processes, no DHHS staff could be interviewed at this stage of the evaluation. DHHS staff will be a key focus of the next stage of the evaluation.

1.3.6.Quantitative dataQuantitative data were collected from multiple sources, primarily:

IFAS internal data including:≈ Status reports;≈ Master spreadsheet; and≈ Client surveys.

13

VLA internal data including:≈ Child protection case expenditure by demographic;≈ Child protection substantive grants by demographic; and≈ VLA financial modelling on child protection costs.

Publicly available data including:≈ 2019-20 Victorian State Budget;≈ 2018-19 Children’s Court of Victoria Annual Report;≈ Children’s Court of Victoria Court Performance Child Protection Cases;≈ Child Protection Australia 2017–18 Australian Institute of Health and Welfare;≈ Child Protection and family services Additional service delivery data 2017–18;≈ Child Protection and Family Services Quarterly incident reporting;≈ Productivity Commission Report on Government Services - Child Protection Services; and≈ Australian Bureau of Statistics demographic data.

Quantitative data are of limited value at this stage of the evaluation. This is largely due to the lag between impact on the system and data collection. It is anticipated that at the point of the final review, IFAS will have been operational for sufficient time to see an impact in the quantitative data, if one exists. As shown below, there are some promising trends in the available data, but more time is required to determine if these changes are sustained.

In addition, the available external data often reflect large unexplained variations and periodic fluctuations, making IFAS’s impact difficult to ascertain. This likely reflects changes in internal DHHS Child Protection process, practices or resource allocation. Some data sets are inconsistent with others, using different collection protocols or practices. For example, DHHS Child Protection data are collected per child with multiple potential children per family, VLA data are collected per client with multiple potential clients per family, or per file, with multiple potential files per client, while IFAS collects some data only for the principal parent and other for the secondary parent. Some data are poorly or inconsistently collected by DHHS or VLA, such as Aboriginal and/or Torres Strait Islander identification and intellectual disability. Data collection tends to have improved in some areas over some years, suggesting increased numbers where none exists. For these reasons, the quantitative data have been interpreted with caution.

The IFAS program plan assumed that IFAS would be able to use established VLA databases. As this has not proved possible for technical and budgetary reasons, IFAS has developed an ad hoc data collection system using a combination of Microsoft Word and Excel documents. This has resulted in less than ideal data consistency despite the best attempts of the IFAS team. Some data are collected per primary contact person, with other data including partners. Some data are collected as monthly totals, while others are incremental increases over the lifetime of the program. No collated data are collected per child, making data matching with DHHS data difficult and potentially impossible. The current system is not adequately protected or securely backed up, meaning an accidental change or technical malfunction could erase or corrupt the existing data. The evaluation team has made a number of recommendations regarding this (Recommendation 2).

1.4. Data analysisAll professional interviews and focus groups were conducted by the lead evaluator, with service user interviews conducted by a lived experience evaluator and supported by the lead evaluator. The interviews and focus groups

14

were audio recorded, professionally transcribed and loaded into NVivo qualitative analysis software. All qualitative data, including interviews, focus groups and advocacy examples, were coded against the research questions by one evaluator, then all interview and focus group data were thematically coded by two evaluators using well-rehearsed conventions of thematic analysis.7 The thematic coding was then synthesised by the lead evaluator. This ensured that the data were handled by at least two evaluators, not including those who conducted the interviews. In total, the evaluation team coded 212671 words to 2375 references.

Qualitative data were processed in Microsoft Excel and SPSS quantitative analysis software.

1.5. Limitations The evaluation deviated from the initial plan in response to issues that arose during the evaluation. This has resulted in limitations which relate to the IFAS database, the lack of concrete measures of success, low numbers of priority group referrals and the inability to access DHHS data. The evaluation team will continue to work with IFAS and VLA to improve data collection, to co-develop a robust evaluation framework, and to refine the plan for the final evaluation. This will allow better referral mapping, detailed case studies and a more accurate and precise cost benefit analysis.

1.5.1. IFAS databaseThe interim evaluation plan was developed on the understanding that IFAS would implement a database that would include much more detailed and accessible information than was previously collected. This database was not implemented which has meant that some data are not provided or provided inconsistently. For example, some data relate to ‘families’ while other data relate to ‘clients’, with potentially multiple clients per family. Other data is not collected at all, such as if and when cases proceed to court. The evaluation team have worked to accommodate these challenges and will continue to work with IFAS to address them.

1.5.1.Measures of performance successThe Monitoring and Evaluation Matrix developed by the Incus Group informed the evaluation design. The evaluation questions were matched to the outputs and goals to ensure no aspect of the evaluation framework was missed. However, while the framework is suitable for developing a model of service provision it has no measurable targets, other than for reaching 150 clients. This means that this framework cannot be used to unilaterally evaluate service quality. It does provide an excellent framework for which meaningful targets can be developed however this process must be done in consultation with the IFAS team and VLA more broadly. In addition, the outputs listed were developed prior to implementing the service and require some adjustment to reflect the realities of service provision. The evaluation team will continue to work with IFAS to develop these measures of success and determine the extent to which they have been met in the final evaluation.

7 Virginia Braun et al, ‘Thematic Analysis’ in Pranee Liamputtong (ed), Handbook of Research Methods in Health Social Sciences (Springer Singapore, 2019) 843 <http://link.springer.com/10.1007/978-981-10-5251-4_103>.

15


1.5.2.Priority group referralsOnly one person who identified as having an intellectual disability, and one person who identified as Aboriginal, were referred by IFAS to the evaluation team. As these two characteristics make up the IFAS priority groups, having only one participant from each of these groups is insufficient in determining how successful IFAS has been in achieving its aims in relation to these groups. These priority groups will be a key focus of the next stage of the evaluation. Detailed strategies for engaging with these groups will be developed in the final stage evaluation plan.

1.5.1.DHHS dataAccess to DHHS data and permission to interview DHHS staff requires external research approval from DHHS. The evaluation team applied for this in November 2019. The evaluation team were informed by DHHS that this timeframe was not sufficient to have data provided by March 2020 and so would not be provided in time to be included in the midterm review. DHHS has since suspended all external research in response to COVID-19. This approval will be applied for prior to the final stage of the evaluation, assuming this restriction is lifted.

1.6. Ethics This evaluation is approved by the RMIT Human Research Ethics Committee (#HREC22471).

16


2. Preliminary evaluation findingsThe overall preliminary findings from the evaluation are very positive. People who used IFAS reported a universally positive experience, with even those who raised issues or concerns with their experience reporting it as an overall positive experience. IFAS is meeting or on track to meet program measures and is working with a range of families consistent with its aims and objectives. The preliminary costs benefit analysis findings are particularly promising, although they must be interpreted with caution. No major differences between the two pilot sites were identified.

This section is structured following the major themes which emerged from the data. Consistent with the overall evaluation approach, the findings section first considers the overall experience of people using IFAS, before turning to IFAS program measures. The context of child protection is briefly examined before the elements of this model are detailed, followed by the findings of the cost-benefit analysis. The final subsection is a discussion of specific findings which emerged from the data.

Quotes are labelled and numbered in parenthesis with service user (SU#), professional stakeholder (PS#), advocate (A#), IFAS Manager (IM), Lived Experience Advisor (LEA), lawyer (Law#) and case bank (CB). Where necessary, quotes have been edited for readability and to ensure confidentiality.

2.1. The IFAS experienceThe overall experience of people who had used IFAS was universally positive. Every participant who spoke to the evaluation team had an overall positive experience:

It’s been really good to have that support … to be honest, I don’t think I would have really got through this process as well as what I have without having that advice and support. (SU1)

And:

I don't know what I would've done without [IFAS Advocate], because it was really scary for me. I was so scared, like my whole situation was out of the blue and out of my hands. (SU8)

This positive experience was often related to the trust the advocates engendered. For people experiencing traumatic life events, such as family violence, and trying to navigate the child protection system, having an advocate was cherished:

Before, I was panicked. I was upset, crying. And when I first met [IFAS Advocate], I was relaxing, no worries, no more crying. They’re just – I just tell them everything. (SU6)

This theme of trust was related to the sense that there was nobody else who could be trusted:

By the time I got a hold of [IFAS Advocate] I guess I needed someone on my team to bat for me that I could trust because I felt like everyone was out for my family for not very good reasons. (SU8)

The advocates also noted this dynamic, highlighting their empathy and understanding of the experience of the child protection process:

17


When the parents feel the most vulnerable and the most alone because they’ve just had their kids removed, or about to have their kids removed, or at risk of having their kids removed. So they’re really alone. They don’t know who to turn to. You can’t turn to the school mother or the neighbour or anything like that because sometimes you don’t have anyone you feel like you can trust. …. So having an advocate to be able to support you makes a whole lot of difference. (A4)

When pressed to share negative experiences, people who used IFAS identified limitations of the service, such as there only being one advocate who was not backfilled while on leave or a preference for in-person advocacy rather than phone advocacy. These limitations are consistent with the service design and relate to the nature of the pilot and to resourcing constraints. Only one genuinely negative experience of the service was reported, where a person felt that their advocate was applying pressure to make a decision, as detailed below in para 2.4.1.

2.2. IFAS program measuresFor the implementation phase, IFAS has had only one measurable target, being a target of 150 clients per year in level 2-5 services. As IFAS moves into the next stage, VLA should identify clear targets for future assessment (Recommendation 1). These can be developed in the context of the existing Monitoring and Evaluation Framework (see Appendix 4) which is fundamentally sound but contains no actual measures of success. For example, VLA should determine meaningful targets for engaging people in the priority groups and what percentage of families diverted from court or reduction in future substantiations represents success. These performance measures should be mapped against data collection procedures (Recommendation 2).

In general, IFAS appears to be performing satisfactorily against key performance measures, although without detailed targets it is not possible to say that the service has performed as expected or otherwise. This section highlights IFAS’s performance in reference to the Monitoring and Evaluation Framework using the available data.

Level 1 services, open to anyone for information, including services and workers, have no files associated with them. Figure 1 shows that these services are being consistently provided. These services are not anticipated to have a significant impact on the sector but serve to fill a gap in information available about DHHS Child Protection processes and to promote IFAS more broadly.

Nov-18 Dec-18 Jan-19 Feb-19 Mar-19 Apr-19 May-19 Jun-19 Jul-19 Aug-19 Sep-19 Oct-19 Nov-19 Dec-19 Jan-20 Feb-2002468

101214161820

Leve

l 1 S

ervi

ces P

rovi

ded

Figure 1 - Level 1 services provided

For level 2 to 5 services, IFAS has not reached the target of 150 families per year, working with 99 families in total for a total of 106 episodes of service and 94 episodes of service in the 12 months March 2019 to February 2020. This is to be expected of a service which is both new to the sector and of a service type unfamiliar to workers in the sector. Despite this, IFAS intakes have risen to nearly nine per month in the six months to February 2020 and

18

based on current trends are on track to comfortably exceed 12.5 intakes per month to achieve 150 intakes in the next 12 month period. Ideally, all people eligible for IFAS would be automatically referred to IFAS by DHHS, eliminating the need to invest in service promotion and ensuring equity of access (Recommendation 5).

The 7% of families returning to IFAS suggests that some families are not being diverted away from the child protection system. The proportion of children across Victoria who were the subject of a decision not to substantiate by DHHS Child Protection and who were also the subject of a subsequent substantiation within 12 months is 16%,8 so a 7% return rate, if maintained over the next 12 months, is still better than would be expected.

Figure 2 – Increase/decrease in number of open files IFAS Levels 2-5

Figure 2 shows the number of files opened and closed over the life of the service. This pattern is consistent with periodic bursts of service promotion followed by a sharp increase in referrals. These data are consistent with the establishment phase of the pilot but do indicate a need to maintain focus on service promotion through referral networks to ensure the target of 150 is reached and IFAS is operating at capacity (Recommendation 3).

Incoming referral data are consistent with expectations but higher for some than for others. These data seem to indicate a need for more engagement with low referring stakeholders (Recommendation 3). There are a large number of referrals from IMHA, reflecting the close relationship between these services as well as the high

8 Report on Government Services 2020 (Productivity Commission, 23 January 2020) <https://www.pc.gov.au/research/ongoing/report-on-government-services/2020>.

19

Referral source % referrals

Self-referral 26%Family Violence Services 22%Child Protection - DHHS 11%Aboriginal service 10%VLA 10%IMHA 9%Hospital 7%Other source 3%VALID 2%

Table 1 - Incoming referrals

proportion of parents who are subject to compulsory mental health treatment who are also subject to DHHS Child Protection investigations. IMHA advocates have also promoted IFAS within mental health services. DHHS Child Protection referrals are fairly low, given DHHS is such a significant stakeholder, which would be avoided with an opt-out referral system (Recommendation 5). Currently, referrals from DHHS are limited by the requirement imposed by IFAS that DHHS get consent for IFAS to contact the person, and by the reluctance of some parents to engage with any services recommended by DHHS.

VLA referrals are also low, as this includes the VLA Legal Help referrals. The interviews and focus groups indicated very low levels of understanding of IFAS in VLA lawyers, as discussed in 2.4.8. The high proportion of referrals from family violence services does not have a clear origin, however, IFAS has been successful in supporting victims/survivors of family violence and has successfully promoted the service in this sector. This high proportion also reflects the significant overlap in people engaged with both family violence and child protection services. Self-referral data show a wide variety of sources where people are finding out about IFAS but is not recorded in a form that can be analysed (Recommendation 2.b).

Demographic data are unremarkable. Of people who used IFAS, 77% identified as female and 23% as male. No people of other genders used IFAS. Figure 3 shows that nearly all people using IFAS were aged between 20 and 50, although the large number of unknown age ranges suggests room for improvement in data collection processes (Recommendation 2).

IFAS has been successful in reaching its primary target groups of families with an Aboriginal child and families where a parent has an intellectual disability, as shown in Figure 4. IFAS is working with proportionately nearly twice as many Aboriginal people who are subjected to substantiated DHHS Child Protection investigations in the pilot sites, and proportionately six times more people with intellectual disability. This is impressive given the challenges in engaging with these groups.

IFAS has not worked with any people identifying as Torres Strait Islander.

20

Figure 3 - Ages range of people using IFAS

IFAS also has a wide coverage of the other IFAS priority groups; culturally and linguistically diverse families, parents with disabilities other than intellectual disabilities, mental health, victims/survivors and perpetrators of family violence, and families where alcohol and other drug use is present. Figure 5 shows that in particular, mental health issues are commonly seen. This is likely due to the close relationship IFAS has with IMHA, something which was also identified in the qualitative data with a number of referrals coming from IMHA. Victims/survivors of family violence were also commonly seen, correlating with the high proportion of referrals from family violence services.

Ultimately, an automatic opt-out referral system would ensure that IFAS was offered to all parents eligible for it, addressing all issues relating to service use and service reach (Recommendation 5).

2.3. ContextUnderstanding the efficacy of IFAS requires an understanding of the context in which IFAS is operating. The child protection context is contested and complex, however, the key contextual aspect which has led to the need for IFAS is the focus on the best interests of the child. This principle is enshrined in international law,9 Victorian law10

9 Tasmanian Department of Health and Human Services, ‘Children, Young Persons and Their Families Act 1997’ <https://www.dhhs.tas.gov.au/children/child_protection_services/children,_young_persons_and_their_families>.

21

Figure 4 - Priority group representation

Figure 5 - Other priority group representation

and in practice guidelines for DHHS Child Protection and related services.11 The requirement for services to focus on the best interests of children is an intentional departure from previous approaches which prioritised parental interests and rights.12 This may have contributed to the experience of many parents that neither their interests nor their rights are prioritised by services, particularly DHHS Child Protection. All participants who had used IFAS identified that they did not feel prioritised by and did not enjoy working with DHHS Child Protection:

I did feel like I was just sort of being pushed around in the dark because no one was really telling me anything. … it was scary and really disappointing. Like, I honestly, I truly believe that they were there to help families and sadly through the way the government’s structured at the moment, it's not the case. (SU8)

Past negative experiences of DHHS Child Protection engagement also contributed to dissatisfaction and resentment. This sentiment was reported to be particularly strong with Aboriginal families and parents with intellectual disability, an assessment which is also supported by the literature.13 Resourcing issues and high DHHS Child Protection caseloads were also commonly reported as hindering communication.

The broader child protection system is not within the scope of this evaluation and that parents do not enjoy being the subject of DHHS Child Protection investigation or legal orders is neither controversial nor unexpected. However, this problematic relationship is why IFAS is so highly valued by people who use it and why it might be effective.

A child protection lawyer explained that this poor relationship made it very difficult for parents to engage with DHHS Child Protection and for DHHS Child Protection to engage with parents, resulting in unnecessary court proceedings:

In terms of what the Department [DHHS] wanted a client to do, but I could see how they didn’t necessarily need to come to court to get that. But because the relationship between the parent and the Department was so bad there was no option other than to bring it to court. But that if there had been someone supporting the parent, there probably could've been some sort of midway points that the Department could satisfy themselves, and the parent would have a service or something that they felt supported them. (Law1)

The DHHS Child Protection practitioner is there to evaluate risk and protective factors, and ensure safety, wellbeing and development of the child. Depending on the approach taken by the practitioner and the response from the family, this can limit communication, which can impair progress and lead to a failure to resolve risk factors that may prevent escalation to court. The intention motivating IFAS is to address this communication barrier.

10 Children, Youth and Families Act 2005 (Vic) s 10.

11 Victorian Department of Health and Human Services, ‘Child Protection Manual’ (2020) <https://www.cpmanual.vic.gov.au/>.

12 Francis Zemljak and Chris Maylea, ‘Families, Children and Young People’ in Social Work and the Law: A Guide for Ethical Practice (Macmillan Education UK, 2019).

13 S Collings et al, ‘Intersectional Disadvantage: Exploring Differences Between Aboriginal and Non-Aboriginal Parents with Intellectual Disability in the New South Wales Child Protection System’ (2018) 12(2) Journal of Public Child Welfare 170; Sheila Gould and Karen Dodd, ‘“Normal People Can Have a Child but Disability Can’t”: The Experiences of Mothers with Mild Learning Disabilities Who Have Had Their Children Removed’ (2014) 42(1) British Journal of Learning Disabilities 25.

22

2.4. Elements of the IFAS ModelIn general, participants confirmed that IFAS was operating as intended and was generally consistent with the IFAS model detailed in the establishing documentation. No significant variation was found between the two pilot sites. Where variation from the model did occur, it was intentional, for example, to work with people not strictly eligible for the program while referrals were low, or to work more intensely with people who did not fit into the higher level service groups. Advocate 1 described IFAS as:

… a non-legal advocacy service for parents and carers in the early stages of Child Protection involvement. … There is a focus on parents of Aboriginal and Torres Strait Islander children and parents with an intellectual disability because these groups tend to be overrepresented in the child protection space.The aim of the service is to help parents and carers to understand the child protection system, understand what their rights are, what their responsibilities may be but be directed by them depending what their goals are. Help support with whatever is needed in their Child Protection process, that could be information, help with referrals, direct advocacy at meetings or perhaps talking, giving them tips around how to engage with Child Protection. (A1)

The model can be described as providing representational, rights-based advocacy and support with a focus on coaching and court diversion. IFAS also uses a relational model based on mediation and facilitation and strong professional collaboration. Each of these elements is described in this section.

2.4.1. Representational advocacyRepresentational advocacy can be seen, particularly in the child protection context, as an alternative to a best-interests approach. As detailed in IFAS policy documentation, a representational approach does not attempt to apply the advocate’s values or give an objective assessment of the person’s situation. Instead, it aims to amplify the voice of the person. The advocate is responsible for outlining options and consequences, but not making any actual decisions. Generally, an advocate cannot act without the explicit consent of the person. IFAS advocates aligned this practice with the instructions-based approach used by lawyers. This approach was demonstrated by all the advocates, although each admitted it was an ongoing process to shift their practice from a best-interests approach to a representational model.

People who had used IFAS highly valued this non-judgemental approach:

They were my other mouthpiece, like if they spoke to [DHHS] on my behalf to find out what was going on, and why they were involved and what needed to be done. (SU3)

There were some limits on the representational model. For example, despite not being mandatory reporters under the Victorian Children, Youth and Families Act 2005, IFAS will make reports to DHHS Child Protection without the person’s consent if they believe a child is at risk of significant harm. IFAS would notify the parent, but not require their consent to make a report. Despite these defined limits, IFAS is working consistently with a representational approach.

The representational model was not universally well understood by stakeholders, who sometimes identified representation as ‘colluding’ with the parent. For example, Stakeholder 6 was concerned that the advocate was not seeking other worker’s perspectives on the family, instead accepting the family’s perspective.

23

You have to also have a broader view on the family because you can be easily caught up in that story. And I’m all about like I’ve, I mean, I’ve worked with vulnerable communities all my life and I’m an Indigenous person, I definitely hear their story and their views, but you also have to have very clear boundaries. … talk to us because we’ve got a long history of some of those families. … Don’t collude. (SU6)

This highlights the importance of clearly communicating the model to services, particularly those who adopt a best-interests approach (Recommendation 7). IFAS staff have conducted numerous information sessions and distributed service factsheets, however, the experience of IMHA indicates that this will be an ongoing process. The IMHA experience shows that workers who have been trained in and work in a best interests approach may find it difficult to conceptualise the representational model.14

The evaluation only found one example of a significant deviation from the representational approach. In this example, the advocate had suggested that if the parent did a certain process, DHHS would end contact:

… it eventually felt like one group of them just squeezing us … [IFAS] was actually either in cahoots with them or being pressurised by them to tell me to do it. You know, and [IFAS’s] advice was that if I do it, I’ll get rid of DHHS. Which wasn’t the case at all. We’ve still got DHHS. (SU4)

Following closer investigation, it became clear that the ‘pressure’ applied was not particularly overbearing from the advocate:

Pressure was a phone call saying, “Look, I’m pretty sure if you do this you’ll get rid of them.” And a week later we got served. (SU4)

This example shows the difficulty in providing advice while ensuring that the person still feels in control of the process. Advocates providing information on how best to satisfy the expectations of DHHS risk inadvertently reinforced these expectations. Though this may lead to beneficial outcomes for the child or the family overall it is a deviation from the representational advocacy model and should not be a feature of IFAS (Recommendation 10).

2.4.2.Rights-based advocacyIn contrast with IMHA, on which the IFAS advocacy model is based, rights were much less of a focus in the daily work of IFAS advocates. Certainly, IFAS advocates worked to uphold the rights of people they worked with, but compared to compulsory mental health treatment, the rights afforded to parents in the pre-court child protection system are much less clearly defined and much more difficult to enforce. Often, advocates were working just to ensure DHHS Child Protection workers complied with DHHS policies, particularly on timeframes and compliance mechanisms, rather than any statutory or higher-level human rights. Advocate 2 reinforced this difference in context:

Underneath the child protection legislation, [parents] have none to very few rights in that [voluntary] space there. There’s really no information we can give them. What they might say is ... “When am I going to get the case plan? That's meant to be done within 28 days”. Or, “You’re only supposed to be investigating voluntarily for 90 days. Why are you still open?” … that's just questioning the Department’s own policy. Their rights in that space are very few and far between. (A2)

14 Maylea et al (n 3).

24

Despite these limitations, people who had used IFAS consistently identified that IFAS had helped them understand their rights better:

It's just the knowledge, letting people know actually that they do have rights. Because when you're in that situation you just think that oh, you do what the DHHS worker says. (SU10)

This was supported by a stakeholder from an Aboriginal Community Controlled Organisation (ACCO) who noted that parents of Aboriginal and Torres Strait Islander children and parents with intellectual disability enter the child protection system with different perspectives than other parents might:

They go into situations where they're being dictated to by a government department, they're being told what’s going to happen next and they take that. They come from a demographic or a lifestyle, I guess, where they don’t understand that they can challenge those things and they have rights to make things different. So that’s really good that [IFAS] can be there to say, no, you don’t have to do this, this doesn’t have to happen that way. You have a right to Family Law, you have a right to protest this stuff and to change it if you need. (PS17)

Advocates did refer to other rights, however, advocates’ understanding of the range of different legal and non-legal rights was somewhat limited to best practice in child protection, rather than a true rights-based orientation (Recommendation 8).

2.4.3.Coaching for self-advocacyIFAS also provides coaching for self-advocacy. This was often related to providing information about the child protection system and process:

Oh, definitely. Definitely. Before I'd got in contact with her I didn't know nothing, basically, and, yeah, she gave me more confidence just to actually deal with them by myself, and to give her a call at any time if I had any questions. (SU10)

Other times, the self-advocacy was less successful. This was explained as a result of the power imbalance between DHHS Child Protection workers and people under investigation. People who had used IFAS told the evaluation team that they received a completely different response from DHHS Child Protection to the response the advocate would receive:

When the Child Protection talked me a lot of thing, I thought there’s something wrong here, it makes no sense, got nothing to do with me. And when I told [IFAS] everything, and some paper where I told them that, and they ring up Child Protection to find out what was going on and everything. And then after that when we came in the meeting, and I find different story. Absolute different story. (SU6)

This was often reported as related by participants to very basic aspects of respect and dignity:

She was a lot different when [IFAS Advocate] was present, she was really nice and kind and friendly and like completely professional. Like how she should be doing her job, respectful the way she spoke to me, all that. (SU1)

And:

25

The way [DHHS] spoke to me and all. Well, they spoke to me like a level-headed adult when [IFAS] around, but when [IFAS] wasn’t I was just another young child that didn't deserve to have another child. (SU11)

This highlights the value of an independent professional advocate in the voluntary pre-court context, where there are few independent oversight mechanisms to balance the power dynamic between families and DHHS Child Protection workers. This does not diminish the value of coaching for self-advocacy but highlights the importance of having direct advocacy available as well.

IFAS has very limited self-advocacy resources that can be provided to parents, particularly when compared to IMHA. People who had used IFAS consistently noted that they did not have sufficient information about the system or their rights to be able to advocate for themselves. The development of self-advocacy resources is reported to be progressing as required by the evaluation and monitoring framework but has not yet been completed (Recommendation 11).

2.4.4.Systemic advocacyThe evaluation found little evidence that IFAS was having a systemic impact, which is not unexpected at this point in the service implementation. There are ad hoc attempts at influencing systemic change, such as feeding back issues to DHHS Child Protection senior management and presenting at conferences, but these have not yet resulted in substantial systemic change.

Some participants expressed a hope that IFAS involvement was contributing to oversight and accountability mechanisms, however, were not confident that there had been any actual change at a system level. There is some evidence in the data of attempts at systemic advocacy, but even the IFAS team acknowledged that systemic change in the child protection context will require a significant investment. At best, IFAS was viewed as having the potential to change individual worker attitudes and to contribute to a larger reform agenda. The IFAS Manager acknowledged this:

I’d say it’s too early to say that they’re having any kind of systemic impact on practice. I think that it’s probably had some impact on some of the staff that have had contact with IFAS advocates, the Child Protection workers potentially, in terms of seeing value in parents and primary caregivers having a voice and understanding the system. (IM)

Any systemic change in this context will require a concerted effort. The experience of IMHA suggests that even with major investment in reform in the form of a Royal Commission into Victoria’s Mental Health services, systemic reform will be an ongoing process. To assist this, IFAS should develop a systemic change strategy in partnership with other services, with measurable outputs and defined outcomes (Recommendation 12).

2.4.5.Support beyond advocacyIFAS is, by definition, independent family advocacy and support. This aspect of support beyond advocacy is less well defined than the advocacy aspects of the service. Support appears to include referrals to other services, providing emotional support and as an oversight function. These are common functions of advocates in various roles and are consistent with the IFAS model.

Support occasionally extended to quasi-therapeutic support more generally associated with case management than advocacy. When pressed for examples of success, advocates often cited situations where they had creatively supported parents to comply with DHHS expectations or requirements on their own terms. Examples included

26

writing letters of support for housing applications, working with family violence services to develop safety plans, or where IFAS advocates provided a secondary consulting style service to NGO workers. This role was recognised by the advocates:

We might find ourselves, at times, being caseworkers, doing caseworker tasks that Child Protection can’t do. But that's, (a) for the benefit of our client because we’re the defined advocate, and (b) it’s an investment in the case. It’s just going to push in the right direction. (A3)

This extra support was highly valued by people who used IFAS and should be incorporated into the IFAS model. However, IFAS documentation and practice should ensure that this is done consistently and in such a way to ensure that IFAS is not providing a service which other organisations are funded to provide (Recommendation 9).

2.4.6.Diverting from courtThe evaluation found clear evidence that IFAS is diverting families away from the court system. As discussed in Appendix 5, the quantitative data is promising, but not of sufficient quality to determine the extent to which IFAS is successful in this aspect. The qualitative data does highlight a number of instances where this is occurring.

The IFAS Case Bank identifies eight families of a total 23 where the IFAS advocate writing the case example identified IFAS playing a role in diverting a family away from the court system. As it is written by IFAS advocates, Case Bank is illustrative but not an unbiased representation of the service. The IFAS advocates also provided the evaluation team with examples of where their practice had resulted in court diversion. They are illustrative in understanding the mechanism of how IFAS is aiding in court diversion, if not in quantifying prevalence. For example, IFAS Advocate 2 outlined how advocacy assisted in helping a mother in a family violence situation:

This was a family violence situation, and although the particular parent we were working with was acting protectively, Child Protection were hanging around. In this particular situation the other parent was very manipulative and very controlling. With the information that IFAS was able to provide that mum, I think she went from being very nervous and very scared and presenting in a way that was raising a lot of concerns for the Department, to now three months later, able to talk to me about everything that’s going on, and everything that the ex is trying to do, really calmly. … I think they probably would have [applied for an order to remove the child]. [Instead], mum went through with the intervention order, she did all of those things, so Child Protection were able to close because mum actually was able to do all the things that they were recommending that she might not necessarily have wanted to. (A2)

Pressed for another example, IFAS Advocate 2 explained how mobilising supports such as family violence orders could lead to court diversion:

There’s an example of Child Protection saying to a mum, you need to get an intervention order on your partner otherwise we’re not going to be able to close and we’ll need to go to court. And we were able to support that mum who had a really good family violence safety plan that she developed herself to get an assessment by a family violence specialist service about that plan, and then take that back to the Department and actually say, well the family violence specialist service says that mum’s plan is absolutely fantastic and it’s really good. So, we get [DHHS] to close when they were actually saying they weren’t going to. (A2)

IFAS Advocate 1 was less clear, noting the difficulty in determining the impact of IFAS in a very complex system:

27

I know that it’s helped close some cases. Whether they were going to go to court anyway with or without IFAS I don’t know. (A1)

Advocate 1 did give a concrete example of having supported a family to avoid out-of-home-care through assisting the family to collect specific evidence, although not actually preventing a court hearing in this case:

The family got to court the next day with that evidence, and the magistrate said, “No, they can go back into the home. The unacceptable risk that the Department needs to meet for dad not being there has not been met by the Department”.

The evaluation team also consulted with service users to determine if IFAS prevented them from continuing to court. Some, such as Service User 3, did not attribute court diversion to IFAS:

Interviewer: Do you think you avoided going to court because of IFAS?Service User 3: No, [DHHS] just didn’t need to because there was no concerns around the

children.

Others, such as Service User 11, also noted the complexity of the process, but felt IFAS were instrumental in the process:

Service User 11: It's hard to say. [DHHS] were wanting to take me to court, but at the same time they wanted to resolve it without taking me to court.

Interviewer: So you're not sure if it would've gone to court if [IFAS] hadn't been involved, but having [IFAS] involved meant that you didn't have to worry about it so much?

Service User 11: Yeah. Well, I can guarantee and safely say that since [IFAS Advocate] was involved that I did not go to court because of [IFAS Advocate] being involved.

Service User 10, however, was very clear that IFAS had prevented the case proceeding to court:

Interviewer: Were [DHHS] looking at going to court too, or did your case go to court at all?Service User 10: No. [DHHS] did say that they were going to apply for an order, a protection

order, but that never came about.Interviewer: Do you feel like that's as a result of [IFAS] being there to advocate for you?Service User 10: Definitely. I definitely think that if I didn't have [IFAS] around when they were

involved, it definitely was happening.

To ascertain an independent view, the evaluation team consulted stakeholders, and while they often saw the potential for court diversion, they could not always conclusively saw that IFAS had made the difference:

Now I can’t specifically say that's because of [IFAS]. …. Big assumptions here, but yeah, look, they did withdraw despite, yeah, heavily threatening for a long time that they were going to become legally involved. (PS4)

Despite the complexity, others could see a clear link in specific cases between IFAS and court diversion:

Facilitator: Is it likely that that person would have ended up in the court system, do you think otherwise without IFAS Advocate 3’s intervention?

Stakeholder 17: Yeah, definitely. … I think without their intervention I think they'd be definitely back within the system of child protection and Family Law courts, definitely.

28

This same stakeholder, a manager at an Aboriginal Controlled Organisation, explained why, from their perspective, IFAS was effective in preventing cases from proceeding to court and or in reducing contact with DHHS Child Protection:

There would be a lot of clients where I think it would have gone to court straightaway regarding an order, safety around kids and whatever. I think, for a lot of our community it’s probably a couple of different things is the white privilege or the white lens society looks through in identifying risk. So I think IFAS Advocate 3 breaks those barriers down and advocates for the family of when we start looking at generational trauma and those type of things. I think it may end up still that the kids are in the system but not as extensive, not on permanent care orders, not on long terms orders. … I think [IFAS has] definitely stopped the flow of Child Protection orders on families. (PS17)

A key aspect of the IFAS model is diverting parents from the court system. This is certainly occurring, however it is not yet possible to precisely determine to what extent. This is discussed in more detail below in relation to the cost-benefit analysis in Appendix 5. The qualitative data indicate that much of the success IFAS has had in diverting people from the court system is a result of playing mediation and facilitation roles.

2.4.7.Mediation and facilitationAs discussed above, the DHHS Child Protection investigation context is commonly adversarial. Many of the people who had used IFAS who participated in the evaluation had negative past experiences with DHHS Child Protection. While some participants noted that they had also had positive experiences, this negative relationship often led to a stalemate, where communication broke down and DHHS Child Protection workers resorted to applying for court orders to address their concerns. IFAS was described as being able to intercept this adversarial relationship and defuse the situation. For example, Service User 1 told the evaluation team how much they appreciated the positive relationship the advocate had with DHHS:

[IFAS Advocate] actually goes out to the [DHHS] office because she’s obviously got more than me as a client. And she actually liaises with the staff at the office in quite a friendly way. Like, checking on the progress of her clients, working as a team with DHHS. That’s what impressed me. Like, it’s not like … in a lot of legal situations it’s like us against them. But [IFAS Advocate] doesn’t operate like that; [IFAS Advocate] tries to bring everyone together and have everyone working on the same page. (SU1)

For Service User 1, the positive relationship was able to be translated to the DHHS worker, leading to an ongoing collaborative relationship with DHHS:

[IFAS Advocate] has acted like as a mediator pretty much between me and the Department, and not in a way where it’s like us against them or anything like that. It’s been more of encouraging positive communication and actually [IFAS Advocate has] been really good at helping me build a good rapport with my worker from DHHS. Things have progressed really well with the Department. (SU1)

This role as a mediator did not mean that the IFAS worker took the side of DHHS, rather that the positive relationship could be used to enable difficult conversations. Often it meant that people could ask questions of their DHHS workers without becoming frustrated or losing their temper. This excerpt from the IFAS case bank highlights how advocates would fulfil this function:

29

IFAS functioned as a conduit between DHHS and client. IFAS practitioner was able to get a functional email relationship with DHHS, gaining a clear sense of protective concerns, also timelines of what DHHS conditions were to be imposed on any family reunification. Sometimes Phyllis15 started to panic and get angry because she had not heard anything back from DHHS within a short timeframe. IFAS practitioner was able to validate her concerns, settle Phyllis, and assist Phyllis to focus on what was next, and what the future might look like soon. The IFAS advocate was always encouraging of Phyllis - which she stated made a big difference to her. (CB)

This was confirmed by Service User 4:

Interviewer: One of the things that the IFAS advocates are telling us that they do is defuse the situation between the families and DHHS.

Service User 4: Yes, that happened a lot. Interviewer: And then he can filter out that and take it back to DHHS. Service User 4: Yes, that happened a lot. Interviewer: If you blew that steam straight at DHHS, that – Service User 4: It would’ve been worse.

Professional stakeholder 19, an Aboriginal community member and family worker from an ACCO, also remarked on this phenomenon:

Currently, we have a case where the parents can become quite heightened and verbally abusive, they hate Child Protection. And they quite often use [IFAS Advocate] as that advocacy person because if they were to contact Child Protection or Child Protection contact them they become very verbally abusive and heightened. So I think it’s stopped that wedge of them becoming heightened but getting their message across which is really amazing, especially for that family because Child Protection, they do not get along. (PS19)

The evaluation team consistently inquired about the extent to which IFAS was only required as a result of limited DHHS Child Protection resources. Resoundingly, this ability to negotiate between DHHS Child Protection and distressed parents was given as the reason that this was not the case, particularly with Aboriginal families who had experienced generational trauma from the Stolen Generations. Stakeholder 19, reinforced this:

Interviewer: If Child Protection had the resources and the time and the high-quality recruits and capacity and all these things, could they engage with that family?

Stakeholder 19: No, that barrier is completely closed. I think it comes back to previous times that they have been involved and it’s not great.

Other times, the communication breakdown was not related to past conflict and was solved through simple translation. This was particularly in relation to people who required support to understand information or make decisions. Service User 6, who lives with intellectual disability, shared their experience:

Sometimes they talk too fast, and sometimes they were angry with phones. First thing I said, and I said, “OK, OK”. Then they hang up on me. And then I ring up [IFAS] because I don’t know what they’re saying. It’s all the big words. That’s the problem. And then I punch up [IFAS], and told them what’s it all about, and that’s where they told me “I’ll ring you back, and I’ll talk to them”. [IFAS] didn’t talk, they just listened to me, everything I said, and then they said, “I’ll ring you back, I’ll talk

15 A pseudonym.

30

to [DHHS] first, and I’ll ring you back.” So that’s it. I felt really, really good. I wait 30 minutes, and [IFAS advocate] rang me back. Wow. I felt really, really happy. (SU6)

While DHHS staff were not able to be interviewed in this stage of the evaluation, the evaluation team were provided with deidentified emails from DHHS Child Protection workers and other stakeholders thanking IFAS staff for the work they had done advocating for specific people. This role, of mediation and facilitation, is supported by the IFAS approach to professional collaboration and supported by colocation.

2.4.8.Professional collaboration and colocationThe strength of IFAS in mediation and facilitation is the advocate’s ability to collaborate with other professionals, which has been greatly enhanced where they have collocated. As discussed below, advocates’ professional backgrounds in child protection roles enabled them to communicate effectively with DHHS Child Protection workers and other related professionals. Advocate 1 explained how effective collaborative systems enhanced referrals:

We’ve got a system with [DHHS Office] where we meet to talk about possible deidentified cases for IFAS. And then it’s a matter of, once the team managers have said that these ones would [be eligible] they tell their workers, and their workers go and get consent. So I'm keeping a tally of what stages each case is at, and every Friday following up; “So, did you get consent?”. And it’s almost like following a sales process [laughs]. So have we consent yet, and that kind of thing. But yeah, it’s been really hard. It’s been tough. (A1)

This professional collaboration and colocation has not had universal success and will require ongoing effort to embed IFAS in the service system (Recommendation 6). In particular, a number of stakeholders did not understand the representational model or that IFAS was not solely focused on the best interests of the child. This was recognised by the IFAS advocates:

I think it’s difficult for other services to understand that we don’t work in that best interest model. … I think those workers that have worked with us once or twice now get it a bit. But I think it’s going to be an ongoing process; the first time we work with [a] Child Protection worker that hasn’t worked with us. (A2)

As mentioned above, representational advocacy was occasionally perceived as ‘colluding’ with the family. Based on the IMHA experience, this will be an ongoing issue and will require ongoing service promotion by the IFAS team (Recommendation 3).

For other stakeholders, the lack of understanding was not related to the fundamental principles of practice, but just a lack of knowledge of the program. For example, two VLA child protection lawyers understood the general purpose of the program, but were not clear on the geographic limitations or the IFAS eligibility criteria:

Do they work with us? In theory, yes, they are embedded within VLA, but I think it’s not fully understood still what everybody does, and that’s a little bit embarrassing to admit and to say. (VLA 4)

VLA Lawyer 2 noted that there was probably information on the internal learning hub that they could have referred to, but had not had time to do this. Other VLA lawyers had a high level of understanding of IFAS, however there is clearly an opportunity to improve this relationship (Recommendation 4b).

31

2.5. Other Specific FindingsThis section presents other miscellaneous findings that emerged from the data during analysis. These are the cost benefits of IFAS, the ways co-design and lived experience elements are supported, how access to legal services are ensured, the risk that advocates are assisting with compliance with DHHS Child Protection expectations, the role of IFAS in advocating outside DHHS Child Protection, and the impact of IFAS on children.

2.5.1.Cost benefitsA detailed cost-benefit analysis is included in Appendix 5, but as data is currently limited and IFAS is still in the implementation phases, the preliminary findings from this analysis are summarised here.

The evaluation team assessed the costs savings of each child diverted from court by IFAS to be $66,616 to the Victorian Government, including DHHS court costs, intensive family support services, Children’s Court costs, legal aid grants and out-of-home care. To return a positive investment, IFAS would need to divert 4% of families away from the court system. To result in a positive return on investment for Victoria Legal Aid only, which has funded the pilot, IFAS would need to achieve upwards of a 28% success rate in court diversion.

Quantitative data is promising but inconclusive, allowing for a possible court diversion success rate of between 5% and 25%. An analysis of the qualitative data identified five distinct examples in a 12-month period where participants told us that IFAS had prevented the case from going to court and one example where the case did go to court but due to IFAS involvement, the court found in favour of the parent. This is a roughly 5% success rate and is estimated to have saved the Victorian Government $400,486 in a 12-month period, comfortably exceeding the cost of IFAS at $344,091.

The final evaluation will incorporate an additional 12 months of data and include DHHS data, allowing for a cost-benefit analysis with fewer caveats and limitations.

2.5.2. Co-design and lived experienceIFAS is based on the IMHA model of advocacy and like IMHA, includes a Lived Experience Advisor on the IFAS team. Unlike IMHA, IFAS advocates do not include lived experience advocates. This difference in implementation was interrogated by the evaluation team, with evaluation participants unanimously supporting advocates with professional child protection backgrounds over advocates with lived experience backgrounds. This included the Lived Experience Advisor. Occasionally it was noted that child protection professionals who also had personal experience of the child protection system would be ideal, as a hybrid approach, but the professional experience was always prioritised. This was for two reasons; the expertise of having worked within the child protection system was essential to providing useful information, and the power dynamics between professionals promoted effective advocacy, a privilege not extended to lived experience advocates. Unlike the mental health sector, there is no established workforce of lived experience child protection advocates to draw on, although such models do exist in other jurisdictions. When the jobs were advertised, both lived experience and professional experience were included as selection criteria, however applicants with professional backgrounds had more system knowledge, viewed as essential for the role:

Those people did really well in interviews because they could talk about the fact that they’ve worked in child protection, they understood the policies and procedures. The way they work, the way the system operates. (IM)

32

Without this lived experience in the actual advocacy teams, IFAS must work harder to embed this principle of lived experience. The Lived Experience Advisor is essential to this process but appears to have more influence over practice in Melbourne than Bendigo (Recommendation 14). The IFAS team values and includes the lived experience perspective in their practice and there are numerous examples of strong leadership from the Lived Experience Advisor. There is also clear evidence of SEAS having real influence over the way that IFAS is run and of positive team dynamics, as described by the Lived Experience Advisor:

It’s amazing, yeah. And I think we do a lot of bouncing off each other really well as well. … A client has got this issue or something’s happened and I don’t know how I’m going to really go about doing it. And it’s about sitting there and having discussions with her about what about this as a solution or this as a solution to think outside the box. (LEA)

However, when compared with the IMHA reference group, Speaking From Experience (SFE), SEAS is still operating at the early stages of development. The numbers of SEAS participants are lower than is ideal and support for individual SEAS members may need to be increased to ensure full and meaningful participation for all. More support is required to ensure that SEAS is operating at full potential and that it can continue to exercise meaningful influence over IFAS (Recommendation 13).

2.5.3.Access to legal servicesA key goal of IFAS is to increase access to legal services for Aboriginal parents, parents with Aboriginal children and parents with intellectual disability. There is clear evidence of this occurring but prevalence cannot be determined with the available data. Outgoing referral data are required to determine this (Recommendation 2.b). Some qualitative evidence was provided that suggests IFAS is linking people to legal services, such as this parent:

And [advocate] also helped find me a good solicitor through Legal Aid and all of that, and [advocate] liaised with the solicitor to give some background on my situation and provide positive comments to the solicitor to actually help me in court. (SU10)

The IFAS advocates saw providing this link as an important part of their role and a measure of success:

There’s been situations where IFAS clients have needed legal advice right away. And so, we’ve been able to get legal advice from a child protection duty lawyer for VLA right then and right there for them, on the phone within half an hour. So, that’s actually worked really well in that space from what I can see. (A2)

There are opportunities to improve relationships with VLA and private lawyers to increase referrals in both directions, particularly in ensuring that lawyers understand IFAS’s role and limitations (Recommendation 4.b). This will be revisited in the final stage of the evaluation.

The IFAS advocates had a clear sense of what did and did not constitute legal advice. This was confirmed by VLA lawyers.

2.5.4.Assisting with complianceAs noted above, two key elements of IFAS’s success are the representational model and the focus on facilitation. These elements risk IFAS existing in tension, with potential for advocates to unconsciously ‘steer’ people towards compliance with DHHS Child Protection and in doing so become an extension of DHHS Child Protection. Although this was only reported by one parent, IFAS risks being the ‘good cop’ to DHHS Child Protection’s ‘bad cop’. The

33

IFAS advocates were both aware of this risk and confident in their ability to navigate it, and there is strong leadership from the IFAS Manager to maintain consistency with the model. This is recognised by the IFAS Manager:

I think that is the risk for them. We have to constantly be thinking about in terms of, “Are we helping people comply?” (IM)

The IFAS Manager also noted that the outcome of the person’s situation would contribute to whether the interaction would be experienced as enforcing compliance:

If they’re the ones that have ended up out of the system, they probably don’t see it as compliance. They see it as actually that worked and that’s what I needed to do. But if they’re still in the system because they did do it and they complied, that might be a very different experience for them.

This seemed to be the case for one service user, who communicated that he was happy to have been assisted to comply with DHHS:

… that will help your situation, that will help you with DHHS in them being pleased that you’re complying with this, and that you’re doing that, and that you’re communicating in this way. (SU1)

Navigating this tension will require continued leadership and ongoing attention (Recommendation 10), particularly if IFAS is expanded following the pilot phase.

2.5.5.Advocacy targetsIFAS is primarily designed to advocate ‘to’ DHHS Child Protection. The potential for IFAS to also focus on other advocacy targets is obvious but poorly defined in the IFAS model. For example, as changes to s 18 of the Act have led to some ACCOs becoming statutory decision-makers, workers of these organisations have become a subset of child protection workers. These workers did not see themselves as valid targets of IFAS advocacy:

We should be having the relationships with the families to where we are the advocate for them, and to where we don’t really need an external person. (PS8)

A further issue is the optics of a mainstream organisation (VLA) advocating at Aboriginal workers in Aboriginal organisations, which was not viewed as ideal.

Similarly, workers who were not in statutory decision-making roles but were still in child protection settings did not see themselves as needed to be advocated ‘at’: to?

Stakeholder 6: Why would I? Why would I refer a family I’m working with to a service that’s going to like advocate to me to like buy stuff and do things. I’ve already got the family doing that. …

Interviewer: Do you think there is any value in having an advocacy service advocate at you?Stakeholder 6: No, I don’t. I think there’s a usefulness in having a service to advocate on behalf of

them for Child Protection, absolutely, because we have so many families with so many different needs. So, there’s just another need. If we can have someone totally dedicated to that but working with us so we’ll be like, yeah, can you like push this issue? We’re getting nowhere. Can you please just work on this, that would take a lot of pressure off us.

34

This phenomenon was also seen in the IMHA evaluation,16 and particularly amongst social workers.17 This highlights two issues; the need for IFAS to clarify the extent to which it will advocate outside of DHHS Child Protection (Recommendation 9) and the need to ensure that other workers understand the IFAS model, particularly around representational advocacy (Recommendation 3).

2.5.6.ChildrenIFAS does not work with children directly, other than when they are themselves parents who are eligible for IFAS. IFAS does not work from the principle of the best interests of the child. The IFAS Manager explained this was one of the reasons IFAS was developed to begin with:

It wouldn’t be appropriate for us to engage with children and young people. I think the whole reason why IFAS took the model it did and chose the group it chose was because Child Protection is supposed to be there looking after the child’s best interests. (IM)

Despite this, every participant who was asked indicated that IFAS had a positive impact on the children of families they worked with. The evaluation team heard some examples of IFAS being accommodating of children. In one example, Service User 10 explained that the IFAS advocate had helped their son understand his rights.

The primary benefit to children was vicarious, through the improved experience of the parent:

They didn’t get to meet [IFAS advocate], but they knew about her and they knew that she was working for us. So I suppose that that made them more comfortable, knowing that there was more support there. (SU3)

Two stakeholders from an ACCO (PS17 & 19) suggested that IFAS was having a positive impact through role modelling:

[IFAS Advocate] is modelling what the parents can do and help them to do that which is then showing the children that well, if mum and dad can do it we can do it too. So it’s that role model effect I think. (PS19)

Finally, participants indicated that if IFAS was successful in diverting families away from the child protection system, this would mean children were staying with their families, which was viewed as an inherently good thing.

3.ConclusionIn conclusion, preliminary findings show that IFAS has been successful in the first stages of implementation. People who had used IFAS reported overwhelmingly positive experiences and stakeholders agreed in the potential for IFAS to make a positive impact on the child protection system more broadly. The preliminary cost-benefit analysis indicates that IFAS has a positive return on investment.

There are several areas of opportunity for improvement, particularly in stakeholder understanding of the service, data collection and monitoring, and the emphasis of lived experience aspects of the model. There is little evidence

16 Maylea et al (n 3).

17 Christopher Maylea et al, ‘Independent Mental Health Advocacy: A Model of Social Work Advocacy?’ [2020] Australian Social Work 1 (‘Independent Mental Health Advocacy’).

35

of systemic change, and it is likely that systemic change will not be achieved with IFAS alone, requiring instead a concerted reform agenda.

These findings will be re-examined in the final stage of the evaluation.

36

4.Interim recommendationsThese interim recommendations are based on the findings of the midterm review. They represent the professional opinions of the evaluation team, based on rigorous analysis of evaluation data against the key evaluation questions and program logic model. These recommendations are intended to support the implementation of IFAS and enable a successful final evaluation.

IFAS is already making progress towards some of these interim recommendations, and IFAS’s success in implementing these will be evaluated in the final stage of the evaluation. The final evaluation will include a set of final recommendations for ongoing service development with reference to the revised Evaluation and Monitoring Framework.

Enabling a successful final evaluation

1. Develop robust measures of performance success based on the existing monitoring and evaluation framework;

2. Develop an appropriate data management system. This must include:a. Both primary parent and secondary parent if involved;b. Referral data, incoming and outgoing, source of self-referrals, word of mouth referrals;c. Child/ren’s names and birthdays to enable data matching with DHHS;d. Gender recording must include non-binary;e. Demographic data should have multiple options including:

i. Yes, not confirmed, confirmed no;ii. Alcohol, drug use, ice;

iii. Historical/ongoing;f. Processes to ensure the security and consistency of the data.

Recommendations 1 and 2 will be progressed in partnership with the evaluation team.

Building on a successful pilot

3. Continue to deliver the IFAS pilot;4. Continue to promote the service and ensure stakeholder understanding of the IFAS model. Particularly to:

a. DHHS Child Protectionb. VLA and private child protection lawyers;

5. Advocate to DHHS to implement automatic referral system for all people eligible to be referred to IFAS;6. Maintain existing colocation and explore additional colocation opportunities;7. Continue to promote the service, including:

a. Assisting stakeholders to understand the nature of the representational model;

Enhancing and embedding the IFAS model

8. Advocates should be provided with training on the variety of legal and non-legal rights available and how these can be maintained in the child protection context. These should include rights in the Children Youth and Families Act 2005 (Vic), Charter of Human Rights and Responsibilities Act 2006 (Vic), privacy legislation and international human rights law;

9. Review IFAS’s role in providing support beyond advocacy;10. Continue to reinforce the representational model;11. Develop self-advocacy resources to be provided to parents;

37

12. Develop a systemic change strategy in partnership with other services, with measurable outputs and defined outcomes;

Maximising the role of lived experience

13. Continue to support the development of SEAS;14. Develop processes to ensure that the Lived Experience Advisor has equal influence in both regions;

38

5.Appendices

Appendix 1. IFAS Logic Model

39

Appendix 2. List of IFAS documents reviewed Child Protection Legal Aid Services Review; Child Protection Early Intervention Unit Project Plan; IFAS Database – Template; IFAS Logic and Monitoring Framework; IFAS Policies and Procedures, including;

o Service Eligibility;o Service Scope;o Service Levels;o Advocacy Model;o Referrals and Intake;o Working with Clients;o Outreach and Access to Services;o Feedback and Complaints;o Access to Records;o Exiting the Service;o Incorporating Lived Experience into Service Design;o Professional Standards;o Communication;o Working with Diversity;o Engaging with Third Parties;o Managing Risk;o Monitoring and Evaluation;o Support and Administration;o Team Structure;

IMHA IFAS Fact sheet - Sharing client stories.

40

Appendix 3. Question Matrix

.

41

Appendix 4. IFAS Monitoring and Evaluation MatrixItem Indicators/Metrics of success Data collection

method/sourceData collection timing

Responsibility Assumptions / comments

IFAS contributes to the CPT vision by providing a better experience for clients through improving access to effective prevention and early intervention for families involved in the CP system

Activity 1: Intake

Output 1.1:

Identification of client eligibility

Number of ATSI clients Number of clients with intellectual

disabilities Number of ineligible clients (and why) Proportion of types of ineligibility Recording of origin of referral

pathway (eligible and ineligible) Number of clients by Level Number of clients by demographics

(priority criteria, age, location)

IFAS Database Ongoing IFAS staff Client Data can also be disaggregated and recorded by Levels (1-5)

Output 1.2

Provision of appropriate information and resources for ineligible clients

Number and type of informational materials provided

IFAS Database Ongoing IFAS staff

Output 1.3

Advocacy needs assessment for eligible clients

Number of interactions with client Documentation of client objectives

and goals Number of clients requesting

advocacy only, self-advocacy only, or combined

IFAS Database Case Notes

Ongoing IFAS staff

Activity 2: Provision of Information and Referrals for eligible clients

Output 2.1:

Provision of appropriate information and resources

Number and type of informational materials provided


Output 2.2

Provision of appropriate legal / non – legal referrals

Number of referrals made Proportion of clients accessing

referrals

IFAS Database Case Notes Follow up with

referral partners Client feedback

Ongoing IFAS staff May not need to aggregate by Warm and Active

Activity 3: Coaching for self-advocacy

Output 3.1:

Provision of appropriate capacity building skills and tools for clients

Number of coaching/self-advocacy sessions held with client

Recording of type of support provided (i.e.: notes, role-playing etc.)

Recording of any required communication with CP workers or other services

IFAS Database Case Notes Case Study bank

Ongoing IFAS staff There may be more than one “coaching session”

We could create categories of support in the

42

Item Indicators/Metrics of success Data collection method/source

Data collection timing


Types of issues addressed by clients

data collection tools to see which are the most frequently used

Activity 4: Direct advocacy with clients

Output 4.1:

Determining client’s position and goals

Number of interactions with client to meet goals

Recording of any required communication with CP workers or other services

Types of issues raised


Ongoing IFAS staff

Output 4.2:

Attending meetings with/on behalf of client

Number of meetings attended with client and CP workers

Number and type of meetings attended

Documented outcomes of meetings


Ongoing IFAS staff

Activity 5: Debrief Meetings

Output 5.1:

Provision of appropriate information and referrals to support client with next steps

Number of debrief meetings Documentation of client outcomes Types of information provided to

clients Provision of warm/active referrals to

clients (by destination) Proportion of referrals taken up by

current clients

IFAS Database Case Notes Follow up with

referral partners

Ongoing IFAS staff

Activity 6: Systemic Advocacy

Output 6.1:

Identification of prevalent issues facing clients

Development of IFAS case study bank Number of client case studies

documented Identification of primary systemic

issues Number of issues escalated to

Manager for systemic advocacy


Ongoing IFAS staff

Output 6.2

Actively engaging with relevant bodies to raise awareness of issues

Number of proactive submissions to relevant bodies (by how – IFAS/CP Program - and where)

Number of invited submissions to relevant bodies (by how – IFAS/CP Program - and where)

Number of proactive presentations at relevant forums (and where)

Number of invited presentations at relevant forums (and where)

Recording of contributions through “other” avenues (and where)

IFAS Database Case Study Bank

Ongoing IFAS staff

Output 6.3

Actively engaging with IFAS partners

Number of meetings with service partners to establish and maintain


43




relationships (with whom) Documentation of identified areas of

support for partners

Activity 7: Building partnerships and relationships

Output 7.1

Identifying and prioritisation of key stakeholders within the CP sector

Creation and updating of a database of relevant service providers and partners


Activity 8: Service Promotion and Community Education

Output 8.1

Development of resources for clients and service partners

Number of resources developed Number of resource updates through

pilot period (if required) Number of resources requested (by

requesting organisation)


Output 8.2

Delivery of appropriate training sessions for relevant service partners and stakeholders

Types of training modules developed Number of training sessions (where

and how many attendees)


Outcome 1: Increases support for clients to navigate CP processes

Improved awareness of system and rights and responsibilities

Client Survey Questions+ Ability to navigate the system+ Ability to better understand

rights and their position (in the system)

+ Relevance of tools provided and appropriateness for reuse

Qualitative client perceptions

Exit interview + Consent for future contact (1+year tbd)

Client Interviews Case

management tools

Exit Interview and follow up survey (TBD)

Mid Term / Final Evaluations

Ongoing

IFAS staff External

evaluator

Clients are better able to articulate their position

Client Survey questions+ Confidence to represent their

view+ Improved relationships with CP

worker+ Clients feel heard and listened

to (VLA and System (CP Worker))

+ Are clients prepared and informed to be able to represent their position with system stakeholders?

+ Perception of IFAS advocate to effectively communicate with


Client Interviews IFAS database



IFAS staff External

evaluator

44




their CP worker about views and preferences (if direct advocacy)

Qualitative client perceptions Comparison between client

objectives and outcomes

Clients trust the IFAS service to build their capacity to self-advocate / advocate on their behalf

Client Survey questions+ Perception on whether IFAS

service impacted the outcome of their case

+ Client understanding of distinction between IFAS service and legal services

Client case studies on impact of service on them


Client Interviews



IFAS staff External

evaluator

Outcome 2: Improved relationship between IFAS priority groups and CP workers

Increased skills for clients and CP workers to build rapport and relationships

Client perceptions on + Extent to which IFAS support

provided to clients has helped their working relationship with CP workers

+ Ways in which IFAS support has improved communication with CP workers

+ Ways in which IFAS has assisted client to articulate their needs and preferences to CP workers

Client interviews Case Study bank


External evaluator

Outcome 3: Clients are empowered to access supports

Increased client awareness of relevant and accessible support services

Client Survey+ Client satisfaction with referrals

to appropriate services+ Level of control over their

future actions.+ Feelings of security and peace

of mind Client case studies of utilisation of

referred support service to assist with needs


Client Interviews



IFAS staff External

evaluator

Outcome 4: Improved CP service provision in target areas for IFAS target groups

Operational cultural change between CP workers and target client groups

CP workers perceptions on + Extent to which IFAS support

and advocacy provided has helped their ability to understand the needs of target clients

+ The ways in which IFAS support has improved working

CP worker interviews / Focus Groups

IFAS database Case Study bank


External evaluator

45




relationships with clients+ What are the ways in which

IFAS has assisted client to articulate their needs and preferences to CP workers

Number of requests for resources (pictogram/flowcharts etc.)

Outcome 5: VLA offers an effective service for clients in need of advocacy support

Client are satisfied with IFAS service provision

Client satisfaction survey+ Client perception of information

and advocacy needs met by the IFAS service.

+ Client perception of comfort and ease in accessing VLA services

+ Clients overall satisfaction with IFAS service

Qualitative client perceptions


Client Interviews

Exit Interview


IFAS staff External

evaluator

IFAS service is delivered in accordance with client needs and appropriate channels

Service partner perceptions+ Is VLA the appropriate

organisation to offer advocacy service? (consider cultural (and clinical?) appropriateness)

+ Is the IFAS service able to address the contextual needs of clients?

+ Has IFAS service had an impact on service delivery / helped clients?

+ Are clients better able to articulate needs after working with IFAS service?

Service Partner Interviews


External evaluator

IFAS service has supported CP clients to participate or have a voice in decisions that affect them

VLA lawyers perception on IFAS service having an impact on the CP system / helped clients, (as far as possible given lawyers may not have capacity to hear this given duty system)

VLA CP lawyer interviews

Other relevant VLA staff interviews


External evaluator

Goal 1: Improved child protection outcomes and processes through a reduction in substantiated cases proceeding to court, proceeding to court as emergency care applications, or proceeding to contested interim proceedings

Reduction in substantiated cases proceeding to court, proceeding to court as emergency care applications, or proceeding to

Trends in reduction in initiations using comparative sites data on substantiation to initiationPossible comparative data sets:+ Previous years’ data with similar

demographic in same LGA+ Same years’ data with similar

demographic in similar LGAs

DHHS data Case notes CP worker

interviews / Focus Groups

ATLAS Database


VLA CP lawyers

External evaluator

Require Vincenzo to provide input on appropriate avenues to collect DHHS data

46




contested interim proceedings

+ ATLAS review of matters Comparison of sample of cases to see

what proportion would have led to initiations by other VLA lawyers

CP workers perceptions on a sample of (non-substantiated?) IFAS cases:

“Did you think this was on a trajectory to go to court?”

Do you think IFAS has anything to do with it not proceeding and if so what if not why not

Identification of further grants of aid related to child protection for IFAS clients, by type/issue

Goal 2: Reduction in future substantiations for clients/future contact with CP

Reduction in future substantiations for clients/future contact with CP

Proportion of IFAS clients who have had further notifications and not led to substantiation meetings

DHHS data Re-contact

clients and find out relevant information


External evaluator

Require Vincenzo to provide input on appropriate avenues to collect DHHS data

47

Appendix 5. Cost-benefit analysisThe principle of high returns on investment for advocacy programs, in particular disability advocacy programs, is well established. A 2017 report found a $3.50 return for each dollar invested in independent disability advocacy. 18 In the IFAS context, the economic costs of the child protection system are significant, costing Australian governments over $5.8bn in 2017/18.19 The Victorian system alone costs over $1.5bn in 2019/20,20 not including legal or Children’s Court costs, or other costs related to healthcare, education, employment, criminal justice or other related systems. Including these costs and costs related to reduced quality of life and premature mortality, studies have estimated the lifetime cost of all children abused in a given year to be around $20bn per year across Australia or $376,06821 per child.22 On these figures, preventing a single child from entering this system would exceed the cost of the IFAS pilot at $344,091 p.a. The cost of a full state-wide rollout of IFAS, estimated at $2.5m, would be offset by diverting just six children from the child protection system.

IFAS does work to prevent children from entering this system, however, it is not an automatic consequence that if IFAS prevents a child entering the child protection system that a saving of any kind would flow from that. Many of the indirect costs included in other cost-benefit analyses are calculating the cost of child abuse, incorporating child protection related costs as consequential costs resulting from child abuse. IFAS is not designed to prevent child abuse, neglect or maltreatment, although it is aimed at mitigating harms associated with involvement in the child protection system.

5.1. Cost-benefit analysis methodologyWhile a key focus for IFAS is diverting families away from the court system, often, as shown above, IFAS involvement will assist in supporting the family to comply with DHHS Child Protection requests and requirements and reduce the length of DHHS Child Protection investigations. For this reason, this analysis triangulates the qualitative and quantitative data to ask:

What are the benefits in relation to cost savings from children diverted from the child protection court system?

What are the benefits in relation to cost savings from reduced child protection involvement in the investigation phase?

Do these benefits outweigh the costs of delivering the current IFAS service, at $344,091 p.a.?

There may be other costs saved by IFAS which are not related to court diversion, such as those associated with timely and appropriate referrals to other services, however, these were not identified by the evaluation team and are not included in this analysis.

18 Anne Daly, Greg Barrett and Rhiân Williams, A Cost Benefit Analysis of Australian Independent Disability Advocacy Agencies (Disability Advocacy Network Australia, 2017).

19 Including protective intervention services, out-of-home care, intensive family support services and family support services. Report on Government Services 2020 (n 8).

20 Victorian Budget 19/20 Service Delivery (Budget Paper No 3, Victorian Government, 2019).

21 Adjusted for 2020 Australian dollars. McCarthy et al.’s original figures were $17.4bn in total and $328,757 per child in 2012-13.

22 Molly M McCarthy et al, ‘The Lifetime Economic and Social Costs of Child Maltreatment in Australia’ (2016) 71 Children and Youth Services Review 217.

48

This analysis identifies the minimum return on investment, being the lowest possible saving based on only empirical data, and the maximum return on investment, being the highest possible saving based on the most generous assumptions and interpretation of the data. Within these upper and lower points, it is possible to determine an estimated return on investment based on reasonable assumptions. As discussed below, the evaluation cannot address the second question at this point and will address it in the final stage.

The final evaluation will employ a nine-step process consistent with the Office of Best Practice and Regulation Guidelines on cost/benefit analyses:23

1. Specify the set of options;2. Decide whose costs and benefits count; 3. Identify the impacts and select measurement indicators;4. Predict the impacts over the life of the proposed regulation;5. Monetise (attach dollar values to) impacts;6. Discount future costs and benefits to obtain present values;7. Compute the net present value of each option;8. Perform sensitivity analysis; and9. Reach a conclusion.

At this point, there are insufficient data to adequately address each of these steps. The nature of IFAS and the complexity of the context in which IFAS works make a traditional cost-benefit analysis difficult to apply. Direct costs, including to VLA, DHHS and the Children’s Court, are relatively straightforward to determine and have been included. Indirect costs are more difficult to precisely determine; however, the evaluation has relied on available data for out-of-home care, foster care and family services costs. The evaluation has not included longer-term impacts such as mental health, education and housing which have been linked to child abuse rather than DHHS Child Protection involvement. Other costs are too complex to predict with any certainty. For example, civil claims for historical institutional child abuse against the Victorian Government was projected to cost $20m in 2019-20.24 Similarly, evidence highlights the generational nature of child protection involvement, and one child diverted from the child protection system may mean generations of children who are also not involved.25 As a result, steps 6 and 7 are unnecessary at this stage as this preliminary analysis only attempts to determine immediate costs within the 12 months following IFAS intervention.

Average costs can be determined with some certainty, but the available data do not allow for reliable determination of the actual impact of the program, in this case, the actual number of families who are diverted away from the court system. Figure 6 identifies the set of outcomes which can arise from IFAS intervention. This model highlights the numerous potential factors influencing the final return on investment. While the pilot budget has stayed static, the eligibility criteria for various IFAS levels have been adjusted, referrals to IFAS have increased and acceptance of IFAS by DHHS Child Protection has increased. These factors will stabilise over the next phase of the pilot and the final evaluation will be better positioned to accurately assess their consequences.

23 Department of Prime Minister and Cabinet, ‘Cost-Benefit Analysis’ (2016) <https://www.pmc.gov.au/sites/default/files/publications/006-Cost-benefit-analysis.pdf>.

24 Victorian Budget 19/20 Service Delivery (n 37).

25 NSW Department of Family and Community Services, The Prevalence of Intergenerational Links in Child Protection and Out-of-Home Care in NSW (Governement of New South Wales, 2017)<https://www.facs.nsw.gov.au/__data/assets/pdf_file/0016/421531/FACS_SAR.pdf>.

49

The key barrier in determining IFAS’s success rate in court diversion is that people who are referred to IFAS are not representative of the whole group of families engaged with DHHS Child Protection. Only half of the reports received by DHHS Child Protection are substantiated, so a significant number of families referred were never going to proceed to the court stage even without IFAS involvement. It can be assumed that IFAS is not mainly working with families who are least likely to proceed to the court stage, as there would seem to be no reason to refer these families to IFAS. These families might have a single one-off issue that is immediately resolved as a result of child protection engagement. It can also be assumed that IFAS are not being referred families who will inevitably end up in the court system, for the same reason. These might be families where there are no carers available, or where abuse is so serious that there is no potential for the parent to have the child returned to their care. It is likely that most families referred to IFAS are in a middle group, where it is possible but unlikely that they could avoid the court system. In determining the potential costs saved, it has been assumed that this ‘middle group’ of families who are being referred to IFAS have average costs associated with them proceeding to court as no data are available to determine which families are actually referred to IFAS. No evidence was found that IFAS is diverting families away from the child protection system who ‘need’ DHHS Child Protection, so this analysis has assumed that this is not occurring.

50

Figure 6 – System level impact logic

This preliminary analysis is also limited by the available qualitative and quantitative data. Quantitative data for court costs and legal grants has a significant lag, as this data is only collected after a family proceeds to court. The IFAS model of court diversion means a likely lag between intervention and reduced court engagement of approximately six months. For example, in March 2019, IFAS had only supported seven level 2-5 families, while legal aid was granted 599 times in IFAS’s catchment for 2019. It is anticipated that at the time of the final report any impact IFAS has had will be more clearly visible in this data.

In addition, the data collected by DHHS have not been provided for the interim review and will be included in the final report, and the data provided by VLA on case grants and expenditure have inconsistent collection relating to Aboriginal status and presence of disability, particularly intellectual disability. Qualitative data have not yet been collected from DHHS staff and will be included in the final report. If IFAS data can be matched to DHHS data, this will allow for stronger links to be made (Recommendation 2). The inclusion of DHHS data will allow for a narrower band between minimum and maximum return, a more accurate estimated return and a robust sensitivity analysis.

5.2. Cost savings from pre-court diversionDirect costs are available per child from DHHS, per case from the Children’s Court and per grant recipient from VLA. The average number of children per family with children is 1.8 in Australia, although probably higher for families in child protection, and the number of grants per case for grants of legal aid is estimated by VLA to be approximately three.

If IFAS is successful in diverting a client from the court system, they will eliminate DHHS costs related to applying for an order ($3,710.69 per order issued) and, for approximately one in three children, intensive family support services ($8,246.89 per child).26 IFAS support will not prevent the costs associated with the original report ($373 per report),27 as IFAS eligibility criteria require the family to be under investigation. The average cost per case in the Family Division of the Children’s Court is $1,235. The average grant of legal aid for child protection clients is $2,742, with an average of three grants per hearing meaning each case totals an average of $8,227. This means that each family diverted from the court system will save the Victorian Government an average of $18,084. If a child is also diverted from out-of-home care, this results in an average additional saving of $67,405 per child per year.28 The most common length of order made is for one year.29 Approximately 40% of all cases result in orders for out-of-home care, discounting this cost to $26,962, but assuming 1.8 children per family, reaching a total of $48,531 per family. The average cost for each family diverted from court is calculated at $66,616,30 as shown in Table 2.

26 All costs in 2020 dollars unless otherwise stated.

27 Report on Government Services 2020 (n 8).

28 This number is inflated by the high cost of residential care at $771,563 per child per year. Non-residential care is only $38,970 per child per year. The figure of $67,405 incorporates the low chance a child will be placed in residential care. It is not possible, using available data, to determine if children of IFAS service users are more or less likely than other families to end up in out of home care.

29 Mean length of out-of-home care order is not publicly available so the median length of 1 year has been used in this calculation. The mode, or most common, length of order is 1 to 2 years, so this is a conservative estimate.

30 This figure is used at this point in the evaluation. In the next stage, using DHHS data, the analysis will distinguish between court related costs and mean cost for each child diverted from out-of-home care.

51

Costs per child

Costs per participant

Costs per case

Number per case

Chance of occurring

Total

DHHS court costs $3,710.69 1 1 $3,710.69

Intensive family support services

$8,246.89 1.8 33% $4,913.05

Children’s Court $1,235.00 1 1 $1,235.00

Legal aid grant $2,742.00 3 1 $8,226.00

Out-of-home care $67,405.00 1.8 40% $48,531.60

$66,616.34

Table 2 - Costs of child protection court involvement

This mean cost to DHHS is unlikely to be close to the median cost. For many families referred to IFAS, DHHS involvement would likely have proceeded to court or avoided court irrespective of IFAS involvement, meaning IFAS involvement would not have resulted in any savings in court or related costs. At the other end of the scale, if IFAS involvement resulted in an entire family of five children not entering residential care who would otherwise have spent an average of five years in residential care that results in a saving on residential care costs alone of over $19 million for a single family. Preventing the same five children from entering non-residential care would save over $970,000.

Early VLA modelling in the original IFAS project plan assumed a 33% success rate, however, Table 3 shows that successfully diverting 4% of families away from the court system is sufficient to result in a positive return on investment for the Victorian Government.

To result in a positive return on investment for Victoria Legal Aid only, which has funded the pilot, IFAS would need to achieve upwards of a 28% success rate in court diversion, as shown in Table 4. This is complicated by another aim of IFAS, which is to increase access to justice. This means it is possible that any savings to VLA through court diversion may be reduced or cancelled out as a result of increased grants of legal aid from people referred by IFAS who otherwise would not receive legal representation. This increase in legal representation may result in further savings for the Victorian Government, as better legal representation can be expected to reduce court costs. If available data allows, this question will be explored in the final evaluation.

52

Table 3 - Return on investment (ROI) at 4% success (whole of government)

No. families p.a. 150Success in court

diversion4%

Cost saved to Vic Gov p.a.

$399,698.04

Cost of IFAS p.a. $344,091.00ROI per $1 invested $1.16

Determining the exact number of families IFAS has diverted from the court system is statistically not possible as there are too many uncontrollable variables leading to a decision by a DHHS Child Protection worker to apply for an order. Even a succession of randomized controlled trials would provide a range of results. Within the scope of this evaluation and this pilot, it is only possible to determine an absolute minimum through anecdotal data and through triangulation to estimate an actual rate.

5.1. Qualitative data on court diversion

As discussed in 2.4.6, IFAS is clearly diverting some families from court. Examples were provided by IFAS advocates, people who had used IFAS and independent third party stakeholders. These qualitative anecdotal examples cannot be independently confirmed and cannot be extrapolated to be applied to all families who engage with IFAS. While the IFAS advocates have a clear incentive to interpret their own work as successful, they are all experienced practitioners with high levels of understanding of the child protection system and processes. Accepting these limitations, the evaluation team was able to identify five distinct examples in a 12-month period where participants told us that IFAS had prevented the case from going to court. Added to this can be one example where the case did go to court but due to IFAS involvement, the court found in favour of the parent, avoiding average costs of $67,405 for out-of-home care. Just these six cases can be shown to have saved the state $400,486, as shown in Table 5 using the figures detailed above.

As IFAS has only operated at two-thirds capacity in the first 12 months prior to March 2020, this suggests a minimum return on investment of $1.75 for each dollar invested when running at full capacity. On the basis of these findings, VLA should continue to deliver the IFAS pilot (Recommendation 3).

5.2. Quantitative data on court diversionAs discussed above, DHHS data are not available for the midterm review, and even with DHHS data, it is likely too early to see a statistically significant impact from IFAS at this stage in the program implementation. The evaluation

53

No. families p.a. 150Success in court diversion 28%

Cost saved to VLA only p.a. $345,492.00Cost of IFAS p.a. $344,091.00

ROI per $1 invested $1.00Table 4 - Return on investment (ROI) at 4% success (VLA only)

Table 5 - Minimum return on investmentCourt diversions (5) $333,081.70

Out-of-home care (1) $67,405.00 Cost saved $400,486.70

Cost of IFAS $344,091.00 Actual ROI $1.16

ROI at capacity $1.75

team has used VLA child protection grants of legal aid case expenditure data as a simplistic proxy for all court costs as this data do appear to show some change in the pilot sites.31 Other analyses of quantitative data, such as numbers of grants rather than expenditure or quarterly expenditure data do not show conclusive results. Furthermore, a goal of IFAS is to increase access to legal services for Aboriginal and Torres Strait Islander parents or parents of Aboriginal and Torres Strait Islander children and parents with intellectual disability, who are overrepresented in the child protection system but underrepresented in grants of legal aid. Following consultation with an experienced statistician, the evaluation team has delayed complex statistical modelling until the final evaluation when all available data can be incorporated. With these caveats, case expenditure data does point to some promising potential in the pilot sites.

Figure 7 shows that in FY 2019-2020 (FY 2020), adjusted for the financial year to date as of 30/3/2020, case expenditure has increased very slightly in Bendigo and decreased in Darebin and Moreland. In each site, FY 2020 YTD case expenditure is below the linear trend. This is promising initial data, but both Darebin and Moreland, in FY 2017 and FY 2018 respectively, show previous more substantial drops than in FY 2020, highlighting the caution with which these data must be interpreted.

FY 2015 FY 2016 FY 2017 FY 2018 FY 2019 FY 2020 (YTD adjusted) $-

$1.00

$2.00

$3.00

$4.00

$5.00

$6.00

$7.00

GREATER BENDIGO (C) Linear (GREATER BENDIGO (C))MORELAND (C) Linear (MORELAND (C))DAREBIN (C) Linear (DAREBIN (C))

Figure 7 - Case expenditure by pilot site with linear trend

Figure 8 shows that with FY 2015 to FY 2018 data, the interpolation (predicted expenditure based on past expenditure) for FY 2019 and FY 2020, the interpolation for FY 2019 accurately reflects actual expenditure. 2020 expenditure, however, adjusted for the financial year to date, is 11% lower than interpolation based on 2015 to 2018 data. This shows a potential link between the introduction of IFAS and an 11% reduction in VLA grants of legal aid. It is not possible, based on this data, to infer that IFAS caused this reduction. The reduction may be a result of other factors, such as increased DHHS Child Protection resources in those areas. There may also be delays in claims for case expenditure or some other factor which impacts on the 2020 to date data.

31 Some community legal centres and block funded services such as the Victorian Aboriginal Legal Service (VALS) do a small amount of child protection work which is not included in these grants.

54

FY 2015 FY 2016 FY 2017 FY 2018 FY 2019 FY 2020 (YTD) $1.50

$2.00

$2.50

$3.00

$3.50

$4.00

$4.50

Pre IFAS Post IFAS InterpolationLower Confidence Bound Upper Confidence Bound

Figure 8 - Interpolated and actual case expenditure, Pilot LGAs

To address this, Figure 9 compares the IFAS pilot LGAs of Bendigo, Darebin and Moreland, with nearby LGAs with similar demographics and child protection service systems which would likely see similar variations in the data. These are Ballarat, Greater Shepparton, Moonee Valley, Maribyrnong and Banyule. Figure 9 shows that while average case expenditure per head increased both in the comparison LGAs and the rest of Victoria, it decreased in the pilot LGAs. Again, these data show that between 2019 and 2020 while case expenditure increased by 3% in the comparison sites and 12% in the rest of Victoria, it decreased by 1% in the pilot sites.

FY 2015 FY 2016 FY 2017 FY 2018 FY 2019 FY 2020 (FYTD adjusted) $1.50

$2.00

$2.50

$3.00

$3.50

$4.00

$4.50

Pilot LGAs Comparison LGAs Rest of VIC

Figure 9 – Case expenditure, Pilot LGAs, Comparison LGAs and Rest of Victoria

This is also not sufficient to determine causation but indicates a reduction of 4% for all child protection court proceedings in the pilot sites, not just people who used IFAS. In 2016-17 there were 1000 substantiations32 and 468 protection applications in the pilot sites, so even at full capacity, IFAS will only be working with approximately

32 A substantiation is a decision made by DHHS child protection staff that a child is in need of protection.

55

15% of people who might be diverted from these grants. These data suggest that something in the pilot sites in FY 2020 led to a reduction of approximately 25 cases from proceeding to court per year. IFAS worked with 99 families over this period. If all this is caused by IFAS, this 25-case reduction represents a 25% maximum success rate.

The qualitative data indicate the IFAS minimum success rate for court diversion is at least 5%, while the quantitative data allows for a maximum success rate of 25%. Taking a conservative approach to calculate where the likely success rate lies within this range, the evaluation team have reached an estimated success rate at 10%. Assuming 150 families p.a., 10% success rate represents 15 families diverted from court per year, saving $66,616 per family for a total of $999,245. This represents a return on investment of $2.90 for each dollar invested. These preliminary findings must be interpreted with caution but are very promising at such an early stage in the program implementation. Even at the minimum rate of 5% success, this results in a total saving of $499,623 or $1.45 per year within 12 months of the diversion occurring.

5.3. Priority group court diversionThe case expenditure data specifically for IFAS priority groups, Aboriginal and Torres Strait Islander families or families where one or both of the parents has an intellectual disability, are not of sufficient quality to determine if IFAS is having an impact on these groups specifically. This VLA data are clouded by historically poor collection processes, such as not linking Aboriginal children to non-Aboriginal parents or private lawyers failing to determine if a person identifies as Aboriginal or Torres Strait Islander. Other changes to the context, such as changes to s 18 of the Children Youth and Families Act 2005 (Vic) and the introduction of Children's Court Family Division at Broadmeadows Court,33 will have also influenced the case expenditure data for Aboriginal and Torres Strait Islander families. Intellectual disability is widely undiagnosed and people who are diagnosed and connected to services may or may not have benefited from the staged rollout of the National Disability Insurance Scheme (NDIS).

These complex factors influencing DHHS Child Protection involvement and access to justice may stabilise over the next phase of IFAS implementation so these groups will be revisited using DHHS data in the final stage of the evaluation.

5.4. Cost savings from shorter DHHS Child Protection investigations

IFAS support may reduce investigation costs, but at $1,601 per notification investigated,34 this is likely to be an insignificant amount when compared with potential savings from court diversion. At this point in the evaluation, it is not possible to say with any certainty to what extent IFAS is reducing costs due to shortened DHHS Child Protection involvement, although it is clear from the qualitative data that IFAS does often reduce the length of the

33 https://www.childrenscourt.vic.gov.au/about-us/court-locations/childrens-court-family-division-broadmeadows

34 Report on Government Services 2020 (n 8).

56

Success in court diversion 10% 5%

No. families p.a. 150 150

Cost saved p.a. $999,245.11 $499,622.55

Cost of IFAS p.a. $344,091.00 $344,091.00

ROI per $1 invested $2.90 $1.45

Table 6 - Return on investment (ROI) at 5% and 10% success

investigation phase. This is because these cases are often sitting open with minimal actual investment by DHHS. Advocate 1 explained this process:

Mother just was not engaging with the Department and they’d had the investigation open for quite a few months beyond their timeframes, and they want to just talk to the children. They were thinking they wanted to close, but they had concerns around family violence, and they needed to speak to the children. And she was just refusing. She said, “I’m not jumping through the hoops anymore”. So, I spoke with her on the phone and explained. She said, “Well, they want to talk to the children, can they do that?”. And I said, “Well, actually their policy states if they are planning on closing, they can, they do need to talk to the kids so yes, they do”. So, she said, “Okay”. The next thing I knew she came into the office and had a meeting and she hadn’t done that before, agreed for the children to be spoken to and they were able to close. (A1)

This case may have stayed open for several more months, but it is not clear that this would have resulted in significantly increased expenditure. Service User 8 had a similar experience:

I think within another 10 days after that they actually closed with me, and they’d been telling me they weren’t going to close with me for months. (SU8)

The IFAS Manager also highlighted this function of IFAS:

Files are closed when previously they’ve just sat open for a range of different reasons, [for example] that there is an improvement in communication. It might still go to court but for a Family Preservation Order. That seems to be the most frequent one that we’ve had, this disconnection and then there’s reengagement when IFAS has become involved as a facilitator … [and a] Family Preservation Order is usually put in place. (IFAS Manager)

Even if IFAS is reducing investigation costs by 25% in 25% of cases, this is a total saving of $15,009. The earlier closing of these files probably only had a negligible saving in terms of DHHS expenditure, and no quantitative data is available to triangulate this phenomenon. This question will be revisited using DHHS qualitative and quantitative data in the next stage of the evaluation.

57

Appendix 6. Mapping against key evaluation questionsThis section summarises the assessment of the evaluation team against each key evaluation question, incorporating all data collected and analysed in the midterm review, highlighting areas for where further data collection is required.

1) Has IFAS successfully achieved its aims? a) To establish and trial a pilot independent child protection non-legal advocacy service, providing

representational advocacy services to parents involved in the child protection system and develop evidence through robust monitoring and evaluation to determine impact and outcomes;

IFAS has been established and is providing representational advocacy to parents in the child protection system. More work is required to develop a measurable monitoring and evaluation framework. Outcomes are emerging on an individual level, but systemic impact remains unclear (see 2.2).

b) To develop and deliver the service working closely with key stakeholders including families and individuals with lived experience and other key stakeholders.

IFAS works closely with key stakeholders. More work is required to ensure stakeholders understand the service. The perspective of DHHS workers will be assessed in the final stage of the evaluation (see 2.4.8).

2) Has IFAS successfully achieved its objectives? a) To enhance clients’ capacity to self-advocate, understand and enact their rights within the child

protection system, better understand their current involvement with child protection, and make informed decisions about their family;

People who had used IFAS reported an increased ability to self-advocate but noted that they lacked the professional status that enabled IFAS advocates to work effectively. People who had used IFAS reported that they understood their rights, or lack of rights, better, and were more able to make decisions as a result of IFAS support (see 2.1).

b) To assist clients to access support services they identify they require;

People who had used IFAS reported some increased access to other services, including legal services, however IFAS data does not adequately reflect outgoing referrals (see 2.2 & 2.4.8).

c) To ensure that clients receive high-quality services and have a positive experience of the advocacy service;

People who had used IFAS reported an extremely high satisfaction with IFAS and overall positive experiences (see 2.1).

d) To enable clients to obtain legal advice about their child protection case from qualified legal practitioners; and

Some evidence was found that people were obtaining legal advice where they might not otherwise do so, however IFAS does not adequately collect outgoing referral data and neither VLA nor private lawyers collect this information in an accessible way (see 2.5.3).

58

e) To support a reduction in the proportion of substantiated cases in the pilot area proceeding to court, proceeding to court as emergency care applications or proceeding to contested interim proceedings.

IFAS is reducing the number of substantiated cases proceeding to court. The rate of families diverted from court can be shown to be above 5% and is estimated to be approximately 10% (see 2.4.6, 2.5.1, 5.1 & 5.2).

3) Is IFAS reducing the number of matters which proceed to court?

IFAS is reducing the number of substantiated cases proceeding to court. The rate of families diverted from court can be shown to be above 5% and is estimated to be approximately 10% (see 2.4.6, 2.5.1, 5.1 & 5.2).

4) Is IFAS better supporting families through the child protection process than a family’s previous experience of child protection without an advocate?

People who had used IFAS reported a very high level of satisfaction with IFAS, but this did not translate into increased satisfaction with the child protection system (see 2.1).

5) Is IFAS delivering a better experience for children in the child protection process?

IFAS does not provide services to children, however stakeholders indicated a range of indirect benefits to children (see 2.5.6).

6) Is IFAS operating consistently with its values and principles?

IFAS is operating consistently with its values and principles (see 2.4), however staff reported tension in embedding these in practice. Continued leadership is required to ensure consistency.

7) Is IFAS delivering advocacy consistent with its advocacy model?

IFAS is operating consistently with the model of representational advocacy (see 2.4), however staff reported tension in embedding this in practice. Continued leadership is required to ensure consistency. In particular, without continued focus on the model, there is a risk that IFAS advocates could inadvertently assist DHHS Child Protection in maintaining compliance (see 2.5.4). Further clarity is required in who the target of IFAS advocacy is intended to be (see 2.5.5).

8) Is IFAS operating consistently with its service model? Is this service model the most appropriate for IFAS?

IFAS appears to be operating consistently with its service model, although elements of lived experience could be further developed (see 2.5.2).

9) Is IFAS operating consistently with its policy and procedures?

The evaluation found no significant deviation between IFAS practice and IFAS policy and procedures.

10) Is IFAS having a systemic impact?

The evaluation found no evidence of systemic change as a result of IFAS and stakeholders acknowledged that this was probably not yet occurring.

59

RMIT Midterm Evaluation of Independent Family Advocacy and ...€¦ · Web viewThe evaluation...

Documents

Transcript of RMIT Midterm Evaluation of Independent Family Advocacy and ...€¦ · Web viewThe evaluation...