Better Results, Stronger Communities PATHFINDER Mission: Maximise consequential outcomes for New...
-
Upload
ashley-blair -
Category
Documents
-
view
221 -
download
0
Transcript of Better Results, Stronger Communities PATHFINDER Mission: Maximise consequential outcomes for New...
Better Results, Stronger Communities
PATHFINDERMission: Maximise consequential
outcomes for New Zealanders
Better Results, Stronger Communities
Outcomes for a Vital Nation
• Focus on outcomes; keep output platform
• Clearly specify & measure outcomes, to inform strategy, plans, purchase, ops
• Continuously improve interventions, etc
• Business models for enhancing outcomes (focus: core activities & new initiatives)
Increased benefit to our communities
Better Results, Stronger Communities
Current Initiatives
PATHFINDER
CAPValue-for-
Money
TOOLBOX; DECISIONMAKING SYSTEMS
STRATEGY,CAPABILITY,
PERFORMANCE
TRADE-OFFS,TRANSFERS(Within Vote)
(Next FY)
Better Results, Stronger Communities
Overall Schedule
• 2 years – like each agency to achieve 3-4 outcome mgmt goals it sets (more later)
• Early milestones (indicative) are by:- Oct ’01: identify / confirm outcome measures- Dec ’01: identify international best practice for managing for improved outcomes in your area- Dec ’01: proposed outcome measurement, analysis and internal reporting frameworks out for discussion
• You set the pace via goals / resources
Better Results, Stronger Communities
Sponsors’ Group• Support & guidance for Working Group• Peer review of learning points, etc• Link to Senior Mgmt teams, CE, Minister
Working Group• Identify / document best practice• Review agencies mgmt systems / models• Advice on design & application
Both: Growing capability to improve outcomes
Better Results, Stronger Communities
Participation / Lead RepresentativeAgency Sponsors’ Group Working Group
CYFS Brenda Pilott Rose Ryan
Conservation Hugh Logan (Chair) Grant Baker
Corrections Jared Mullin Leon Bakker
Customs Tim Horner Margaret Niven
Health Gillian Durham Mark Booth
Inland Revenue Alan Pinder David Toohey
Labour Geoff Bascand Marg Harvey
LTSA Tony Bliss Martin Small
SSC Tony Hartevelt Greg Claridge
Treasury Andrew Kibblewhite Roger Waite (Chair)
Better Results, Stronger Communities
PATHFINDERMission: Maximise consequential
outcomes for New Zealanders
Better Results, Stronger Communities
Principles of Pathfinder
• Bottom-up, organic process• Coalition of the willing• Eight Govt Agencies, each with diverse
businesses & business processes• Hot house for sharing / building capacity• DC / Vote Teams remain point of contact• SSC & TSY advise; collate lessons learnt
Better Results, Stronger Communities
Ownership from Agencies, backed by Collective Action
• Want ownership / steerage from your CE, senior management and your organisation
• Benefits your agency (& community served)• Want you to own process & mgmt models• Other agencies offer ‘wise counsel’• Positive culture: sharing, helping, learning
Better Results, Stronger Communities
Working Group Functions?
• Monthly meeting (2-3 topics, & material for SG)
• Focus teams +/- workshops
• Establish mentoring / bilateral relationships• Brokerage (SSC/TSY/Others putting you in contact)
• E-mail discussion (hot topics needing urgent advice)
• Web Site (password access to outcome mgmt info)
• Collate WG papers into ‘living manual’
Better Results, Stronger Communities
Resource Commitment
• Gains will require high calibre staffing
• Low marginal cost (<0.2 FTE?), if committed to outcome-driven mgmt
• For newer entrants, marginal costs are driven by the objectives you set (and orientating business to achieve them)
• Central agencies cover admin. support (for meetings / process - not an offer to do your photocopying)
Better Results, Stronger Communities
Agency Team Profile?
• Highly competent analyst-managers• Highly motivated, results-driven• Good communicators, working within
and with strong internal networks• Knowledgable about your business (incl.
purpose, goals and operational realities)• Continuity of staffing critical
Better Results, Stronger Communities
Unique Sector, Unique Outcomes,Unique Opportunities
• Primary outcomes differ …• What can be measured will differ …• What applications will work varies by sector …• No one approach will work for all agencies …• For some functions, no approach may work well
Be clear about what you want / need, butBe prepared to innovate & make compromises
Better Results, Stronger Communities
PATHFINDERMission: Maximise consequential
outcomes for New Zealanders
Better Results, Stronger Communities
Managing for Outcomes
• Scope (measures, maximisation, strategy)
• Decision making, better outcomes (evaluation just a tool)
• Measuring outcomes (measure types, groups)
• Linking measures• Outcome hierarchies• Defining your work agenda (‘straw man’)
• Next Steps
Better Results, Stronger Communities
Feedback +/- motivation for staff
Active prioritisation within teams
‘Blind’ managers see hurdles
Business and strategic planning activities informed and focused
Better Results, Stronger Communities
3 Families of Outcome Measure ?3 Families of Outcome Measure ?
Family Typical Use (Comparison Group)
State IndicatorState Indicator Situation report; Trend report; Benchmarking; Target setting; Broad targeting of individuals
(Comparison group: None, or other state indicators)
Intervention Intervention Effect SizeEffect Size
Intervention effectiveness with target group; Investment decisions (CBA, total $, etc)
(Comparison group: untreated or ‘status quo’ best treatment - matched pair, random selection, etc)
Decision Making Decision Making (e.g. assessment systems)(e.g. assessment systems)
Risk management; Priority setting; Demand analysis; Investment decisions (CBA, total $)
(Comparison group: None, or adjacent risk groups)
Better Results, Stronger Communities
Outcome Management
• Outcomes measured to inform decisions (start by defining the decisions to be informed)
• Rhetorical vs. quantified & measurable • Defining measurement groups critical• Focus on critical measures / groups
(overdesign will result in poor prioritisation)
• Check utility of systems & measures against needs of different stakeholders
Increasing benefit to our communities
Better Results, Stronger Communities
Short-Term Wins – Long-term Gains1. Define & measure ‘mission critical’ outcomes
(using ‘state’ or ‘situation’ outcome indicators)
2. Map causal logic linking outcomes to outputs3. Assess impact of interventions / targeting4. Assess cost-effectiveness (vs. outcomes achieved)
5. Managing to maximise outcomes (e.g. core/pilots)
6. Benchmarking with outcomes (business units / nations)
7. Focus strategic / annual plans on improving o/c8. Redesign planning & operations to maximise o/c
(incl. feedback & continuous improvement)
See information pack for details
INTERVENTION FRAMEWORK(Systems Analysis & Design)
SCREENING?
ASSESS?
ASSESSMENT?
TREAT?
ALLOCATETREATMENTS
?PROGRAM
?
NOTREATMENT
(Screened Out)
NOTREATMENT
(Assessed Out)
TREATMENTA
TREATMENTB
TREATMENTEtc.
YY Y Y Y
No
No
CLASSIFICATION ERRORS (Types 1 & 2)
PERFORMANCE OF DECISION MAKING / ALLOCATION SYSTEMS
INTERVENTION EFFECT SIZE;PERFORMANCE OF INTVN. MIX
NOTREATMENT
(Logistics,or Poor Fit)
Use outcomes of assessment system to enhance system and set intervention thresholds to maximise intervention outcomes (go beyond just measuring targeting errors)
Better Results, Stronger Communities
When is Targeting Useful?Universal / Defined Access
Mildly Problematic
/ Remedial
Highly Problematic / Remedial
Low Marg-inal Cost
Interventionn.a.
Low cost,
Lo selectiveHi. Selective Hi. Selective (Least cost) (Least cost)
Moderate Cost Intvntn. n.a.
Least cost, Selective
Hi. Selective Hi. Selective (Least cost)(Least cost)
High Cost Intervention Rare
Hi. Selective Hi. Selective (Least cost)(Least cost)
Hi. Selective Hi. Selective (Least cost)(Least cost)
Goal: maximise intervention effect from (targeting + intervention) effort
Target
Intervention
Sector / Department Level Outcome Frameworks
V ic tim isa tion R ates(In d ivid u a ls / E n tit ies )
F ewer V ic tim s(Ju s tice )
F rom C rim e P rosecu tin g C rim e
P erce ived S a fe ty(Ju s tice )
R eso lved
Tim es from C h arg esF iled to P rosecu tion
(C ou rts )
C ou rt H ou rsb y C rim e Typ e
(C ou rts )
P rop ort ion C on vic tion sO vertu rn ed on A p p ea l
(C ou rts )
P rosecu ted C on vic ted
C rim es S o lved(P o lice )
[P ro po rtio n; T im eliness]
R ed u cedR e-o ffen d in g
(C orrec tion s , F G C ,D ivers ion s )
L ess R ep orted C rim e(Ju s tice / P o lice )
C O M M UNIT Y S A F E T Y(F rom C rim e)
(Disaggregation generally required by demographic / service groups)
Better Results, Stronger Communities
Cross-Service Outcome Framework
New Initiatives M oney: Interventions w ith Com m on Outcom es
O p en R oad S p eedM an ag em en t
L ig h t V eh ic leS tan d ard s
E n forcem en t
E xp an dC on s tru c tion
E n g in eerin g
Tran sp ort W e lfa re
C on d it ion -Trea tm en tP a irs # 1 to 6 6 6
Trea tm en t P reven ta tive
H ea lth
D ie t S m ok in g
L ife Q u a lityE d u ca tion
E d u cation
Q uali ty A djus ted L ife YearsIn te rven tion s F rom A n y S ec to r
(M easu re o f E ffec tiven ess :C os t p er Q A L Y G a in ed )
Pool all proposals with a common outcome, and allocate new initiatives funding using cost-effectiveness or cost-benefit principles?
Better Results, Stronger Communities
Next Steps• Identify learning fora on topics where you
have special skills & experience to share, or have needs (build collective knowledge base)
• Identify your development objectives (3-4 ‘wins’?) & indicative timeline
• SG (you?) to comment on ToR out of session
• Early Sept: second Working Group meeting• Indicative work plan to discuss Sept at SG
Comment on ToR ASAP please; Identify your work agenda
Better Results, Stronger Communities
PATHFINDERMission: Maximise consequential
outcomes for New Zealanders
Better Results, Stronger Communities
(Slides to be Presented End)
Outcome Performance Measurement & Alignment to Govt Goals
Ex-Ante Intervention Analysis
Ex-Post Policy Evaluation
Governance &
Accountability
Arrangements
Output Measurement
& Delivery
Production & Processes
Input Monitorin
g & Control
Output Pricing & Funding
Arrangements
1
2
3
4 5
6
7
8
Economy
Efficiency
Delivery (current and future = capability)
Dir
ectio
n
Effe
ctiv
enes
sInputsOutcomes
Outputs
Value for Money
Treasury
19 April 2001
Better Results, Stronger Communities
Outcome Measurement Traps
• Outcome definition specific to particular service(s)• Selective outcome measures• Fuzzy outcome definitions• Measuring outcomes for few / wrong groups• Slow feedback (decision cycle times critical)
• Measurement error• Poor accountability / attribution / disclosure• Outcomes via other agents (e.g. policy; science?)
• Generating capability only (e.g. Defence)
Better Results, Stronger Communities
• Measure(s) too specific (if feasible, identify at least one unifying measure spanning service group(s), preferably in terms linked to risks & costs, e.g. days unemployment [average or avoided], Quality Adjusted Life Years, lives lost / saved, criminal re-offending risks / costs avoided; future costs to the Crown / society avoided)
• Selective measures (define range & hierarchy of potential outcome measures and groups against strategic goals and plans; review & plagiarise outcome definitions used by overseas agencies)
• Fuzzy definitions (specify clear units of measure; ensure measures & measurement groups cover critical interests & expectations of the agency / sector [see ‘Poor Accountability’])
• Slow feedback (reduce selection interval if sample size permits; reduce follow-up period for initial measures; intermediate outcomes)
Measurement Tips (1 of 3)
Better Results, Stronger Communities
• Few / wrong groups (report overall outcomes for major service categories; must also identify & report outcomes for key demographic, treatment, geographic, business and/or risk sub-groups [see strategic plans for sector/agency]) (for intervention outcomes, ensure intervention and comparison groups differ ‘only’ vs. intervention(s) being evaluated, or use specialist statistical tools [e.g. Odds Ratio]; can also partition individual intervention groups by demographic variables, risk, service provider, etc for more resolution)
• Measurement error (match outcomes to best data; enhance input and grouping data quality; determine statistical variation; allow for statistical variation in interpreting outcome information) (for intervention outcomes, randomly select or match comparison group) (for risk tools, must validate precision of risk measurements against NZ longitudinal data set other than that used to create the risk tool)
Measurement Tips (2 of 3)
Better Results, Stronger Communities
• Poor accountability (ensure groups used in primary care of accountable agency; and outcome is specific to that agency; record external influences as inputs to statistical tests) (for intervention outcomes, must create reference group [e.g. randomly assigned, paired or wait-list controls, alternate treatment(s)]; allow for interactions between interventions via evaluation protocols)
• Outcomes delivered via others (Problem: accountability limited unless agency has control or substantive influence over other agencies; otherwise excuses abound. In the case of sector policy agencies, limited accountability could be expected for polices contributing across the sectors’ hierarchy of outcomes)
• Capability outcome (define clear performance expectation; model likely performance against expectation or assess during unplanned contingencies +/- field exercises +/- deployments)
Measurement Tips (3 of 3)
Better Results, Stronger Communities
Costs & Risks of Targeting
• Information distortion (‘cheat strategies’, error)
• Incentive distortion (e.g. reduced motivation)
• Negative effects & stigma (for those targeted)
• Administrative costs & invasive losses• Political sustainability (incl. quality reduction)
After Amartya Sen, in Public Spending & the Poor (World Bank, 1995)
Without recognising it, we are targeting all the time …
The question is how far to push … discrimination
Better Results, Stronger Communities
Targeting Efficiency (Error)
CLASSIFICATION ERRORS (Types 1 & 2)
“… greatest source of error is with false-negative diagnoses, which are
detected only rarely by review …” AA Renshaw,
Am.J. Clin. Pathol. 115 (3): 338-341, 2001
Poverty line
INC
OM
E (
$)
PROPORTION OF FAMILIES (%)
B
A
C
D
Effect of transfer = A + B + CEfficiency = A / (A + B + C)
Break-Even
Income before interventionIncome after intervention
Better Results, Stronger Communities
Outputs and OutcomesHM Treasury (UK)
Value for Money
Better healthMore treatmentsMore nursesMore money
Other influences
Resources Input Output Outcome
Economy Efficiency Effectiveness
Better Results, Stronger Communities
PATHFINDERMission: Maximise consequential
outcomes for the public from Government activity