Understanding Economic Evaluations 101:The Challenge of Demonstrating Value

Graham Clyne, Executive DirectorPeel Children and Youth Initiative

Crime Prevention Conference, MississaugaFriday November 9, 2012

CANADIAN INSTITUTE FOR ECONOMIC EVALUATION"Making the Case for Value: the Tools and Methods of Economic

Evaluation”

CIEE

CIEE

Your presenter

» Special interest in children and prevention» Researcher, administrator, funder, community

developer, volunteer, consultant – diverse perspectives

» Background in public policy

» Prevention Dividend Project 2000» Canadian Institute for Economic Evaluation

2001

CIEE

Background

» Founding Director: Canadian Institute for Economic Evaluation

» Associate Researcher: System-Linked Research Unit, McMaster University

» Organizational Consultant: Public and Non Profit Sectors

» Research Director: Prevention Dividend Project» Currently, Executive Director: Peel Children &

Youth Initiative

CIEE

Today’s presentation

» Prevention Dividend Project /CIEE» Key Principles and Concepts Involved» Common methods of Economic Evaluation» Costing Studies: Purpose and Method» Measuring System Service Utilization» Communication Strategies» Questions and Discussion

CIEE

The Prevention Dividend Project:Why do Economic Evaluation?

Dr. Paul Steinhauer: “If prevention really works, why do we seem to struggle in the battle for public support?”

GC: “Because we – the people in our business - haven’t really demonstrated that prevention is abetter, more efficient use of scarce resources. We get by implying that it works and telling stories……but the burden of proof is going up.”

CIEE

Prevention Dividend Project: The Burden of Proof Goes Up

» Exponential growth in programs and services; many competing societal “goods”

» Narrow organizational mandates = an evaluation focus on very few outcomes

» “Outcomes” described in the language of professionals, service providers (e.g. Improved well being)

CIEE

Prevention Dividend Project:Burden of Proof…

» Weak evaluative structures and processes – inverse relationship between quality of evaluation and program assessment

» Confusion of Process and Outputs – with end state and/or intermediary outcomes

» Singular focus on effectiveness – not efficiency

» Evaluations tend to rationalize the ongoing use of resources

CIEE

Prevention Dividend Project: Societal Motivations

» General growing public skepticism re: effectingchange/risk factors

Phillip Donne: Kelloggs Canada…“We spend $12 million annually for new products expecting that only a few will work, but in your business everything stays on the shelf”

» United Way annual review process – Units of Service Assessments… is this really cost-effective?

CIEE

Prevention Dividend Project:Motivations…

» Social Health and Community Services always described as “soft” services ..despite measurable economic impacts

» Changes in Donor Preferences Measurable Impact / Direct linkage to outcomes expressed in terms of value

» Requests for “Return on Investment” Impacting all sectors of public and charitable work

CIEE

Initial Negative Reactions…

“Prevention actually costs society money – re: smoking kills – saves money!?!

“Prevention is about improving the quality of our lives – not about saving money”

“Do this in another sector – we’re not fat comparatively speaking”

“We don’t know enough about outcomes – much less value”

CIEE

Economic Evaluations:Common Myths

E.E.: is a difficult or impossible methodology for health promotion and community based services

E.E.: is an inappropriate tool for measuring human care services

E.E.: primary focus is on budget reductions, generating savings

CIEE

Economic Evaluations: Actual Barriers

» Practitioners perspective/justification paradigm

» Comparative nature of economic evaluation – “It’s all good”

» Motivated by external threat or funding and resource challenges

» Poor outcome measures and causality issues

» Training, sills, tools, resources

CIEE

Economic Evaluations: Applications

» Provides new and better information for decision making – “at what cost”

» Speaks to effectiveness – and efficiency; where and when “returns” will diminish

» As a Management or funding tool – choosing between competing options

» Speaks in a language that those outside the business can understand

CIEE

Economic Evaluations: Applications

» As a “tipping point” – a necessary but not sufficient element

» Allows for creative and useful cross-sectoral comparisons

» Advocating for support without a shared “social justice” perspective

» Adds a greater policy relevance to “soft” services and programs

CIEE

Economic Evaluations: Applications

» Shows the interconnectedness of programs and services

» Captures wider range of outcomes and benefits (societal perspective)

» “Cost of” (burden of illness) studies: a community motivator and a different approach to collaborative action

CIEE

Economic Evaluations: Applications

» Creating new allies; those who would benefit from change

» Building and sustaining funder/donor support

» Studies and messages can be tailored to suit the need

» “Community Benefit” – better described in monetary terms with broad impacts

CIEE

Economic Evaluations: Limitations

» Ideological Opposition may Prove Intractable

» Policy Outcomes – blend of art, science, politics, timing, etc.

» Resonance of “Viewpoint” to Intended Audience

» Some outcomes cant be valuated with credibility

CIEE

Economic Evaluations: Limitations

» Professional/internal resistance

» Re-stigmatizing populations as “expensive”

» Dilution of language/terminology

» Logistical/resource challenges

» Imposed methods – may be very poorly suited to your work

CIEE

Economic Evaluations: Looking Ahead

» Impacting a wide range of sectors

» Successful illustrations, creative applications growing

» Shift to S.R.O.I – extrapolating “benefits”

» Largest challenges – expert support and fiscal resources

» Limited training opportunities

Key concepts, principles and methods of economic evaluation

CIEE

CIEE

What do we mean by economics?

ECONOMICS TOPICS:» Things to do with the economy (e.g.,

employment, interest rates etc.)» Many disciplines contribute to understanding the

economy (e.g., sociology, finance, psychology etc.)

ECONOMICS DISCIPLINE:» The tools and methods used by economists to

analyse problems» Not confined to contributing to understanding

economics topics

CIEE

Fundamental Concepts

SCARCITY OF RESOURCES:» Possible uses of resources exceed their availability

CHOICES:» Choices required about how to use available

resources - implicit, but cannot be avoided

OPPORTUNITY COST:» Choosing to use resources one way means choosing

not to use them in other way

Question: Do these “concepts” apply in your work?

CIEE

Economics and Money

Canada’s Health Care “Solution”Increase expenditures by $15 billion

» Does more expenditure lead to more resources?

» Do more resources lead to more care?» Does more care lead to more health?» Does more health lead to more well being?

MONEY: A MEANS OF EXCHANGE NOT A RESOURCE FOR PRODUCTION

CIEE

Other Critical Concepts

» Equity – in your indicators

» Discounting – time preferences

» Sensitivity analysis

CIEE

Discounting and Time Preferences

» Discounting reflects the loss in economic value that occurs when there is a delay in incurring a cost or realizing a benefit (value today vs. value tomorrow)

» Not to be confused with inflation – all calculations should be in common (current) year dollars

» All expenditures and benefits must be included and multiplied by a percentage over the number of years that costs and benefits occur

» The discount rate is pivotal especially in prevention/early intervention

CIEE

Discount Table

CIEE

Sensitivity Analysis

» Every evaluation will contain some degree of uncertainty, imprecision, or methodological controversy

» Sensitivity analysis is performed to test if variations in the assumptions or estimates underlying an analysis do not produce significant alterations in the results

» Deterministic (i e given as point estimate) vs. stochastic (i.e., given as distributions) type data

» Know the Implications of your sensitivity analysis

CIEE

Viewpoint and Cost Identification

» Whose Costs and Whose Consequences are included in the analysis?

» Some possible Viewpoints:- Agency- Government- Consumer/client- Regional, State- Board of Directors/market share- Societal (best and most inclusive)

» Broader viewpoints are useful to capture any costshifting – a common practice

CIEE

What is Economic Evaluation

» The comparative analysis of alternative courses of action in terms of both their costs and consequences

» Involves both the concept of technical efficiency and allocative efficiency

» Differs from SROI – a calculation or roll up of (attributable) benefits

CIEE

Basic Methods of Economic Evaluations

» Cost Effectiveness Analysis (CEA)

» Cost Utility Analysis (CUA)

» Cost-Benefit Analysis (CBA)

» Costing Studies – Burden of Illness

» Service System Utilization

CIEE

Cost Effectiveness Analysis

» Compares different interventions with same sort of unambiguous outcome

» Expressed in common units of outcome – lives saved, employment attained

» Determines which intervention most efficiently produces the desired outcome by comparing costs per unit of outcome (search for dominance)

CIEE

Cost Effectiveness Analysis

» Demonstrates correlation between amount spent and results

» achieved (generally more costs = improved effectiveness)

» Used to capture marginal or incremental rates of effectiveness (i.e.: generally a diminishing marginal returns)

» Ratios demonstrate the additional cost of one intervention per unit of outcome achieved – a more accurate reflection of alternative costs/benefits

» Associated with the rarely used Cost Minimization Analysis (CMA) (the search for the lowest cost alternative where outcomes are identical)

CIEE

Choosing a Measure of Effectiveness

» Program’s effects are measured in natural units – e.g. avoided episodes

» The Chosen measure of effectiveness depends on the objective(s) of the programs being evaluated

» A CEA requires one unambiguous objective of the intervention and therefore a clear dimension along which effectiveness can be assessed

CIEE

Cost Utility Analysis

» Commonly used in health care – to compare complex / multiple impacts from various interventions (uncertainty)

» Creates a single - subjective - measure of utility or preference

» Subsumes multiple effects into a single – most important – measure of quality (eg: QALY’s ; HYE’s)

» Most efficient option = best incremental ratio of costs/outcomes – or costs per QALY gained

CIEE

Cost Benefit Analysis

» Generally broadest, most inclusive assessment of costs and consequences using the societal perspective

» All inputs valued in monetary terms – allows differing interventions with differing outcomes to be compared

» Describes and compares the net gain (assuming benefits surpass costs) of various options

CIEE

Cost Benefit Analysis

» CBA requires program consequences to be valued in the same units as the costs (i.e.; typically monetary units), thus enabling a direct comparison of the program’s incremental costs with its incremental consequences in commensurate units of measurement.

» CBA is not “cost analysis” or “cost saving” analysis.

CIEE

Costing Studies / Burden of Illness

» Commonly used in health care, disease prevention messages – cost of illness (COI)

» Used to value morbidity outcomes in CBA» Aggregate assumed costs of treatment, lost

productivity, out of pocket (cost shifting), cash transfers – some include suffering, care giver costs. Administrative cost usually excluded due to differing cost structures (US and Canada)

» Not a full economic evaluation ( no comparison next best alternative uses for resources)

» Problems with assumptions re: lost productivity – not a measure of preferences to avoid outcome

CIEE

Measuring Service Utilization

System-Linked Philosophy and Approach…» Assumes multiple / overlapping conditions» Expects differential effectiveness – “do no

harm”» Takes a comprehensive view of economic

impacts» Uses a straightforward methodology – shopping

basket approach…. Add it up!

Effectively demonstrates the substantial cost of “doing nothing”

CIEE

Comprehensive vs. Self Directed Care:Measuring Service Utilization

» Randomized large sample of single parents / children on welfare

» Comprehensive care includes health, recreation, daycare, public health, employment training, etc.

» Self-directed care = fending for themselves» Blend / mix of single services» Recorded all program costs; measured against

reduced use of inappropriate remedial services» Findings: CC pays for itself in a single year

CIEE

Total Per Child Expenditures for 214 Children’s Direct Use of Health and Social Services 2 Years After Proactive, Subsidized Recreation

CIEE

Recreation/Childcare PAYS for itself by reductions

½ THE USE OF SPECIALISTS½ USE OF C.A.S SERVICES

¼ USE OF OCCUPATIONAL THERAPIST1/3 USE OF PHYSIOTHERAPISTS

½ USE OF PSYCHOLOGIST1/10 USE OF SOCIAL WORKERS

1/10 USE OF PROBATION OFFICERS ½ USE OF CHIROPRACTORS

½ USE OF 911 SERVICES

Communications Strategiesfor Economic Evaluations

CIEE

CIEE

Communicating Results: Intentions…

» Creating, building and sustaining support

» Demonstrating the interconnected of services, decision-making and cost shifting

» Communicating impact in a new way; appeal to different audiences in the language of business

» A better and more comprehensive description of “Community Benefit”

CIEE

Communicating Results: Opportunities…

» Meaningful data for communities and other service organizations; collaboration possibilities

» “Hardens” softer services / program relevance during difficult economic circumstances – the business case

» Framing the Economic Question FocusesProgramming on results / efficiency questions. E.g. “maximizing benefits”

CIEE

Communicating Results: Content…

» Success starts with the quality of content, data and sources used

» Use transparent and reasonable assumptions» Be tight on effectiveness; causality; and

attribution» Ensure relevance of viewpoint to audience» Show it all: Table your results if appropriate» Know your underlying equity statements

Most of all: don’t overstate your results!

CIEE

Communications: Limitations

» Professional and internal resistance to share information

» Re-stigmatizing populations (sub-sets; or with particular risk characteristics) as expensive

» Ongoing Dilution of the language and terminology of “investment”

» Imposition of inappropriate and imposed frameworks

» Expert and Resource Issues

CIEE

Communications: Internal Messages

» EE doesn’t change our Mission – but helps to demonstrate and clarify its impact

» EE can show there are real costs associated with “doing nothing”

» EE demonstrates how we impact other organizations / systems

» EE can create new allies – including some non-traditional partners

CIEE

Communications: Internal Messages

» EE are a tool: a method that helps us put resources where we have the greatest impact – frees up resources for reallocation

» Cost-effectiveness/ Cost benefit analysis will be one – but never the only – consideration of what we do

» Some things we do – will always be driven by our Mission

CIEE

Some closing thoughts

» Economic Evaluations: All about the relative relationship between costs and benefits

» Effectiveness, causality, and measurable outcomes – an a priori requirement for any EE

» Portability of ratios’ from research problematic (population, duration / intensity, measures)

» SROI – understand the method, its use and limitations

» Economic Evaluations: Powerful tool if we use it for good instead of evil

CIEE

Thanks for listening… your turn.

QUESTIONS…..

COMMENTS….

DISCUSSION…..