Introduction to Cost Evaluation for Child Welfare Programs

James Bell, MAJames Bell Associates, Inc.

June 27, 2011

Why Evaluate Costs?Why Evaluate Costs?

• Cost evaluation informs child Cost evaluation informs child welfare program management welfare program management and policy decision making.and policy decision making.

• For example, future adopters of For example, future adopters of

an evidence-based service model an evidence-based service model want to know the cost want to know the cost implications of implementing a implications of implementing a new program. new program.

22

Today’s AimToday’s Aim

• Increase attendees’ knowledge Increase attendees’ knowledge of cost evaluation terminology of cost evaluation terminology and methods.and methods.

• Increase attendees’ practical Increase attendees’ practical ability to design and conduct a ability to design and conduct a cost evaluation of a child cost evaluation of a child welfare program.welfare program.

33

Today’s AgendaToday’s Agenda

• Basic cost evaluation terms and Basic cost evaluation terms and methods.methods.

• Collection of program-level and Collection of program-level and case-level service utilization & cost case-level service utilization & cost data. data.

• Analysis, interpretation, & reporting Analysis, interpretation, & reporting of program-level and case-level cost of program-level and case-level cost data. data.

• Questions & open discussion.Questions & open discussion.44

Basic Terms: Cost and Basic Terms: Cost and Cost ComponentsCost Components

• Cost is the dollar amount spent Cost is the dollar amount spent providing a service.providing a service.

• Total cost is the sum of all direct Total cost is the sum of all direct costs and indirect costs.costs and indirect costs.

• Other types of cost include time Other types of cost include time cost, social cost, opportunity cost, cost, social cost, opportunity cost, etc.etc.

55

• Direct costs are those costs that can Direct costs are those costs that can be assigned to a particular case or be assigned to a particular case or program; for example, the monetary program; for example, the monetary value of a case worker’s person-time value of a case worker’s person-time used to counsel a client.used to counsel a client.

• Indirect costs are costs that are not Indirect costs are costs that are not identified specifically with a particular identified specifically with a particular case or program; for example, the case or program; for example, the monetary value of a case worker’s monetary value of a case worker’s fringe benefits or the cost of facilities.fringe benefits or the cost of facilities.

Basic Terms: Cost and Cost Basic Terms: Cost and Cost Components (cont.)Components (cont.)

66

• Both direct and indirect costs are Both direct and indirect costs are comprised of multiple discreet cost comprised of multiple discreet cost categories, such as labor, facilities, categories, such as labor, facilities, equipment, supplies, equipment, etc.equipment, supplies, equipment, etc.

• These items are recorded as line items in These items are recorded as line items in budgets or accounting/financial reports.budgets or accounting/financial reports.

• For most service programs, labor costs For most service programs, labor costs drive total cost and are the cornerstone of drive total cost and are the cornerstone of analyses.analyses.

Basic Terms: Cost and Cost Basic Terms: Cost and Cost Components (cont.)Components (cont.)

77

• Describe/analyze program costs using Describe/analyze program costs using aggregate data that distinguish cost aggregate data that distinguish cost categories by type of programmatic activity categories by type of programmatic activity (or phases of program maturation).(or phases of program maturation).

• Describe/analyze program costs using case-Describe/analyze program costs using case-level data that distinguish cost categories level data that distinguish cost categories by type of programmatic activity for each by type of programmatic activity for each case.case.

• Describe/analyze program costs using both Describe/analyze program costs using both aggregate and case-level data. aggregate and case-level data.

Basic Cost Evaluation Basic Cost Evaluation MethodsMethods

88

• Cost-effectiveness analysis compares Cost-effectiveness analysis compares 2 or more interventions on costs 2 or more interventions on costs andand effectiveness.effectiveness.

• Effectiveness is measured in client Effectiveness is measured in client level outcomes (e.g., child level outcomes (e.g., child maltreatment events).maltreatment events).

• Result: cost per unit of effect (e.g., Result: cost per unit of effect (e.g., cost per prevented child cost per prevented child maltreatment.)maltreatment.)

Advanced Cost Advanced Cost Analysis MethodsAnalysis Methods

99

• Cost Utility Analysis is like cost Cost Utility Analysis is like cost effectiveness analysis, except effectiveness analysis, except effectiveness is measured as effectiveness is measured as length of life (e.g., survival) or length of life (e.g., survival) or quality of life such as, “cost per quality of life such as, “cost per quality-adjusted life year (QALY) quality-adjusted life year (QALY) saved.”saved.”

Advanced Cost Advanced Cost Analysis Methods Analysis Methods

(cont.)(cont.)

1010

• Cost-Benefit Analysis compares 2 Cost-Benefit Analysis compares 2 or more interventions on costs or more interventions on costs and monetized benefits.and monetized benefits.

• Benefits are expressed in Benefits are expressed in monetary units, which allows for monetary units, which allows for easier comparison of results easier comparison of results across programs with different across programs with different types of effects.types of effects.

Advanced Cost Advanced Cost Analysis Methods Analysis Methods

(cont.)(cont.)

1111

• Return on investment analysis is Return on investment analysis is like cost benefit analysis, except like cost benefit analysis, except monetized benefits are monetized benefits are aggregated over an extended aggregated over an extended period and costs include all period and costs include all phases of program development, phases of program development, implementation and operation.implementation and operation.

Advanced Cost Advanced Cost Analysis Methods Analysis Methods

(cont.)(cont.)

1212

DATA COLLECTIONDATA COLLECTION

Phase 1Phase 1

1313

• More informative to decision-making More informative to decision-making than budget reports or accounting than budget reports or accounting system output.system output.

• Less expensive and burdensome than Less expensive and burdensome than case-level cost analysis.case-level cost analysis.

• Complements case-level cost analysis Complements case-level cost analysis and other more advanced types of and other more advanced types of cost analysis.cost analysis.

Why Collect Why Collect Categorized Categorized

Aggregate Data?Aggregate Data?

1414

• The amounts in each budgeting and The amounts in each budgeting and accounting line item are re-distributed accounting line item are re-distributed into programmatically relevant into programmatically relevant categories usually based on post-hoc categories usually based on post-hoc estimates of the use of labor estimates of the use of labor (personnel costs) in a preceding (personnel costs) in a preceding period.period.

• For example, personnel (labor) costs For example, personnel (labor) costs associated with delivering services to associated with delivering services to clients can be divided between clients can be divided between activities “directly involving a client” activities “directly involving a client” and activities conducted “on behalf of and activities conducted “on behalf of a client” a client”

Categorizing Categorizing Aggregate Program Aggregate Program

Cost DataCost Data

1515

• These two board categories of client-related These two board categories of client-related personnel costs can be further personnel costs can be further disaggregated into subcategories; for disaggregated into subcategories; for example, distinguishing between “in-take example, distinguishing between “in-take assessment” and “parenting skills assessment” and “parenting skills enhancement counseling.”enhancement counseling.”

• Labor costs for administration—personnel Labor costs for administration—personnel not providing front-line services--can be not providing front-line services--can be distinguished between client-related and distinguished between client-related and management-related administrative management-related administrative activities.activities.

Categorizing Aggregate Categorizing Aggregate Program Cost Data Program Cost Data

(cont.)(cont.)

1616

Service Activities Conducted With a Client

Service Activities Conducted on Behalf of a Client

· Advocate with client · Advocate without client

· Assess · Clinical documentation

·   Counseling/ support ·   Research

·   Provide ·   Consult/ collaborate

·   Plan ·   Locate resources

·   Refer ·   Risk management meeting

·   Teach ·   Schedule

·   Schedule ·   Staff travel

·   Transport

·   Contact attempt

Programmatic Cost Analysis: Programmatic Cost Analysis: Personnel Data Elements Personnel Data Elements

(Family Connections) (Family Connections)

1717

Administrative Activities Related to Client Activities

Administrative Activities Related to Management Activities

· Give supervision · Referral screening

·   Receive supervision ·   Outreach/ marketing

·   Team meeting/ Group consultation ·   Writing

·   Provide training ·   MIS entry

·   Attend training ·   Staff organizations meetings/ committees

·    Mentor program technical assistance

·   Other administrative activities

·    Professional/ community committees

·    Organization committees

·    Other administrative activities

Programmatic Cost Analysis: Programmatic Cost Analysis: Personnel Data Elements (Family Personnel Data Elements (Family

Connections) Connections)

1818

• Examine cross-case cost variation.Examine cross-case cost variation.

• Increase measurement precision inside the Increase measurement precision inside the “black box”. “black box”.

• Enable analysis of outcomes in relation to Enable analysis of outcomes in relation to amounts or types of services received amounts or types of services received (dose-intensity/type analysis.)(dose-intensity/type analysis.)

• Improve ability to conduct multivariate Improve ability to conduct multivariate statistical analysis; especially if intent-to-statistical analysis; especially if intent-to-treat analysis fails in a randomized study.treat analysis fails in a randomized study.

Why Collect Case-Level Why Collect Case-Level Data?Data?

1919

• Case-related staff activitiesCase-related staff activities– Service activities with a client Service activities with a client (face-to-(face-to-

face or phone/email)face or phone/email)

– Service activities on behalf of a Service activities on behalf of a client client (preparation for or following interaction (preparation for or following interaction with or on behalf of client)with or on behalf of client)

• Activities required to operate the Activities required to operate the program, but not related to individual program, but not related to individual clients.clients.

• Administrative activities.Administrative activities.

Identify and Define Identify and Define Program Activity Program Activity

CategoriesCategories

2020

• Relies on counts of the amount of staff time (labor) Relies on counts of the amount of staff time (labor) devoted to each activity by each provider involved devoted to each activity by each provider involved with a case.with a case.

• Multiplying the amount(s) of staff time by the Multiplying the amount(s) of staff time by the “loaded” hourly labor rate (monetary value/dollars) “loaded” hourly labor rate (monetary value/dollars) yields “cost per activity.”yields “cost per activity.”

• Sum of per activity costs yields “cost per case.”Sum of per activity costs yields “cost per case.”

• Loaded hourly labor rate includes base Loaded hourly labor rate includes base compensation (salary), fringe benefits, and compensation (salary), fringe benefits, and overhead expenses such as facilities, equipment, overhead expenses such as facilities, equipment, and general administrative support not assigned to and general administrative support not assigned to a specific case.a specific case.

Building Case-Level Building Case-Level CostsCosts

2121

• Invaluable feedback about identifying Invaluable feedback about identifying activity categories and defining/populating activity categories and defining/populating categories.categories.

• Problem-solve efficient data collection Problem-solve efficient data collection processes.processes.

• Sets foundation for assuring data quality.Sets foundation for assuring data quality.

• Builds commitment and buy-in.Builds commitment and buy-in.

• Helps to increase the validity of the Helps to increase the validity of the interpretation of findings.interpretation of findings.

Involve Key Programmatic Involve Key Programmatic Staff, Including Frontline Staff, Including Frontline

CaseworkersCaseworkers

2222

• What is the period of time during which cost What is the period of time during which cost (and outcome) data are collected?(and outcome) data are collected?

• How long will data be collected?How long will data be collected?– Pre-Implementation/Planning Phase?Pre-Implementation/Planning Phase?– Seasonal component of intervention?Seasonal component of intervention?– Length of time between startup and full capacity?Length of time between startup and full capacity?– Full intervention period or sampling of Full intervention period or sampling of

intervention period?intervention period?

• The time horizon has possible implications The time horizon has possible implications (e.g., need for a discount/depreciation (e.g., need for a discount/depreciation rate).rate).

Determine the Determine the Measurement TimeframeMeasurement Timeframe

2323

• PersonnelPersonnel• Determine how staff time will be trackedDetermine how staff time will be tracked

– FTE (including sick, vacation, and other non-productive FTE (including sick, vacation, and other non-productive time)time)

– Time spent working on projectTime spent working on project

• Non-personnelNon-personnel• Space and utilitiesSpace and utilities• TravelTravel• Consumable and non-consumable supplies and Consumable and non-consumable supplies and

equipmentequipment

• Possible to include donated resourcesPossible to include donated resources• Volunteer time and servicesVolunteer time and services• Donated facilities, equipment, suppliesDonated facilities, equipment, supplies

Identify and Define CostsIdentify and Define Costs

2424

• Dates that client started and ended Dates that client started and ended participation in programparticipation in program

• Demographic characteristicsDemographic characteristics– GenderGender– Race and ethnicityRace and ethnicity– Employment statusEmployment status– Marital statusMarital status– Education statusEducation status– Date of birthDate of birth– Number in householdNumber in household

Collect Additional Case-Collect Additional Case-Related InformationRelated Information

2525

• Develop detailed procedures manual for Develop detailed procedures manual for data collection.data collection.

• Provide initial and ongoing training (data Provide initial and ongoing training (data collection procedures and data collection collection procedures and data collection form/system.)form/system.)

• Conduct a pilot trial period.Conduct a pilot trial period. • Develop data collection supervision and Develop data collection supervision and

support systems (e.g. help line for data support systems (e.g. help line for data collectors to submit questions.)collectors to submit questions.)

Prepare StaffPrepare Staff

2626

• Establish rules for coding activities and Establish rules for coding activities and allocating staff time increments on allocating staff time increments on activities activities (minutes/fractions of hours.) (minutes/fractions of hours.)

• Develop a activity tracking form/database Develop a activity tracking form/database ((time used, staff ID, client ID, activity code, time used, staff ID, client ID, activity code, location.)location.)

• Develop a reporting planDevelop a reporting plan– How frequently should staff submit activity tracking How frequently should staff submit activity tracking

forms?forms?

Data Collection Data Collection PreparationPreparation

2727

• Establish rules for coding activities and Establish rules for coding activities and allocating staff time increments on allocating staff time increments on activities activities (minutes/fractions of hours.) (minutes/fractions of hours.)

• Develop a activity tracking form/database Develop a activity tracking form/database ((time used, staff ID, client ID, activity code, time used, staff ID, client ID, activity code, location.)location.)

• Develop a reporting planDevelop a reporting plan– How frequently should staff submit activity tracking How frequently should staff submit activity tracking

forms?forms?

Example: Data Example: Data Collection FormCollection Form

2828

• Develop data entry processDevelop data entry process• Complete paper form and:Complete paper form and:

– Key-enter data using database Key-enter data using database screensscreens

– Scan documents into databaseScan documents into database• Direct data entry using agency Direct data entry using agency

workstationsworkstations• Data entry through internet Data entry through internet

connectionsconnections• Component of existing database or Component of existing database or

stand-alone database?stand-alone database?

Data Repository Data Repository PreparationPreparation

2929

Example: DatabaseExample: Database

Staff

Case #

DATELoc of

Visit

Mode of Visit (1=Face-to-

face, 2=Phone, 3=Written, 4=Other)

Advocate WITH Client

AssessCounseling/Support

Schedule

Filene 1000 12/4/2006 HOME FACE TO FACE   120  20   

Filene 1001 12/4/2006 OFFICE PHONE        5

Filene 1230 12/4/2006 COMM FACE TO FACE 120    

Filene 1009 12/8/2006 HOME FACE TO FACE   60 20 5 

3030

• Develop systems and procedures for Develop systems and procedures for data quality assurance:data quality assurance:

– Data entry screens with built-in Data entry screens with built-in completeness and log-error checkscompleteness and log-error checks

– Additional logic-error checks across Additional logic-error checks across cases cases

– Monitor cross-provider consistencyMonitor cross-provider consistency

– Compare service use records with Compare service use records with case/clinical notes for completeness case/clinical notes for completeness (all clients in day) and accuracy (for (all clients in day) and accuracy (for all services and duration)all services and duration)

Data Quality AssuranceData Quality Assurance

3131

• Ensure data collection completeness:Ensure data collection completeness:– To reduce invalid estimates, assign To reduce invalid estimates, assign

daily deadlinesdaily deadlines

– Conduct ongoing supervisory checks Conduct ongoing supervisory checks of dataof data

– Examine data monthly (so not Examine data monthly (so not forgotten)forgotten)

– Reward staff with highest timely Reward staff with highest timely completed ratescompleted rates

Data Quality Assurance Data Quality Assurance (cont.)(cont.)

3232

• Staff turnoverStaff turnover

• Data collector driftData collector drift

• Unexpected influences of local Unexpected influences of local events/contextual factorsevents/contextual factors

• Multiple staff and sites/agency Multiple staff and sites/agency locations collecting datalocations collecting data

Common Challenges in Common Challenges in Case-Level Data Case-Level Data

CollectionCollection

3333

• These costs can not be assigned to a These costs can not be assigned to a specific case (fringe benefits, space specific case (fringe benefits, space and other forms of infrastructure and other forms of infrastructure support and labor associated with support and labor associated with functions such as payroll, invoicing, functions such as payroll, invoicing, accounts receivable, etc.).accounts receivable, etc.).

• Two approaches: 1)use established, Two approaches: 1)use established, existing rates, or 2)create rates for existing rates, or 2)create rates for fringe, overhead/general fringe, overhead/general administration. administration.

Accounting for Overhead Accounting for Overhead & Administrative Costs& Administrative Costs

3434

Analylsis & ReportingAnalylsis & Reporting

Phase 2Phase 2

3535

• Who is the intended audience?Who is the intended audience?– Society?Society?– Policy makers?Policy makers?– Funding agency?Funding agency?– Agency implementing the intervention?Agency implementing the intervention?– Other agencies considering implementing the Other agencies considering implementing the

intervention?intervention?

• Intended audience determines which type Intended audience determines which type of economic evaluation is needed and of economic evaluation is needed and which costs (and benefits) to examine.which costs (and benefits) to examine.

Determine the Determine the Measurement/Analysis Measurement/Analysis

PerspectivePerspective

3636

Example: Total Example: Total Programmatic Costs by Programmatic Costs by

Site and Year Site and Year

Total Costs per Site

$0

$50,000

$100,000

$150,000

$200,000

$250,000

$300,000

$350,000

$400,000

$450,000

$500,000

A B C D E F G H

Pre- Imp Yr

Yr 1 Imp

Yr 3 Imp

3737

• A sequence of progressively more A sequence of progressively more sophisticated case-level cost analyses is sophisticated case-level cost analyses is conducted building from univariate conducted building from univariate analyses to multivariate analyses.analyses to multivariate analyses.

• A seminal analyses focuses on the A seminal analyses focuses on the

relationship between per-case cost and relationship between per-case cost and case outcomes using cost and outcome case outcomes using cost and outcome data and applying multivariate statistical data and applying multivariate statistical analysis techniques such as repeated analysis techniques such as repeated ANOVA, hierarchical linear modeling, or ANOVA, hierarchical linear modeling, or general equation modeling.general equation modeling.

Case-Level Cost AnalysisCase-Level Cost Analysis

3838

• Encourage sites to be involved in all phases based Encourage sites to be involved in all phases based on their interest-each site should receive on their interest-each site should receive numerous interim and final products from these numerous interim and final products from these analyses. analyses.

• The privacy of individual sites’ data and analyses The privacy of individual sites’ data and analyses should be protected according to agreed-upon should be protected according to agreed-upon rules for data sharing (though it should be rules for data sharing (though it should be understood that the case-level cost data analysis is understood that the case-level cost data analysis is an open, transparent, collaborative multi-site an open, transparent, collaborative multi-site endeavor.)endeavor.)

• Develop rules with regard to publications, access Develop rules with regard to publications, access to data and statistical analysis programming code, to data and statistical analysis programming code, and other shared issues.and other shared issues.

Special Considerations on Special Considerations on Conducting Multi-Site Cost Conducting Multi-Site Cost

AnalysisAnalysis

3939

• Develop and collaboratively Develop and collaboratively review output reports (graphics review output reports (graphics and printouts) on service activity and printouts) on service activity frequency distributions. frequency distributions.

• Reports should describe the types Reports should describe the types and amounts of labor expended and amounts of labor expended on each case during the on each case during the observation period.observation period.

Conduct Service Conduct Service Utilization AnalysesUtilization Analyses

4040

Example: Total Minutes Example: Total Minutes by Category for One Yearby Category for One Year

4141

Intervention vs. Control Intervention vs. Control by Category in Minutes by Category in Minutes

(One Year)(One Year)

4242

Example: Minutes per Example: Minutes per Direct Activity (One Direct Activity (One

Year)Year)

4343

Example: Minutes per Example: Minutes per Direct Activity per Staff ID Direct Activity per Staff ID

for Case 1for Case 1

4444

Example Display of Service Example Display of Service Duration and Intensity (One Duration and Intensity (One

Year)Year)

4545

• Client characteristic variables enable Client characteristic variables enable analyses that compare service analyses that compare service utilization and cost patterns across utilization and cost patterns across subgroups defined by client subgroups defined by client characteristics.characteristics.

• Staff characteristic variables enable Staff characteristic variables enable

analyses that compare service analyses that compare service provision and cost patterns across provision and cost patterns across subgroups defined by staff subgroups defined by staff characteristics.characteristics.

Obtain and Merge Client Obtain and Merge Client & Program Staff & Program Staff

Characteristics DataCharacteristics Data

4646

Example: Staff Example: Staff CharacteristicsCharacteristics

Staff ID Gender Race/Ethn Education Degree

Years Exp. in SW Field Position Title

A F White Masters MSW, LMSW 16-20 Program Manager

B F White Masters MSW, LCSW 16-20Program Coordinator

C F White Masters M.Ed Counseling, LPC, LMFT 2-5 Staff Clinician

D F WhiteCollege

Grad BS Psychology 2-5 Case Manager

E FBlack/

Hispanic Masters MS Counseling, LPC Intern 2-5 Staff Clinician

F F Black/AACollege

Grad BSW, LBSW > 20 Case Manager

G F White MastersMA Counseling, LPC, NCC, RPT 2-5 Staff Clinician

H F White Masters MSW, LMSW, MPH 2-5 Process Evaluator

I F Other Masters MA Sociology 11-15 Outcome Evaluator

J F White Ph.D DrPH Behavioral Health 11-15Evaluation Supervisor

4747

• Develop staff member/class unit Develop staff member/class unit cost factors or hourly labor rates cost factors or hourly labor rates and overhead rates expressed in and overhead rates expressed in dollarsdollars

– Staff compensationStaff compensation– Fringe benefitsFringe benefits– Overhead expensesOverhead expenses

Establish Unit Cost Establish Unit Cost FactorsFactors

4848

Example: Staff Hourly Example: Staff Hourly Labor & Labor &

Fringe RatesFringe Rates

Staff IDBase Salary A

Hourly Base Salary B

Hourly Avg. Annual Base Salary

Empl. Status

Fringe Rate

Annual Hourly Total Comp.

A 30.87 “” 30.87 FT 26.8% 39.14

B 24.04 24.76 24.40 FT 26.8% 30.94

C 17.42 18.29 17.86 FT 26.8% 22.64

D 14.17 “” 14.17 FT 26.8% 17.97

E 17.25 “” 17.25 FT 26.8% 21.87

F 15.24 15.70 15.46 PT(50%) 26.8% 19.61

G 16.29 “” 16.29 FT 26.8% 20.66

H 0 19.07 19.07 FT 26.8% 24.18

I 20.83 21.45 21.14 PT(50%) 26.8% 26.81

J 24.28 “” 24.28 FT 26.8% 30.79 4949

• Descriptive Cost Analyses Descriptive Cost Analyses – These analyses rely on time-based service These analyses rely on time-based service

utilization variables and dollar-based cost utilization variables and dollar-based cost variables. The first activity is to calculate total variables. The first activity is to calculate total cost per case using the unit cost factors cost per case using the unit cost factors developed earlier. This enables both a summary developed earlier. This enables both a summary of costs for all cases and cross-case comparisonsof costs for all cases and cross-case comparisons

• Conduct Multivariate Cost Analyses Conduct Multivariate Cost Analyses – These analyses treat case-level cost as the These analyses treat case-level cost as the

primary dependent variable and incorporate primary dependent variable and incorporate variables, such as client characteristics and variables, such as client characteristics and service utilization patterns, to investigate service utilization patterns, to investigate predictors of cost variation across cases.predictors of cost variation across cases.

Remaining Analysis Remaining Analysis StepsSteps

5050

• Interpretation of dataInterpretation of data– Include program staff in discussion Include program staff in discussion

across all stages of analysesacross all stages of analyses

• ReportingReporting– Graphic presentationGraphic presentation– Document and/or present findings in Document and/or present findings in

language that providers and language that providers and policymakers can understand (i.e., policymakers can understand (i.e., no economic jargon)no economic jargon)

Interpretation & Interpretation & ReportingReporting

5151

Open DiscussionOpen Discussion

5252

• Corso, P. & Filene, JF. (in press). Programmatic cost analysis Corso, P. & Filene, JF. (in press). Programmatic cost analysis of interventions to prevent child maltreatment.of interventions to prevent child maltreatment.

• Corso, P.S., & Lutzker, J.R. (2006). The need for economic Corso, P.S., & Lutzker, J.R. (2006). The need for economic analysis in research on child maltreatment. analysis in research on child maltreatment. Child Abuse & Child Abuse & Neglect, 30Neglect, 30, 727-738., 727-738.

• Drummond, M.F., O’Brien, B., Stoddart, G.L., & Torrance, Drummond, M.F., O’Brien, B., Stoddart, G.L., & Torrance, G.W. (1997). G.W. (1997). Methods for the economic evaluation of health Methods for the economic evaluation of health care programscare programs (2 (2ndnd ed.). Oxford: Oxford University Press. ed.). Oxford: Oxford University Press.

• Foster, E.M., Johnson-Shelton, D., & Taylor, T.K. (2007). Foster, E.M., Johnson-Shelton, D., & Taylor, T.K. (2007). Measuring time costs in interventions designed to reduce Measuring time costs in interventions designed to reduce behavior problems among children and youth. behavior problems among children and youth. American American Journal of Community PsychologyJournal of Community Psychology, 40, 64-81., 40, 64-81.

• Foster, E.M., & Jones, D. (2006). Can a costly intervention be Foster, E.M., & Jones, D. (2006). Can a costly intervention be cost-effective? An analysis of violence prevention. cost-effective? An analysis of violence prevention. Archives Archives of General Psychiatryof General Psychiatry, 63, 1284-1291., 63, 1284-1291.

• Foster, E.M., Porter, M.M., Ayers, T.S., Kaplan, D.L., & Foster, E.M., Porter, M.M., Ayers, T.S., Kaplan, D.L., & Sandler, I. (2007). Estimating the costs of preventative Sandler, I. (2007). Estimating the costs of preventative interventions. interventions. Evaluation ReviewEvaluation Review, 31, 261-286., 31, 261-286.

ReferencesReferences

5353

• Haddix, A.C., Teutsch, S.M., & Corso, P.S. (Eds). (2003). Haddix, A.C., Teutsch, S.M., & Corso, P.S. (Eds). (2003). Prevention effectiveness: A guide to decision analysis and Prevention effectiveness: A guide to decision analysis and economic evaluation.economic evaluation. New York, NY: Oxford University Press. New York, NY: Oxford University Press.

• Herman, P.H., Avery, D.J., Schemp, C.S., & Walsh, M.E. (2009) Herman, P.H., Avery, D.J., Schemp, C.S., & Walsh, M.E. (2009) Are cost-inclusive evaluations worth the effort? Are cost-inclusive evaluations worth the effort? Evaluation Evaluation and Program Planning, 32, 55-61and Program Planning, 32, 55-61..

• Hornick, J.P., Paetsch, J.J., & Bertrand, L.D. (2002). Hornick, J.P., Paetsch, J.J., & Bertrand, L.D. (2002). A manual A manual on conducting economic analysis of crime prevention on conducting economic analysis of crime prevention programsprograms. National Crime Prevention Center.. National Crime Prevention Center.

• Levin, H.M., & McEwan, P.J. (2001). Levin, H.M., & McEwan, P.J. (2001). Cost-effectiveness Cost-effectiveness analysisanalysis (2 (2ndnd ed.). Thousand Oaks, CA: Sage Publications. ed.). Thousand Oaks, CA: Sage Publications.

• NIDA ManualNIDA Manual• Yates, B.T. (2009). Cost-inclusive evaluation: A banquet of Yates, B.T. (2009). Cost-inclusive evaluation: A banquet of

approaches for including costs, benefits, and cost-approaches for including costs, benefits, and cost-effectiveness and cost-benefit analyses in your next effectiveness and cost-benefit analyses in your next evaluation. evaluation. Evaluation and Program Planning, Evaluation and Program Planning, 32, 52-54.32, 52-54.

References (cont.)References (cont.)

5454

Contact Information:Contact Information:

James BellJames BellBell@jbassoc.com

703-528-3230703-528-3230

Follow-up Questions?Follow-up Questions?

5555