INDUSTRIAL ECONOMICS, INCORPORATED IEc INDUSTRIAL ECONOMICS, INCORPORATED Measuring Impact of...
-
Upload
elijah-wiggins -
Category
Documents
-
view
223 -
download
2
Transcript of INDUSTRIAL ECONOMICS, INCORPORATED IEc INDUSTRIAL ECONOMICS, INCORPORATED Measuring Impact of...
INDUSTRIAL ECONOMICS, INCORPORATED
IEc
INDUSTRIAL ECONOMICS, INCORPORATED
Measuring Impact of Compliance Assistance on Auto Body Shops using an Experimental & Quasi-Experimental Evaluation Design
Tracy Dyke RedmondSenior Associate
June 23, 2011
INDUSTRIAL ECONOMICS, INCORPORATED
Typology of Evaluation Designs*
Experimental Design
Quasi-Experimental Design
Non-Experimental Design
2
IncreasingStatistical Strength*
*Notnecessarilyoverall evaluation strength
• Pre-test + post-test comparison with statistical matching
• Regression discontinuity• Pre-test + post-test
comparison with judgmental matching
• Pipeline control group• Pre-test + post-test
comparison with delayed baseline
• Pre-test + post-test treatment group with post-test only comparison group
• Post-test only for treatment and comparison group
* Not all possible evaluation designs shownAdapted from Bamberger, Rugh, and Margy: RealWorld Evaluation,Sage Publications 2006.
INDUSTRIAL ECONOMICS, INCORPORATED
“Statistically Valid” Pilot Project for Auto Body Shops
• Pre-test + post-test comparison with statistical matching
• Regression discontinuity• Pre-test + post-test
comparison with judgmental matching
• Pipeline control group• Pre-test + post-test
comparison with delayed baseline
• Pre-test + post-test treatment group with post-test only comparison group
• Post-test only for treatment and comparison group
Treatment and comparison areas selected judgmentally, with subjects randomly selected from within these areas
Pro:•Flexible•Reasonably good estimate of project impact when there are good matching criteriaCon:•Assumes comparison group similar to treatment group and equally willing to participate•Does not assess project implementationAdapted from Bamberger, Rugh, and Margy: RealWorld Evaluation, Sage Publications 2006.
INDUSTRIAL ECONOMICS, INCORPORATED
“Statistically Valid” Pilot Project for Auto Body Shops
• Pre-test + post-test comparison with statistical matching
• Regression discontinuity• Pre-test + post-test
comparison with judgmental matching
• Pipeline control group• Pre-test + post-test
comparison with delayed baseline
• Pre-test + post-test treatment group with post-test only comparison group
• Post-test only for treatment and comparison group
Project implemented in phases: subjects in phase 2 serve as control group for subjects in phase 1
Pro:•Does not require external control group; design relatively inexpensive and easy to useCon:•Assumes phase 1 and 2 groups are similar (which may not be true)•Requires that phase 2 group does not have access to phase 1 treatment (spillover problem)
Adapted from Bamberger, Rugh, and Margy: RealWorld Evaluation, Sage Publications 2006.
INDUSTRIAL ECONOMICS, INCORPORATED
• Goal: test impact of EPA compliance assistance (e.g., workshops, webinars, materials) on auto body shop compliance with air and hazardous waste regulations
• Context: existing hazardous waste regulations, new air regulations coming into effect 2011 (Surface Coating Rule)
• Requirements: • Conduct representative measurement of all regulated
entities, not just voluntary participants
• Do not prevent shops from receiving compliance assistance (or not for very long)
• Also test phone survey validity (that methodology not covered in this presentation)
“Statistically Valid” Pilot Project for Auto Body Shops
INDUSTRIAL ECONOMICS, INCORPORATED
Control and Comparison Groups
• Population = auto body shops located in areas with elevated air toxics risks and subject to the Surface Coating Rule
• Massachusetts selected as study area because EPA Region 1 planned compliance assistance campaign
• Treatment and control group in Massachusetts: randomly assigned, considered equivalent
• Includes auto body shops in eastern MA with elevated risk
• Excludes communities with pre-existing aggressive assistance/enforcement campaigns
• Comparison group selected in Tidewater and Piedmont regions of Virginia on the basis of:
• No expected compliance assistance from EPA or state
• State regulations related to RCRA and air emissions
• Number of shops located in areas of elevated-risk
INDUSTRIAL ECONOMICS, INCORPORATED
“Statistically Valid” Pilot Project Design
MA - A
MA - B
VA
October 2009 –January 2010
March – Early July 2010
ComplianceAssistance (CA):• Mailings• Workshops/ Webinars
On-site surveysFollowed by on-site CA
On-site surveys
Summer 2010-January 2011
CA:• Mailings• Webinars
CA:• Mailings• Webinars
March – Early July 2010
On-site surveysFollowed by on-site CA
On-site surveysFollowed by on-site CA
On-site surveysFollowed by on-site CA
On-site surveysFollowed by on-site CA
INDUSTRIAL ECONOMICS, INCORPORATED
“Statistically Valid” Pilot Project Design
MA - A
MA - B
VA
October 2009 –January 2010
March – Early July 2010
ComplianceAssistance (CA):• Mailings• Workshops/ Webinars
On-site surveysFollowed by on-site CA
On-site surveys
Summer 2010-January 2011
CA:• Mailings• Webinars
CA:• Mailings• Webinars
March – Early July 2010
On-site surveysFollowed by on-site CA
On-site surveysFollowed by on-site CA
On-site surveysFollowed by on-site CA
On-site surveysFollowed by on-site CA
Compare randomly assigned treatment vs. control group in MA
INDUSTRIAL ECONOMICS, INCORPORATED
“Statistically Valid” Pilot Project Design
MA - A
MA - B
VA
October 2009 –January 2010
March – Early July 2010
ComplianceAssistance (CA):• Mailings• Workshops/ Webinars
On-site surveysFollowed by on-site CA
On-site surveys
Summer 2010-January 2011
CA:• Mailings• Webinars
CA:• Mailings• Webinars
March – Early July 2010
On-site surveysFollowed by on-site CA
On-site surveysFollowed by on-site CA
On-site surveysFollowed by on-site CA
On-site surveysFollowed by on-site CA
Compare “difference-in-differences”: Pre-test to Post-test in MA, compared to Pre-test to Post-test in VA
INDUSTRIAL ECONOMICS, INCORPORATED
Methodological Notes and Next Steps
• Among valid shops, response rates between 80 – 85% for both states and both years
• However, list problems (e.g., shops going out of business) led to need for many “backup shops;” increased cost and effort of project
• Evidence of shops not on the list, operating “under the radar” in VA
• Some evidence of spillover problems (e.g., some VA shops may have accessed EPA Region 1 webinars)
• Currently working to analyze 2011 data and develop comparisons; report expected in late 2011
INDUSTRIAL ECONOMICS, INCORPORATED
Thanks to:
• EPA HQ for developing the vision and supporting the project
• EPA Region 1 for identify the list of shops, implementing the assistance, and helping conduct site visits
• EPA Region 3, Virginia, and Massachusetts for participating
• ERG and Ski Fabyanic for conducting hundreds of site visits
• Chris Leggett and Michael Crow for helping develop the methodology and analyzing the data
• The auto body shops, for letting us in the door!