Overview of the Final Report and Findings from the Review of Sampling

Methods in Extrapolated New Base-Year Generation Studies

May 11-12, 2004

Types of New Base Year Generation Studies

Nonextrapolated “Actual” diversion data collected from

diversion programs implemented within the jurisdiction

Extrapolated Includes diversion data collected from a sample

of the non-residential businesses and extrapolated to the entire non-residential business sector

Number of New Base-Year Generation Studies

Since April 1998, Board staff have reviewed 184 submitted new base year studies for Board consideration.

143 Nonextrapolated studies

41 Extrapolated studies

Background

Board expressed concerns regarding validity of extrapolated studies

Working group convened

Board staff performed analyses on extrapolation vs. nonextrapolation study methodologies

Extrapolated vs. Nonextrapolated Study Data Comparisons

Generation pounds per person per day was 79% higher for extrapolated methodologies

Diversion rates for extrapolated studies were significantly higher (53%) than nonextrapolated studies (41%).

Average source reduction tonnage in extrapolated studies exceeded nonextrapolated studies by an average of 300%.

Response to Concerns regarding New Base-Year Study Data

Base-Year Modification Certification Request form

Board staff began on-site verification of new base-year generation study data

Board pursued independent, third-party review of extrapolated study methodologies

Scope of Review

Review 20 extrapolated studies.

Only 2 of the 20 studies reviewed submitted sufficient data.

Provide observations and recommendations for improving extrapolated methodologies used.

Impact of poor quality or lack of study data.

Findings

The findings in the final report address the impact of errors relating to: Study design;

Sampling errors; and

Technical data components of nonresidential diversion surveys.

Study Design

Jurisdictions should list assumptions made in the study design and use sample data to assess their validity and appropriateness.

Studies did not include listing of such assumptions.

Jurisdictions did not use the study data to test the validity of assumptions.

Sampling Errors

Accurate data on the selection of samples is needed for Board staff to fully evaluate the sampling methodology.

Lack of documentation regarding sample selection.

Studies employed nonrandom sampling methods.

Technical Data

Lack of documentation regarding nonresponse.

Most jurisdictions provided outlier analysis, but did not address all study design variables.

Lack of access to study data.

Additional Analyses

Case Study of One Extrapolated New Base-Year Generation Study

Review of 35 extrapolated new base-year generation study data Diversion of Top 10 Businesses Compared to All

Surveyed Businesses

Commercial Business diversion Represents Approximately 38% of Total Diversion

25%

38%

37% Non Residential

Business Audits

Res/ADC/Landfill Salvage

Residential

New Base Year Case Study: Impact of the Largest Businesses (333) Audited Relative to the Total Business Audits (approx. 7,600)

0

10

20

30

40

50

60

70

80

90

top 10 top 20 top 30

Business Generators

% o

f D

ive

rsio

n t

on

nag

e fr

om

Bu

sin

esse

s A

ud

its

Diversion of Top 10 Businesses Compared to All Surveyed Businesses

By surveying 10 top businesses:

94% of the jurisdictions captured 50% or more of the total diversion;

60% of the jurisdictions captured 80% or more of the total diversion; and

37% of the jurisdictions captured 90% or more of the total diversion.

Maximizing Efficiencies in New Base-Year Generation Studies

Many of the jurisdictions would have been better off targeting their largest diverters and submitting a nonextrapolated study.

These jurisdictions could have captured most of the industrial/commercial diversion without having to address problems associated with sample selection, nonresponse, outliers, or choosing an estimator.

Future Efforts

Board staff will: Update the Board’s Base-Year Modification

Request Certification form for extrapolated studies.

Update the Board’s Diversion Study Guide to explicitly underscore the importance of documenting all critical aspects of study design, implementation and diversion rate calculation.

Develop a Web-based reporting format.

Future Efforts (continued)

Board staff will continue to

Assist jurisdictions regarding their future new base-year generation studies; and.

Stress the benefits of targeting the largest businesses first to measure the non-residential sector.

Summary

Board staff were unable to perform a complete statistical review for 18 of the 20 studies due to insufficient study data.

Findings support the value of targeting the top 10 businesses.