Deana M. Crumbling, M.S. Technology Innovation Office U.S. Environmental Protection Agency

Deana M. Crumbling, M.S.Deana M. Crumbling, M.S.Technology Innovation OfficeTechnology Innovation Office

U.S. Environmental Protection AgencyU.S. Environmental Protection AgencyWashington, D.C.Washington, D.C.(703) 603-0643(703) 603-0643

[email protected]@epa.gov

Northeast States’ Improving the Quality of Site Northeast States’ Improving the Quality of Site Characterization ConferenceCharacterization Conference

June 4 and 6, 2002June 4 and 6, 2002

What Could Be Wrong with the What Could Be Wrong with the Traditional Approach?Traditional Approach?

2

USEPA USEPA Technology Innovation OfficeTechnology Innovation Office

Advocates for better technologies and Advocates for better technologies and strategies to clean up contaminated sites:strategies to clean up contaminated sites:– Site investigation/characterizationSite investigation/characterization– Site remediation Site remediation – Monitoring during or after remedial actionMonitoring during or after remedial action

Acts as an Acts as an agent for change agent for change – Disseminates others’ good ideasDisseminates others’ good ideas

Cleanup Information Website: http://cluin.orgCleanup Information Website: http://cluin.org

3

DataQuality

Remedial systems optimization

Decision quality/uncertainty

mgt

Sound

Science

PBMS

DLQL

Dynamic Work Plans

Field Analytical Methods

DQOs

Budgets

NELAC

QA

Risk Assessment

education

States

Enforcement

contracting

Decision

Theory

Precautionary

principle

political &economic

constraints

Innovation

SAPs

QAPPs

Lab Certification

4

Putting all the Pieces Together:

Manage Decision Uncertainty

5

Take-Home Message # 1Take-Home Message # 1

UsingUsingSOUND SCIENCESOUND SCIENCE

in the cleanup of contaminated sites means that thein the cleanup of contaminated sites means that thethe scale of data generation and interpretation the scale of data generation and interpretation

must closely “match”must closely “match” the scale of project decisions the scale of project decisions being based on that data.being based on that data.

Sound science also means managing Sound science also means managing uncertainty uncertainty since since

an an exactexact match usually is not feasible. match usually is not feasible.

The current environmental data quality model is The current environmental data quality model is inadequate to ensure that this matching occurs.inadequate to ensure that this matching occurs.

6

Take-Home Message # 2Take-Home Message # 2

The Triad Approach uses innovative data generation The Triad Approach uses innovative data generation and interpretation tools to make scientific and interpretation tools to make scientific defensibility cost-effective for contaminated site mgt defensibility cost-effective for contaminated site mgt

Triad ApproachTriad Approach = Integrates = Integrates systematic project systematic project planningplanning, , dynamic work plansdynamic work plans, and , and real-time real-time analysisanalysis to to time & costs time & costs andand decision certainty decision certainty

Theme for the Triad Approach = Explicitly identify Theme for the Triad Approach = Explicitly identify and manage the largest sources of decision error, and manage the largest sources of decision error, especially the especially the samplingsampling representativeness of datarepresentativeness of data

7

Data Quality Data Quality as a Tool to Achieve as a Tool to Achieve

Decision QualityDecision Quality

8

First Generation Data Quality ModelFirst Generation Data Quality Model The SYSTEM functions as if it believes that…The SYSTEM functions as if it believes that…

ScreeningMethods

ScreeningData

UncertainDecisions

“Definitive”Methods

“Definitive”Data

CertainDecisions

Methods = Data = Decisions

Distinguish: Distinguish: Analytical MethodsAnalytical Methods from from Data Data fromfrom Decisions Decisions

9

First Generation Data Quality ModelFirst Generation Data Quality Model AssumptionsAssumptions

““Data quality” depends on analytical methodsData quality” depends on analytical methods

Using regulator-approved methods ensures “definitive Using regulator-approved methods ensures “definitive data” data”

QC checks that use ideal matrices are representative of QC checks that use ideal matrices are representative of method performance for real-world samplesmethod performance for real-world samples

Laboratory QA is substitutable for project QALaboratory QA is substitutable for project QA

One-size-fits-all methods eliminate the need for One-size-fits-all methods eliminate the need for analytical chemistry expertiseanalytical chemistry expertise

10

Non-Non-Representative Representative

SampleSample

Perfect Perfect Analytical Analytical ChemistryChemistry

++

““BAD” DATABAD” DATA

Distinguish: Distinguish: Analytical QualityAnalytical Quality from from Data QualityData Quality

Data used for Project Decision Making Data used for Project Decision Making is Generated on is Generated on SamplesSamples

11

Distinguishing ConceptsDistinguishing Concepts

Analytical Methods

Overall Data

Quality

Decision Quality

{Manage Uncertainty

in Decision Making

ClarifyAssumptions

Draw Conclusions

Representative Sampling

{Manage Uncertainty

in Data Generation

Data Assessment/ Analytical Integrity

Non-scientific considerations

Method Modifications

MethodSelection

{

AnalyticalQuality

12

So, how should “Data Quality” be So, how should “Data Quality” be defined?defined?

Data Quality = The ability of data to provide Data Quality = The ability of data to provide information that meets user needsinformation that meets user needs --Condensed from USEPA Office of Environmental Information QMP, 2000--Condensed from USEPA Office of Environmental Information QMP, 2000

Users need to make correct decisionsUsers need to make correct decisions

Data quality is a function of data’s…Data quality is a function of data’s…

– ability to ability to representrepresent the “true state” the “true state” in the context of the in the context of the decision to be madedecision to be made

» The decision defines the appropriate scale over which the “true The decision defines the appropriate scale over which the “true state” should be measured (i.e., the scale of data generation)state” should be measured (i.e., the scale of data generation)

– information contentinformation content (including its uncertainty) (including its uncertainty)

13

Second-Generation Data Quality ModelSecond-Generation Data Quality Model Scientific FoundationScientific Foundation

““Data quality” = data’s ability to support decisionsData quality” = data’s ability to support decisions

Anything that compromises data representativeness Anything that compromises data representativeness compromises data qualitycompromises data quality

““Data” representativeness = sampling Data” representativeness = sampling representativeness + analytical representativenessrepresentativeness + analytical representativeness

Project-specific planning: matches scale(s) of data Project-specific planning: matches scale(s) of data generation with scale(s) of decision-making. generation with scale(s) of decision-making.

Technical expertise required to manage sampling and Technical expertise required to manage sampling and analytical uncertaintiesanalytical uncertainties

14

The Data Quality “Chain”The Data Quality “Chain”

SamplingSampling AnalysisAnalysis

Sample Support

DECISIONGoal Making

DECISION

15

#1#1 #2#2 #3#3

The decision driving sample collection: Assess contamination resulting from

atmospheric deposition

Sample Support: Critical to RepresentativenessSample Support: Critical to Representativeness

Sample Volume & Orientation

16

#1#1 #2#2 #3#3




Sample Volume & OrientationSample Volume & Orientation We assume the volume of a sample should have no effect on the concentration of contaminant in that sample.

17

#1#1 #2#2 #3#3



The Nugget Effect

Same Contaminant Mass in Nugget, but

Different Sample Volumes Produce

Different Concentrations

Sample Prep


Sample Volume & Orientation

18

The Data Quality “Chain”The Data Quality “Chain”

SamplingSampling AnalysisAnalysis

Sample Support

Sampling Design

Sample Preservation

Sub-Sampling

Sample Preparation Method(s)

Determinative Method(s)

DECISIONGoal

Result Reporting

Making

DECISION

Extract Cleanup

Method(s)

All links in the Data Quality chain must be intact for Decision Quality to be supported !

e.g., Method 8270

19

Summing UncertaintiesSumming Uncertainties

Uncertainties add according to (aUncertainties add according to (a22 + b + b22 = c = c22))

Ex. 33 X

Total UncertaintyAnalytical Uncertainty

Sampling Uncertainty

Ex. 1

Ex. 21/3 X

Ex. 1

Ex. 3Ex. 2

20

Partitioning Data UncertaintyPartitioning Data Uncertainty

Std Dev Std Dev Sampling Sampling : Std Dev Std Dev Analytical Analytical = SampSamp:AnalAnal RatioRatio

Brownfields Project Example: Brownfields Project Example: Scrap yard site w/ Scrap yard site w/ contaminated soil contaminated soil

Effect of matrix on analytical variability for B(a)PEffect of matrix on analytical variability for B(a)P Using LCS data (no matrix effect) :Using LCS data (no matrix effect) : 6,5206,520 : 4.44.4 = 14641464 : : 11

Using MS/MSD data (matrix incl’d) :Using MS/MSD data (matrix incl’d) : 6,5206,520 : 12.712.7 = 513513 : : 11

Different metals (LCS data used to estimate analytical variability) Different metals (LCS data used to estimate analytical variability)

High spatial variability,High spatial variability, PbPb 32553255 : 33 = 10851085 : : 11Natural background present, AsNatural background present, As 22.422.4 : 77 = 33 : : 11

Total UncertaintyAnalytical Uncertainty

Sampling Uncertainty

aa

bbcc

21

Sample Location ~ 95%

Analytical (between methods) ~ 5%Analytical (between methods) ~ 5%

Example of Variability: Example of Variability: Sample Location vs. Analytical MethodSample Location vs. Analytical Method

39,800 On-site41,400 Lab

500 On-site416 Lab

164 On-site136 Lab

27,800 On-site42,800 Lab

24,400 On-site27,700 Lab

1,280 On-site1,220 Lab

1

27

6 3

45

331 On-site 286 Lab

Marrying Analytical Methods to Make Sound Decisions Marrying Analytical Methods to Make Sound Decisions Involving Heterogeneous MatricesInvolving Heterogeneous Matrices

Costly definitive analytical methods

Cheaper/screening analytical methods

High spatial densityLow DL + analyte specificity

Manages analytical uncertainty= analytical representativeness = analytical quality

Definitive analytical qualityScreening sampling quality

Manages sampling uncertainty= sampling representativeness = sampling quality

Definitive sampling qualityScreening analytical quality





Manages analytical uncertainty

Manages sampling uncertainty

Collaborative Data Sets





Manages analytical uncertainty

Manages sampling uncertainty

Decision Quality Data

Reliable (yet Cost-Effective) Scientifically Defensible Decisions

25

Improve Decision Quality--Manage UncertaintiesImprove Decision Quality--Manage Uncertainties

¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢

¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢

¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢

¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢

$ $ $

$ $ $

$ $ $

$ $ $

FROM THIS

TO THIS

Ex 1Ex 2Ex 3

Fixed Lab Analytical

UncertaintySampling Uncertainty

Ex 1

Sampling Uncertainty Controlled through Increased Density

Field Analytical

DataEx 2

Fixed Lab DataDecreased Sampling

Variability after Removal of Hotspots

Ex 3

Remove hot spots

26

Managing Decision Managing Decision UncertaintyUncertainty

Using Modern ToolsUsing Modern Tools

27

A Systems Approach FrameworkA Systems Approach Framework The Triad ApproachThe Triad Approach

Systematic Project

Planning

Dynamic Work Plan Strategy

Real-time Measurement Technologies

28

Unifying Concept for Unifying Concept for Triad:Triad: Managing Managing

UncertaintyUncertainty

Manage uncertainty about project goalsManage uncertainty about project goals – Identify decision goals with tolerable overall uncertaintyIdentify decision goals with tolerable overall uncertainty– Identify major uncertainties (cause decision error)Identify major uncertainties (cause decision error)– Identify the strategies to manage each major uncertaintyIdentify the strategies to manage each major uncertainty

Manage uncertainty in dataManage uncertainty in data– Sampling uncertainty: Sampling uncertainty: manage sample representativenessmanage sample representativeness – Analytical uncertainty: Analytical uncertainty: especially if field methods are usedespecially if field methods are used

Multidisciplinary expertise criticalMultidisciplinary expertise critical– A TEAMA TEAM is the best way to bring needed knowledge to bear is the best way to bring needed knowledge to bear

Systematic planning is used to proactively…Systematic planning is used to proactively…

Dynamic Work PlansDynamic Work Plans

Real-time decision-making “in the field”Real-time decision-making “in the field”– Evolve CSM in real-timeEvolve CSM in real-time– Implement pre-approved decision tree using senior staffImplement pre-approved decision tree using senior staff– Contingency planning: most seamless activity flow possible to Contingency planning: most seamless activity flow possible to

reach project goals in fewest mobilizationsreach project goals in fewest mobilizations

Real-time decisions need real-time dataReal-time decisions need real-time data– Use off-site lab w/ short turnaround?Use off-site lab w/ short turnaround?

» Use screening analytical methods in fixed lab?Use screening analytical methods in fixed lab?– Use on-site analysis?Use on-site analysis?

» Use mobile lab with conventional equipment?Use mobile lab with conventional equipment?» Use portable kits & instruments?Use portable kits & instruments?

MixAndMatch}

In all cases, must generate data of In all cases, must generate data of knownknown quality quality

30

Generating Real-time Data Using Field MethodsGenerating Real-time Data Using Field Methods Manage Uncertainty through Systematic PlanningManage Uncertainty through Systematic Planning

Need clearly defined data uses—tie to project goalsNeed clearly defined data uses—tie to project goals

Understand dynamic work plan—branch points & work flowUnderstand dynamic work plan—branch points & work flow

Project-specific QA/QC protocols matched to intended data useProject-specific QA/QC protocols matched to intended data use

SelectSelect field analytical field analytical technologies totechnologies to– Support theSupport the dynamic work plan dynamic work plan (greatest source of $$ savings) (greatest source of $$ savings)

– Manage Manage sampling uncertaintysampling uncertainty (improves decision quality) (improves decision quality)

SelectSelect fixed lab fixed lab methods (methods (as neededas needed) to) to– Manage Manage uncertainties in field datauncertainties in field data (just ONE aspect of QC for field data)(just ONE aspect of QC for field data)

– Supply analyte-specific data and/or lower quantitation limits Supply analyte-specific data and/or lower quantitation limits (if needed for regulatory compliance, risk assessment, etc.)(if needed for regulatory compliance, risk assessment, etc.)

31

Sample Representativeness is Key! Sample Representativeness is Key!

Cheaper analyses permit Cheaper analyses permit increased sample densityincreased sample density– New software for statistical/geostatistical decision supportNew software for statistical/geostatistical decision support

» VSP software pkg FREE: VSP software pkg FREE: http://http://dqodqo..pnlpnl..govgov/VSP/index./VSP/index.htmhtm

» SADA software pkg FREE:SADA software pkg FREE: http://www.http://www.tiemtiem..utkutk..eduedu/~/~sadasada//

» FIELDS/SADA software:FIELDS/SADA software: http://www.http://www.epaepa..govgov/region5fields/static/pages/index.html/region5fields/static/pages/index.html

Real-time measurements support Real-time measurements support real-time decision-real-time decision-makingmaking– Rapid feedback for course correction Rapid feedback for course correction smarter sampling smarter sampling

Data Quality: Focus on Data Quality: Focus on overall data uncertainty; overall data uncertainty; analytical uncertaintyanalytical uncertainty usually a small fraction usually a small fraction

FinallyFinally able to address able to address defensibly and affordably!defensibly and affordably!

http://dqo.pnl.gov/VSP/index.htm








http://www.tiem.utk.edu/~sada/









http://www.epa.gov/region5fields/static/pages/index.html





32

Case Study: Wenatchee Tree Fruit SiteCase Study: Wenatchee Tree Fruit Site Pesticide IA kits guide dynamic work plan: remove and Pesticide IA kits guide dynamic work plan: remove and

segregate contaminated soil for disposalsegregate contaminated soil for disposal230 230 IA analysesIA analyses (w/ thorough QC) (w/ thorough QC)

Managed Managed sampling uncertaintysampling uncertainty: : achieved very high confidence achieved very high confidence that all contamination above that all contamination above action levels was located and action levels was located and removedremoved

Clean closure data setClean closure data set– 33 fixed lab samples for analyte-specific pesticide analysis 33 fixed lab samples for analyte-specific pesticide analysis – Demonstrate Demonstrate fullfull compliance with compliance with allall regulatory requirements for regulatory requirements for allall 33 pesticide analytes to >95% statistical confidence33 pesticide analytes to >95% statistical confidence the first timethe first time!!

Projected cost: ~$1.2M; Actual: $589K (Save ~ 50%)Projected cost: ~$1.2M; Actual: $589K (Save ~ 50%) Field work completed: <4 months; single mobilizationField work completed: <4 months; single mobilization

http://cluin.org/char1_edu.cfm#site_charhttp://cluin.org/char1_edu.cfm#site_char

Managed Managed field analyticalfield analytical uncertaintyuncertainty as additional QC on as additional QC on critical samples: confirmed & critical samples: confirmed & perfected field kit action levels)perfected field kit action levels)

++ 29 29 fixed-labfixed-lab samples for 33 analytes samples for 33 analytes

33

Terminology to Link Terminology to Link

Data Quality Data Quality

with with

Decision QualityDecision Quality

34

““Data Quality” TerminologyData Quality” Terminology

Current terminology usage does not focus Current terminology usage does not focus on the goal of decision qualityon the goal of decision quality

Irony: Great focus on the quality of data Irony: Great focus on the quality of data points;points; but overall quality of but overall quality of decisionsdecisions easily unknown. easily unknown.

Current usage does not distinguish Current usage does not distinguish – Methods vs. data vs. decisionsMethods vs. data vs. decisions– The factors that impact each step in the processThe factors that impact each step in the process– Relationships between different aspects of qualityRelationships between different aspects of quality

35

Proposed Clarification of TermsProposed Clarification of TermsQuality AssuranceQuality Assurance

Project QA:Project QA: ID causes of potential intolerable decision errors ID causes of potential intolerable decision errors & the strategies to manage and prevent those decision errors& the strategies to manage and prevent those decision errors

Data QA:Data QA: manage manage bothboth sampling and analytical uncertainties sampling and analytical uncertainties to degree needed to avoid decision errorsto degree needed to avoid decision errors– Analytical representativeness evaluated,Analytical representativeness evaluated, including impact of including impact of

sample/matrix effects on analytical performancesample/matrix effects on analytical performance

– Sample representativeness must be evaluatedSample representativeness must be evaluated

Lab QA:Lab QA: manage technical performance of analytical manage technical performance of analytical instruments, processes, and operators to meet instruments, processes, and operators to meet lab’slab’s QA goals QA goals– Sample/matrix effects on analytical performance may or may not Sample/matrix effects on analytical performance may or may not

be evaluated—depends on contract specifications.be evaluated—depends on contract specifications.

36

Analytical QA TerminologyAnalytical QA Terminology

Demonstration of method applicabilityDemonstration of method applicability– Shows that a particular method, project-specific Shows that a particular method, project-specific

SOP, and selected QC acceptance criteria are SOP, and selected QC acceptance criteria are appropriate for a project-specific application or appropriate for a project-specific application or site-specific matrixsite-specific matrix

Demonstration of proficiencyDemonstration of proficiency– Shows that a particular operator or lab can perform Shows that a particular operator or lab can perform

a method properlya method properly

37

Proposed Clarification of TermsProposed Clarification of TermsData QualityData Quality

Decision quality data*Decision quality data* = = Effective data*Effective data* = data shown = data shown to be effective for decision-makingto be effective for decision-making

Screening quality data*Screening quality data* = some useful information = some useful information provided; but too uncertain to support decision-making provided; but too uncertain to support decision-making alonealone

Collaborative data setsCollaborative data sets = distinct data sets used in = distinct data sets used in concert with each other to co-manage sampling and/or concert with each other to co-manage sampling and/or analytical uncertainties to an acceptable levelanalytical uncertainties to an acceptable level

* Includes sampling uncertainty. Nature of the analytical method irrelevant.* Includes sampling uncertainty. Nature of the analytical method irrelevant.

38

Misleading TerminologyMisleading Terminology

This term falsely implies that:This term falsely implies that:

• All methods run in the field are screening methods;All methods run in the field are screening methods;

• Therefore, all data produced in the field are of Therefore, all data produced in the field are of screening quality.screening quality.

Field Screening

39

““Effective Data”Effective Data” “Decision Quality Data” “Decision Quality Data”

Data ofData of known qualityknown quality

that can be logically demonstrated to bethat can be logically demonstrated to be effective for making the specified decisioneffective for making the specified decision

because both thebecause both the sampling and analytical uncertaintiessampling and analytical uncertainties

are managed to the degree necessary to meet clearlyare managed to the degree necessary to meet clearly defined (and stated) decision confidence goalsdefined (and stated) decision confidence goals

40

The Diffusion of InnovationThe Diffusion of Innovation

““At first people refuse to believe that a strange At first people refuse to believe that a strange new thing can be done, then they begin to new thing can be done, then they begin to hope it can be done—then it is done and all hope it can be done—then it is done and all the world wonders why it was not done the world wonders why it was not done centuries ago.”centuries ago.”

— —Francis Hodges BurnettFrancis Hodges Burnett

Deana M. Crumbling, M.S. Technology Innovation Office U.S. Environmental Protection Agency

Documents

Transcript of Deana M. Crumbling, M.S. Technology Innovation Office U.S. Environmental Protection Agency