Post on 03-Jan-2016
description
Deana M. Crumbling, M.S.Deana M. Crumbling, M.S.Technology Innovation OfficeTechnology Innovation Office
U.S. Environmental Protection AgencyU.S. Environmental Protection AgencyWashington, D.C.Washington, D.C.(703) 603-0643(703) 603-0643
crumbling.deana@epa.govcrumbling.deana@epa.gov
Northeast States’ Improving the Quality of Site Northeast States’ Improving the Quality of Site Characterization ConferenceCharacterization Conference
June 4 and 6, 2002June 4 and 6, 2002
What Could Be Wrong with the What Could Be Wrong with the Traditional Approach?Traditional Approach?
2
USEPA USEPA Technology Innovation OfficeTechnology Innovation Office
Advocates for better technologies and Advocates for better technologies and strategies to clean up contaminated sites:strategies to clean up contaminated sites:– Site investigation/characterizationSite investigation/characterization– Site remediation Site remediation – Monitoring during or after remedial actionMonitoring during or after remedial action
Acts as an Acts as an agent for change agent for change – Disseminates others’ good ideasDisseminates others’ good ideas
Cleanup Information Website: http://cluin.orgCleanup Information Website: http://cluin.org
3
DataQuality
Remedial systems optimization
Decision quality/uncertainty
mgt
Sound
Science
PBMS
DLQL
Dynamic Work Plans
Field Analytical Methods
DQOs
Budgets
NELAC
QA
Risk Assessment
education
States
Enforcement
contracting
Decision
Theory
Precautionary
principle
political &economic
constraints
Innovation
SAPs
QAPPs
Lab Certification
4
Putting all the Pieces Together:
Manage Decision Uncertainty
5
Take-Home Message # 1Take-Home Message # 1
UsingUsingSOUND SCIENCESOUND SCIENCE
in the cleanup of contaminated sites means that thein the cleanup of contaminated sites means that thethe scale of data generation and interpretation the scale of data generation and interpretation
must closely “match”must closely “match” the scale of project decisions the scale of project decisions being based on that data.being based on that data.
Sound science also means managing Sound science also means managing uncertainty uncertainty since since
an an exactexact match usually is not feasible. match usually is not feasible.
The current environmental data quality model is The current environmental data quality model is inadequate to ensure that this matching occurs.inadequate to ensure that this matching occurs.
6
Take-Home Message # 2Take-Home Message # 2
The Triad Approach uses innovative data generation The Triad Approach uses innovative data generation and interpretation tools to make scientific and interpretation tools to make scientific defensibility cost-effective for contaminated site mgt defensibility cost-effective for contaminated site mgt
Triad ApproachTriad Approach = Integrates = Integrates systematic project systematic project planningplanning, , dynamic work plansdynamic work plans, and , and real-time real-time analysisanalysis to to time & costs time & costs andand decision certainty decision certainty
Theme for the Triad Approach = Explicitly identify Theme for the Triad Approach = Explicitly identify and manage the largest sources of decision error, and manage the largest sources of decision error, especially the especially the samplingsampling representativeness of datarepresentativeness of data
7
Data Quality Data Quality as a Tool to Achieve as a Tool to Achieve
Decision QualityDecision Quality
8
First Generation Data Quality ModelFirst Generation Data Quality Model The SYSTEM functions as if it believes that…The SYSTEM functions as if it believes that…
ScreeningMethods
ScreeningData
UncertainDecisions
“Definitive”Methods
“Definitive”Data
CertainDecisions
Methods = Data = Decisions
Distinguish: Distinguish: Analytical MethodsAnalytical Methods from from Data Data fromfrom Decisions Decisions
9
First Generation Data Quality ModelFirst Generation Data Quality Model AssumptionsAssumptions
““Data quality” depends on analytical methodsData quality” depends on analytical methods
Using regulator-approved methods ensures “definitive Using regulator-approved methods ensures “definitive data” data”
QC checks that use ideal matrices are representative of QC checks that use ideal matrices are representative of method performance for real-world samplesmethod performance for real-world samples
Laboratory QA is substitutable for project QALaboratory QA is substitutable for project QA
One-size-fits-all methods eliminate the need for One-size-fits-all methods eliminate the need for analytical chemistry expertiseanalytical chemistry expertise
10
Non-Non-Representative Representative
SampleSample
Perfect Perfect Analytical Analytical ChemistryChemistry
++
““BAD” DATABAD” DATA
Distinguish: Distinguish: Analytical QualityAnalytical Quality from from Data QualityData Quality
Data used for Project Decision Making Data used for Project Decision Making is Generated on is Generated on SamplesSamples
11
Distinguishing ConceptsDistinguishing Concepts
Analytical Methods
Overall Data
Quality
Decision Quality
{Manage Uncertainty
in Decision Making
ClarifyAssumptions
Draw Conclusions
Representative Sampling
{Manage Uncertainty
in Data Generation
Data Assessment/ Analytical Integrity
Non-scientific considerations
Method Modifications
MethodSelection
{
AnalyticalQuality
12
So, how should “Data Quality” be So, how should “Data Quality” be defined?defined?
Data Quality = The ability of data to provide Data Quality = The ability of data to provide information that meets user needsinformation that meets user needs --Condensed from USEPA Office of Environmental Information QMP, 2000--Condensed from USEPA Office of Environmental Information QMP, 2000
Users need to make correct decisionsUsers need to make correct decisions
Data quality is a function of data’s…Data quality is a function of data’s…
– ability to ability to representrepresent the “true state” the “true state” in the context of the in the context of the decision to be madedecision to be made
» The decision defines the appropriate scale over which the “true The decision defines the appropriate scale over which the “true state” should be measured (i.e., the scale of data generation)state” should be measured (i.e., the scale of data generation)
– information contentinformation content (including its uncertainty) (including its uncertainty)
13
Second-Generation Data Quality ModelSecond-Generation Data Quality Model Scientific FoundationScientific Foundation
““Data quality” = data’s ability to support decisionsData quality” = data’s ability to support decisions
Anything that compromises data representativeness Anything that compromises data representativeness compromises data qualitycompromises data quality
““Data” representativeness = sampling Data” representativeness = sampling representativeness + analytical representativenessrepresentativeness + analytical representativeness
Project-specific planning: matches scale(s) of data Project-specific planning: matches scale(s) of data generation with scale(s) of decision-making. generation with scale(s) of decision-making.
Technical expertise required to manage sampling and Technical expertise required to manage sampling and analytical uncertaintiesanalytical uncertainties
14
The Data Quality “Chain”The Data Quality “Chain”
SamplingSampling AnalysisAnalysis
Sample Support
DECISIONGoal Making
DECISION
15
#1#1 #2#2 #3#3
The decision driving sample collection: Assess contamination resulting from
atmospheric deposition
Sample Support: Critical to RepresentativenessSample Support: Critical to Representativeness
Sample Volume & Orientation
16
#1#1 #2#2 #3#3
The decision driving sample collection: Assess contamination resulting from
atmospheric deposition
Sample Support: Critical to RepresentativenessSample Support: Critical to Representativeness
Sample Volume & OrientationSample Volume & Orientation We assume the volume of a sample should have no effect on the concentration of contaminant in that sample.
17
#1#1 #2#2 #3#3
The decision driving sample collection: Assess contamination resulting from
atmospheric deposition
The Nugget Effect
Same Contaminant Mass in Nugget, but
Different Sample Volumes Produce
Different Concentrations
Sample Prep
Sample Support: Critical to RepresentativenessSample Support: Critical to Representativeness
Sample Volume & Orientation
18
The Data Quality “Chain”The Data Quality “Chain”
SamplingSampling AnalysisAnalysis
Sample Support
Sampling Design
Sample Preservation
Sub-Sampling
Sample Preparation Method(s)
Determinative Method(s)
DECISIONGoal
Result Reporting
Making
DECISION
Extract Cleanup
Method(s)
All links in the Data Quality chain must be intact for Decision Quality to be supported !
e.g., Method 8270
19
Summing UncertaintiesSumming Uncertainties
Uncertainties add according to (aUncertainties add according to (a22 + b + b22 = c = c22))
Ex. 33 X
Total UncertaintyAnalytical Uncertainty
Sampling Uncertainty
Ex. 1
Ex. 21/3 X
Ex. 1
Ex. 3Ex. 2
20
Partitioning Data UncertaintyPartitioning Data Uncertainty
Std Dev Std Dev Sampling Sampling : Std Dev Std Dev Analytical Analytical = SampSamp:AnalAnal RatioRatio
Brownfields Project Example: Brownfields Project Example: Scrap yard site w/ Scrap yard site w/ contaminated soil contaminated soil
Effect of matrix on analytical variability for B(a)PEffect of matrix on analytical variability for B(a)P Using LCS data (no matrix effect) :Using LCS data (no matrix effect) : 6,5206,520 : 4.44.4 = 14641464 : : 11
Using MS/MSD data (matrix incl’d) :Using MS/MSD data (matrix incl’d) : 6,5206,520 : 12.712.7 = 513513 : : 11
Different metals (LCS data used to estimate analytical variability) Different metals (LCS data used to estimate analytical variability)
High spatial variability,High spatial variability, PbPb 32553255 : 33 = 10851085 : : 11Natural background present, AsNatural background present, As 22.422.4 : 77 = 33 : : 11
Total UncertaintyAnalytical Uncertainty
Sampling Uncertainty
aa
bbcc
21
Sample Location ~ 95%
Analytical (between methods) ~ 5%Analytical (between methods) ~ 5%
Example of Variability: Example of Variability: Sample Location vs. Analytical MethodSample Location vs. Analytical Method
39,800 On-site41,400 Lab
500 On-site416 Lab
164 On-site136 Lab
27,800 On-site42,800 Lab
24,400 On-site27,700 Lab
1,280 On-site1,220 Lab
1
27
6 3
45
331 On-site 286 Lab
Marrying Analytical Methods to Make Sound Decisions Marrying Analytical Methods to Make Sound Decisions Involving Heterogeneous MatricesInvolving Heterogeneous Matrices
Costly definitive analytical methods
Cheaper/screening analytical methods
High spatial densityLow DL + analyte specificity
Manages analytical uncertainty= analytical representativeness = analytical quality
Definitive analytical qualityScreening sampling quality
Manages sampling uncertainty= sampling representativeness = sampling quality
Definitive sampling qualityScreening analytical quality
Marrying Analytical Methods to Make Sound Decisions Marrying Analytical Methods to Make Sound Decisions Involving Heterogeneous MatricesInvolving Heterogeneous Matrices
Costly definitive analytical methods
Cheaper/screening analytical methods
High spatial densityLow DL + analyte specificity
Manages analytical uncertainty
Manages sampling uncertainty
Collaborative Data Sets
Marrying Analytical Methods to Make Sound Decisions Marrying Analytical Methods to Make Sound Decisions Involving Heterogeneous MatricesInvolving Heterogeneous Matrices
Costly definitive analytical methods
Cheaper/screening analytical methods
High spatial densityLow DL + analyte specificity
Manages analytical uncertainty
Manages sampling uncertainty
Decision Quality Data
Reliable (yet Cost-Effective) Scientifically Defensible Decisions
25
Improve Decision Quality--Manage UncertaintiesImprove Decision Quality--Manage Uncertainties
¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢
¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢
¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢
¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢
$ $ $
$ $ $
$ $ $
$ $ $
FROM THIS
TO THIS
Ex 1Ex 2Ex 3
Fixed Lab Analytical
UncertaintySampling Uncertainty
Ex 1
Sampling Uncertainty Controlled through Increased Density
Field Analytical
DataEx 2
Fixed Lab DataDecreased Sampling
Variability after Removal of Hotspots
Ex 3
Remove hot spots
26
Managing Decision Managing Decision UncertaintyUncertainty
Using Modern ToolsUsing Modern Tools
27
A Systems Approach FrameworkA Systems Approach Framework The Triad ApproachThe Triad Approach
Systematic Project
Planning
Dynamic Work Plan Strategy
Real-time Measurement Technologies
28
Unifying Concept for Unifying Concept for Triad:Triad: Managing Managing
UncertaintyUncertainty
Manage uncertainty about project goalsManage uncertainty about project goals – Identify decision goals with tolerable overall uncertaintyIdentify decision goals with tolerable overall uncertainty– Identify major uncertainties (cause decision error)Identify major uncertainties (cause decision error)– Identify the strategies to manage each major uncertaintyIdentify the strategies to manage each major uncertainty
Manage uncertainty in dataManage uncertainty in data– Sampling uncertainty: Sampling uncertainty: manage sample representativenessmanage sample representativeness – Analytical uncertainty: Analytical uncertainty: especially if field methods are usedespecially if field methods are used
Multidisciplinary expertise criticalMultidisciplinary expertise critical– A TEAMA TEAM is the best way to bring needed knowledge to bear is the best way to bring needed knowledge to bear
Systematic planning is used to proactively…Systematic planning is used to proactively…
Dynamic Work PlansDynamic Work Plans
Real-time decision-making “in the field”Real-time decision-making “in the field”– Evolve CSM in real-timeEvolve CSM in real-time– Implement pre-approved decision tree using senior staffImplement pre-approved decision tree using senior staff– Contingency planning: most seamless activity flow possible to Contingency planning: most seamless activity flow possible to
reach project goals in fewest mobilizationsreach project goals in fewest mobilizations
Real-time decisions need real-time dataReal-time decisions need real-time data– Use off-site lab w/ short turnaround?Use off-site lab w/ short turnaround?
» Use screening analytical methods in fixed lab?Use screening analytical methods in fixed lab?– Use on-site analysis?Use on-site analysis?
» Use mobile lab with conventional equipment?Use mobile lab with conventional equipment?» Use portable kits & instruments?Use portable kits & instruments?
MixAndMatch}
In all cases, must generate data of In all cases, must generate data of knownknown quality quality
30
Generating Real-time Data Using Field MethodsGenerating Real-time Data Using Field Methods Manage Uncertainty through Systematic PlanningManage Uncertainty through Systematic Planning
Need clearly defined data uses—tie to project goalsNeed clearly defined data uses—tie to project goals
Understand dynamic work plan—branch points & work flowUnderstand dynamic work plan—branch points & work flow
Project-specific QA/QC protocols matched to intended data useProject-specific QA/QC protocols matched to intended data use
SelectSelect field analytical field analytical technologies totechnologies to– Support theSupport the dynamic work plan dynamic work plan (greatest source of $$ savings) (greatest source of $$ savings)
– Manage Manage sampling uncertaintysampling uncertainty (improves decision quality) (improves decision quality)
SelectSelect fixed lab fixed lab methods (methods (as neededas needed) to) to– Manage Manage uncertainties in field datauncertainties in field data (just ONE aspect of QC for field data)(just ONE aspect of QC for field data)
– Supply analyte-specific data and/or lower quantitation limits Supply analyte-specific data and/or lower quantitation limits (if needed for regulatory compliance, risk assessment, etc.)(if needed for regulatory compliance, risk assessment, etc.)
31
Sample Representativeness is Key! Sample Representativeness is Key!
Cheaper analyses permit Cheaper analyses permit increased sample densityincreased sample density– New software for statistical/geostatistical decision supportNew software for statistical/geostatistical decision support
» VSP software pkg FREE: VSP software pkg FREE: http://http://dqodqo..pnlpnl..govgov/VSP/index./VSP/index.htmhtm
» SADA software pkg FREE:SADA software pkg FREE: http://www.http://www.tiemtiem..utkutk..eduedu/~/~sadasada//
» FIELDS/SADA software:FIELDS/SADA software: http://www.http://www.epaepa..govgov/region5fields/static/pages/index.html/region5fields/static/pages/index.html
Real-time measurements support Real-time measurements support real-time decision-real-time decision-makingmaking– Rapid feedback for course correction Rapid feedback for course correction smarter sampling smarter sampling
Data Quality: Focus on Data Quality: Focus on overall data uncertainty; overall data uncertainty; analytical uncertaintyanalytical uncertainty usually a small fraction usually a small fraction
FinallyFinally able to address able to address defensibly and affordably!defensibly and affordably!
32
Case Study: Wenatchee Tree Fruit SiteCase Study: Wenatchee Tree Fruit Site Pesticide IA kits guide dynamic work plan: remove and Pesticide IA kits guide dynamic work plan: remove and
segregate contaminated soil for disposalsegregate contaminated soil for disposal230 230 IA analysesIA analyses (w/ thorough QC) (w/ thorough QC)
Managed Managed sampling uncertaintysampling uncertainty: : achieved very high confidence achieved very high confidence that all contamination above that all contamination above action levels was located and action levels was located and removedremoved
Clean closure data setClean closure data set– 33 fixed lab samples for analyte-specific pesticide analysis 33 fixed lab samples for analyte-specific pesticide analysis – Demonstrate Demonstrate fullfull compliance with compliance with allall regulatory requirements for regulatory requirements for allall 33 pesticide analytes to >95% statistical confidence33 pesticide analytes to >95% statistical confidence the first timethe first time!!
Projected cost: ~$1.2M; Actual: $589K (Save ~ 50%)Projected cost: ~$1.2M; Actual: $589K (Save ~ 50%) Field work completed: <4 months; single mobilizationField work completed: <4 months; single mobilization
http://cluin.org/char1_edu.cfm#site_charhttp://cluin.org/char1_edu.cfm#site_char
Managed Managed field analyticalfield analytical uncertaintyuncertainty as additional QC on as additional QC on critical samples: confirmed & critical samples: confirmed & perfected field kit action levels)perfected field kit action levels)
++ 29 29 fixed-labfixed-lab samples for 33 analytes samples for 33 analytes
33
Terminology to Link Terminology to Link
Data Quality Data Quality
with with
Decision QualityDecision Quality
34
““Data Quality” TerminologyData Quality” Terminology
Current terminology usage does not focus Current terminology usage does not focus on the goal of decision qualityon the goal of decision quality
Irony: Great focus on the quality of data Irony: Great focus on the quality of data points;points; but overall quality of but overall quality of decisionsdecisions easily unknown. easily unknown.
Current usage does not distinguish Current usage does not distinguish – Methods vs. data vs. decisionsMethods vs. data vs. decisions– The factors that impact each step in the processThe factors that impact each step in the process– Relationships between different aspects of qualityRelationships between different aspects of quality
35
Proposed Clarification of TermsProposed Clarification of TermsQuality AssuranceQuality Assurance
Project QA:Project QA: ID causes of potential intolerable decision errors ID causes of potential intolerable decision errors & the strategies to manage and prevent those decision errors& the strategies to manage and prevent those decision errors
Data QA:Data QA: manage manage bothboth sampling and analytical uncertainties sampling and analytical uncertainties to degree needed to avoid decision errorsto degree needed to avoid decision errors– Analytical representativeness evaluated,Analytical representativeness evaluated, including impact of including impact of
sample/matrix effects on analytical performancesample/matrix effects on analytical performance
– Sample representativeness must be evaluatedSample representativeness must be evaluated
Lab QA:Lab QA: manage technical performance of analytical manage technical performance of analytical instruments, processes, and operators to meet instruments, processes, and operators to meet lab’slab’s QA goals QA goals– Sample/matrix effects on analytical performance may or may not Sample/matrix effects on analytical performance may or may not
be evaluated—depends on contract specifications.be evaluated—depends on contract specifications.
36
Analytical QA TerminologyAnalytical QA Terminology
Demonstration of method applicabilityDemonstration of method applicability– Shows that a particular method, project-specific Shows that a particular method, project-specific
SOP, and selected QC acceptance criteria are SOP, and selected QC acceptance criteria are appropriate for a project-specific application or appropriate for a project-specific application or site-specific matrixsite-specific matrix
Demonstration of proficiencyDemonstration of proficiency– Shows that a particular operator or lab can perform Shows that a particular operator or lab can perform
a method properlya method properly
37
Proposed Clarification of TermsProposed Clarification of TermsData QualityData Quality
Decision quality data*Decision quality data* = = Effective data*Effective data* = data shown = data shown to be effective for decision-makingto be effective for decision-making
Screening quality data*Screening quality data* = some useful information = some useful information provided; but too uncertain to support decision-making provided; but too uncertain to support decision-making alonealone
Collaborative data setsCollaborative data sets = distinct data sets used in = distinct data sets used in concert with each other to co-manage sampling and/or concert with each other to co-manage sampling and/or analytical uncertainties to an acceptable levelanalytical uncertainties to an acceptable level
* Includes sampling uncertainty. Nature of the analytical method irrelevant.* Includes sampling uncertainty. Nature of the analytical method irrelevant.
38
Misleading TerminologyMisleading Terminology
This term falsely implies that:This term falsely implies that:
• All methods run in the field are screening methods;All methods run in the field are screening methods;
• Therefore, all data produced in the field are of Therefore, all data produced in the field are of screening quality.screening quality.
Field Screening
39
““Effective Data”Effective Data” “Decision Quality Data” “Decision Quality Data”
Data ofData of known qualityknown quality
that can be logically demonstrated to bethat can be logically demonstrated to be effective for making the specified decisioneffective for making the specified decision
because both thebecause both the sampling and analytical uncertaintiessampling and analytical uncertainties
are managed to the degree necessary to meet clearlyare managed to the degree necessary to meet clearly defined (and stated) decision confidence goalsdefined (and stated) decision confidence goals
40
The Diffusion of InnovationThe Diffusion of Innovation
““At first people refuse to believe that a strange At first people refuse to believe that a strange new thing can be done, then they begin to new thing can be done, then they begin to hope it can be done—then it is done and all hope it can be done—then it is done and all the world wonders why it was not done the world wonders why it was not done centuries ago.”centuries ago.”
— —Francis Hodges BurnettFrancis Hodges Burnett