TSAs “The Mother of All Audits”

Maria Salomon

What a TSA Shouldn’t Be

What’s a TSA? Why TSAs? What to expect? What’s been done? What’s been found? What we learned?

What Will I Cover?

Why Should it Matter to You?

What is a TSA?

An objective, systematic, and comprehensive assessment

Why Do We Do Them? Assess compliance with all requirements

Data quality objectives

Identify strengths and opportunities for

improvement

Who is Audited?

All of us, each monitoring organization performing work for EPA

When?

“Technical systems audits of each ambient air

monitoring organization shall be conducted at least every 3 years.”

Prevents data loss Improves overall data quality Identifies areas for improvement Small issues won’t become big issues Reduces findings during the regulatory TSA

Benefits of Internal TSAs

Tools for Conducting TSA

40 CFR, Part 58, Appendix A, Section 2.5 EPA QA Handbook, Section 15.3 EPA QA/G-7 ARB QA Manual, Volume 5 TSA Questionnaire - Appendix H

(EPA QA Handbook Volume II)

The entire air monitoring program is evaluated

What is the Scope?

Major Components

Pre-Audit On-site Audit Post Audit

Send TSA Questionnaire Schedule a conference call Data analysis Review prior audit reports Evaluate QA documents Review prior Air Quality Data Actions (AQDAs)

and Corrective Action Notifications(CANs)

Pre-Audit Activities

Data completeness (75% vs 85%) Data gaps Monitors not reporting Data trends

Data Quality Audit

Data Gap

Data Gap

Data Trend for Ozone

Intro briefing Evaluate and review

Organization Network design Technical procedures Monitoring sites Documents/records Data procedures

Exit briefing

Onsite Visit

“We don’t want you to view this audit committee as being confrontational in any

way”

Assess the entire data process to determine if: Procedures are adequately documented? Data can be replicated? Data is of sufficient quality for intended use?

Data Management

Validated Backups Limited access Verify manually entered data (human error,

typos are unacceptably high)

Data Systems

Finalize TSA report

Target is about 6 months

Implement corrective action A corrective action plan for each findings

Post-Audit Activities

TSAs conducted by EPA of ARB’s PQAO

2007 - ARB, Great Basin, San Joaquin, and Northern Sierra

2011 - ARB, Mendocino, San Joaquin, and Imperial

TSA History

TSAs conducted by ARB since 2009

Monterey Bay Unified Placer County Northern Sonoma County Sacramento Metropolitan Yolo-Solano Shasta

TSA History

Documentation Training Data management Corrective action Communication

Top 5 Findings

QMPs QAPPs SOPs Roles and Responsibilities

QA Documents

Station and instrument logbooks

Maintenance records Residence time

Documenting Findings

Training programs

Specific to job duties Cross training Succession planning

Training records Not documented

Training Findings

Data procedures not documented Data not verified, validated or certified Independent levels of review Data loss Incorrect AQS data

Data Management Findings

Expansion to AQDA process needed

ARB developed the CAN process

Corrective Action

Improved communication, coordination, and

oversight throughout network

Communication

Lessons Learned Streamline the entire audit process We need to improve TSA questionnaire Improve communication Increase consistency throughout network

Your PQAO Liaisons Carissa Ganapathy (916) 322-7105 Eastern Kern, Great Basin, Lake, Northern Sonoma, and Sacramento

Greg Gilani (916) 445-9391

Monterey, Northern Sierra, Placer, Santa Barbara, and San Luis Obispo

Darsi Goto (916) 324-9656 Imperial, Mojave/Antelope, Shasta, Siskiyou, and Tehama

Maria Salomon (916) 322-3267 Mendocino, North Coast, San Joaquin, Ventura, and Yolo/Solano

Questions