Note: This presentation is being recorded for training purposes. 1 AHPS River Data Quality Control...

Note: This presentation is being recorded for training purposes.Note: This presentation is being recorded for training purposes. 11

AHPS River DataAHPS River DataQuality ControlQuality Control

Techniques and ToolsTechniques and Tools

OCWWS/HSDOCWWS/HSDHydrologic Support BranchHydrologic Support Branch

December 15, 2011December 15, 2011


OutlineOutline

IntroductionIntroduction SolutionSolution Review of Quality Control (QC) OperationsReview of Quality Control (QC) Operations Operational DiscussionOperational Discussion

Identifying ...Problem DataIdentifying ...Problem Data Reporting ...Problem DataReporting ...Problem Data Correcting ...Problem DataCorrecting ...Problem Data

ConclusionConclusion


Intro: ProblemIntro: Problem

Bad observed data at water.weather.govBad observed data at water.weather.gov Data is very visibleData is very visible

Evident on AHPS maps/graphs/tablesEvident on AHPS maps/graphs/tables Increased use of web info brings greater scrutinyIncreased use of web info brings greater scrutiny Data used by public, countless partners at all levelsData used by public, countless partners at all levels

Bad data characteristicsBad data characteristics Usually involves data spikesUsually involves data spikes Focus is on observed river dataFocus is on observed river data Forecast and precipitation data on web not coveredForecast and precipitation data on web not covered Current WHFS tools provide only limited support Current WHFS tools provide only limited support


Intro: GoalsIntro: Goals

1)1) Eliminate bad data presented to publicEliminate bad data presented to public2)2) Ensure reliable observations, best forecastsEnsure reliable observations, best forecasts3)3) Provide decision support at local, regional, and Provide decision support at local, regional, and

national levels using web infonational levels using web info4)4) Minimize burden on WFO staff - provide Minimize burden on WFO staff - provide

automated tools where possibleautomated tools where possible


Solution: SummarySolution: Summary

QC process modeled as three fundamental steps:QC process modeled as three fundamental steps:

1)1) IdentifyIdentify bad data automatically bad data automatically- Minimizes burden of manual review- Minimizes burden of manual review

2)2) Report/noticeReport/notice detected bad data detected bad data- Reported to forecasterReported to forecaster

- Automated, still requires user to take actionAutomated, still requires user to take action- Noticed by forecasterNoticed by forecaster

- - Manual review can simultaneously Manual review can simultaneously identify/”report”identify/”report”......

3)3) CorrectCorrect reported bad data reported bad data- - For AHPS, manually set data to missing (not delete)For AHPS, manually set data to missing (not delete)


Solution: PrinciplesSolution: Principles Focus on data spike problemsFocus on data spike problems

Most “bad” data spikes detectable via rate-of-change Most “bad” data spikes detectable via rate-of-change (ROC) check(ROC) check

But not all ROC-failed data are actually bad or But not all ROC-failed data are actually bad or questionablequestionable

Focus on spikes assumes range checks locally Focus on spikes assumes range checks locally configuredconfigured

Maintain separation of QC and Alert-Alarm (AA) functionsMaintain separation of QC and Alert-Alarm (AA) functions ROC thresholds exist for both QC and AAROC thresholds exist for both QC and AA QC judges data quality, AA judges threatQC judges data quality, AA judges threat Difficulty in trusting precise thresholds for ROC Difficulty in trusting precise thresholds for ROC

complicates automated QC processcomplicates automated QC process


Solution: DetailsSolution: Details

IdentifyIdentify using rate-of-change (ROC) check using rate-of-change (ROC) check Configure WHFS ROC_checker Configure WHFS ROC_checker

• Ensure thresholds are appropriateEnsure thresholds are appropriate – – Done by local office• Set token so ROC-failed data set as questionable, not bad Set token so ROC-failed data set as questionable, not bad

(qc_questorbad)(qc_questorbad) – – HSD Support will doHSD Support will do ReportReport using new QC report utility using new QC report utility

Complements Alert-Alarm report utilityComplements Alert-Alarm report utility Configure new utility operationsConfigure new utility operations

• Set up new utility in existing cron -Set up new utility in existing cron - HSD Support will doHSD Support will do• Set up trigger in textdb operationsSet up trigger in textdb operations - - Done by local office• Define QC product id in report scriptDefine QC product id in report script - - Done by local office

CorrectCorrect using WHFS TimeSeries application using WHFS TimeSeries application


QC Review: Info AvailableQC Review: Info Available Web QC part of general hydro data QC and Web QC part of general hydro data QC and

Alert-Alarm operationsAlert-Alarm operations Previous WHFS QC/Alert-Alarm webinarPrevious WHFS QC/Alert-Alarm webinar

Given October 6, 2009Given October 6, 2009 Powerpoint and recording available at: Powerpoint and recording available at: http://www.nwstc.noaa.gov/Hydrology/hydromain/webinars/index.htmlhttp://www.nwstc.noaa.gov/Hydrology/hydromain/webinars/index.html

Available WHFS documentationAvailable WHFS documentationhttps://https://ocwws.weather.gov/intranet/whfsocwws.weather.gov/intranet/whfs

http://www.nwstc.noaa.gov/Hydrology/hydromain/webinars/index.html

https://ocwws.weather.gov/intranet/whfs

https://ocwws.weather.gov/intranet/whfs


QC Review: QC ChecksQC Review: QC Checks All value set to one of three QC levels: All value set to one of three QC levels:

Good, Questionable, Bad (G, Q, B)Good, Questionable, Bad (G, Q, B) QC Checks in WHFSQC Checks in WHFS

Rate-of-changeRate-of-change• Only one ROC thresholdOnly one ROC threshold• If exceeded, set to Q or B (token controlled)If exceeded, set to Q or B (token controlled)

RangeRange• Separate thresholds for Q and BSeparate thresholds for Q and B

Obs-Forecast Difference - set to QObs-Forecast Difference - set to Q Station ConsistencyStation Consistency

• New report utility on LADNew report utility on LAD• Compares data from multiple sources at same stationCompares data from multiple sources at same station• Does not set QC flagDoes not set QC flag

Assume already setup ok

New focus


QC Review: ProcessingQC Review: Processing

Hydro data organized by type (obs, fcst, other) and physical element Hydro data organized by type (obs, fcst, other) and physical element (PE)(PE)

River Data Access for AHPSRiver Data Access for AHPS Height / FcstHeight tables (stage)Height / FcstHeight tables (stage) Discharge / FcstDischarge tablesDischarge / FcstDischarge tables RejectedData table *not* readRejectedData table *not* read

• Written by SHEFdecode for out-of-range data (per token Written by SHEFdecode for out-of-range data (per token shef_post_baddata) shef_post_baddata)

• ROC Bad data not moved there! (regardless of token)ROC Bad data not moved there! (regardless of token) HydroGen Data ProductsHydroGen Data Products

Tables read by genXML function within HydroGen operationsTables read by genXML function within HydroGen operations HML product file generated and sent every 15 minutes! HML product file generated and sent every 15 minutes! HydroGen does NOT filter Bad data HydroGen does NOT filter Bad data [say what!? change???][say what!? change???]


QC Review: HydroBase ControlsQC Review: HydroBase Controls QC thresholds defined as:QC thresholds defined as:

Default values for physical elementDefault values for physical element Location-specific values override defaultsLocation-specific values override defaults

ROC thresholdsROC thresholds ROC QC threshold should be setROC QC threshold should be set ROC Alert-Alarm thresholds are separateROC Alert-Alarm thresholds are separate

• Help identify hydromet threats, not bad dataHelp identify hydromet threats, not bad data If sensor continually sends bad data, consider disabling the station or If sensor continually sends bad data, consider disabling the station or

specific data elements specific data elements Location->Modify Location-> Additional Info: disable station for AHPS Location->Modify Location-> Additional Info: disable station for AHPS

and WHFSand WHFS Data Ingest->IngestFilter: disable specific element for AHPS and WHFSData Ingest->IngestFilter: disable specific element for AHPS and WHFS Setup->HydroGen Configuration: disable stationSetup->HydroGen Configuration: disable station Can also use AHPS CMS to...Can also use AHPS CMS to...

• Define annotated hydro note to describe data QC problem Define annotated hydro note to describe data QC problem • Disable station able using “Sensor in Service” optionDisable station able using “Sensor in Service” option


Operational DiscussionOperational Discussion

HydroDatabase

----Good

QuestionableBad

ROC_checker

report_hydroQC

TimeSeries

HydroGen

QC reportproduct

HML product Web

TextDatabase

Identify:

Report:

Correct:

report_alarmAlert/Alarm report

product

NEW

trigger


Identify Bad Data: ROC ChecksIdentify Bad Data: ROC Checks

Automated ROC checksAutomated ROC checks Detects data spikes (QC) or quick changes indicating Detects data spikes (QC) or quick changes indicating

threat (AA) threat (AA) Compares successive time series values Compares successive time series values Only second value in value pair being checked is flaggedOnly second value in value pair being checked is flagged Ignores B data, ignores Q data if program argument setIgnores B data, ignores Q data if program argument set

Data exceeding Data exceeding QC ROC threshold ROC threshold Flagged as Q or B based on token qc_questorbad Flagged as Q or B based on token qc_questorbad Default behavior is to set to BDefault behavior is to set to B Even if B, not moved to the RejectedData table so AHPS Even if B, not moved to the RejectedData table so AHPS

will still use it since AHPS does not filter bad data!!!will still use it since AHPS does not filter bad data!!! HSD Support will change token to set to QHSD Support will change token to set to Q


Report Bad Data: OptionsReport Bad Data: Options

Options for becoming aware of data QC problems:Options for becoming aware of data QC problems:

1)1) Internal text reports using textdb notification:Internal text reports using textdb notification:- New QC summary productNew QC summary product- Existing AA summary productExisting AA summary product

- Could set up dedicated report for only report ROC alert-alarmsCould set up dedicated report for only report ROC alert-alarms- Use –fRMT in run_roc_checker scriptUse –fRMT in run_roc_checker script

- Must consider this since some “bad” data may actually only be Must consider this since some “bad” data may actually only be flagged for alert-alarm purposes flagged for alert-alarm purposes

2)2) Manual review using WHFS applicationsManual review using WHFS applications3)3) Manual review using AHPS web pages themselvesManual review using AHPS web pages themselves4)4) Manual review using local CRH appManual review using local CRH app5)5) Other ideas?Other ideas?


Report Bad Data: New QC ReportReport Bad Data: New QC Report Name: report_hydroQCName: report_hydroQC Arguments:Arguments:

Table – Height, Discharge, or bothTable – Height, Discharge, or both Lookback time – number of minutes (MM)Lookback time – number of minutes (MM) Location filter – all, HSA only, HSA AHPS onlyLocation filter – all, HSA only, HSA AHPS only Other switches/filters to be added per requestOther switches/filters to be added per request

Function:Function: Reads database table(s)Reads database table(s)

• Considers all questionable or bad valuesConsiders all questionable or bad values• Filters on location filterFilters on location filter• Filters on time period ending at current time and beginning MM Filters on time period ending at current time and beginning MM

minutes earlierminutes earlier Formats text productFormats text product Sends text product to textdb if Q/B values foundSends text product to textdb if Q/B values found


Report Bad Data: New QC ReportReport Bad Data: New QC Report

** Report of Questionable/Bad Hydro Obs Data **

Period (Z): Tue 12-13 06:04:09 to Tue Dec 13 18:04:09

BOGW1 HG RP Dec 13 10:30 [ROC 1/3] (Bogachiel River near La Push) CENW1 HG RP Dec 13 12:00 [RNG 1/2] (Chehalis River at Centralia)CRNW1 HG RP Dec 13 10:30 [ROC 1/1] (Snoqualmie River near Carnation)UIL HZ RZ Dec 13 12:00 [SHF 0/1] (Quillayute)ALRW1 QR RZ Dec 13 15:00 [ROC 1/0] (Alder Dam)

Created: 12-13-2011 18:04:09 GMTReading Height, Discharge data.Shows latest obs time [QCreason questionable/bad value count]

Sample QC Report:


Notice Bad Data: WHFS ApplicationsNotice Bad Data: WHFS Applications HydroView LiveData-> HydroView LiveData->

Questionable/Bad windowQuestionable/Bad window Rejected Data (‘trash”) windowRejected Data (‘trash”) window

TimeSeriesTimeSeries Tabular/graphical displayTabular/graphical display

River MonitorRiver Monitor Automatically updatesAutomatically updates


Sample HydroView Questionable/Bad Window


Sample RiverMonitor Main Window“A” means alert/alarmIn effect


Sample RiverMon AlertAlarm Window


Notice Bad Data: AHPS PagesNotice Bad Data: AHPS Pages AHPS Page featuresAHPS Page features

““Area HydroGraph” feature Area HydroGraph” feature • Under “Additional Resources” Under “Additional Resources”

headingheading• Allows quick, multi-station Allows quick, multi-station

reviewreview Monitor AHPS map on screen in Monitor AHPS map on screen in

officeoffice Snapshots only - must refreshSnapshots only - must refresh

Local CRH app is compact, Local CRH app is compact, auto-auto-refreshrefresh of “area hydrograph: feature of “area hydrograph: feature

http://intra.crh.noaa.gov/metdat/http://intra.crh.noaa.gov/metdat/DDC/AHPS/ahps_qc.htmlDDC/AHPS/ahps_qc.html

On METDAT server, under On METDAT server, under DDC/AHPS/ DDC/AHPS/

http://intra.crh.noaa.gov/metdat/DDC/AHPS/ahps_qc.html

http://intra.crh.noaa.gov/metdat/DDC/AHPS/ahps_qc.html


Correcting Bad Data:TimeSeriesCorrecting Bad Data:TimeSeries General functions General functions

Primary tool to edit dataPrimary tool to edit data Also very helpful for reviewing dataAlso very helpful for reviewing data Using the "pre-defined groups" mode, can be pre-Using the "pre-defined groups" mode, can be pre-

configured to show select station dataconfigured to show select station data Correcting of dataCorrecting of data

Data already sent to AHPS must be set to missing to Data already sent to AHPS must be set to missing to correct it at the web farm database correct it at the web farm database

Next HydroGen run will retransmit data to webNext HydroGen run will retransmit data to web Presentation, titled "Data QC Procedural Change for Presentation, titled "Data QC Procedural Change for

WHFS", describing this: WHFS", describing this: <http://www.werh.noaa.gov/Hyd/ahps/

ahps_resource_phase6_final.htm#9._Data_QC_Procedural_change_for_WHFS_>


Sample TimeSeries Tabular Window


Conclusion: HQ To-DoConclusion: HQ To-Do HSD Support To-DoHSD Support To-Do

Will update token Will update token Will update script to create hydroQC productWill update script to create hydroQC product Will provide instructions for local changesWill provide instructions for local changes May do this as early as next weekMay do this as early as next week

Future ChangesFuture Changes Will push for future AWIPS work to improve Will push for future AWIPS work to improve

displays and tools for QC and situational displays and tools for QC and situational awarenessawareness


Conclusion: WFO To-DoConclusion: WFO To-Do

ConfigureConfigure Review QC/AA thresholds in HydroBaseReview QC/AA thresholds in HydroBase Define triggers in textDBDefine triggers in textDB Define product id in report_hydroQC scriptDefine product id in report_hydroQC script

Review data regularlyReview data regularly RiverMon/PrecipMon, AA reports, HydroView, RiverMon/PrecipMon, AA reports, HydroView,

TimeSeries…TimeSeries… Correct Correct

Set AHPS data to missing...don’t deleteSet AHPS data to missing...don’t delete


ConclusionConclusion Questions?Questions? Contact: [email protected]: [email protected] ThanksThanks

Note: This presentation is being recorded for training purposes. 1 AHPS River Data Quality Control...

Documents

Transcript of Note: This presentation is being recorded for training purposes. 1 AHPS River Data Quality Control...