Interactive Verification Program Training for AWIPS OB8.2

Interactive Verification ProgramInteractive Verification ProgramTraining for AWIPS OB8.2Training for AWIPS OB8.2

Hank HerrHank Herr

Office of Hydrologic DevelopmentOffice of Hydrologic Development

RFC Verification Workshop, 08/15/2007

22

OverviewOverview

IntroductionIntroduction IVP TerminologyIVP Terminology Software ComponentsSoftware Components Database UsageDatabase Usage Output FilesOutput Files DesignDesign DemonstrationDemonstration TipsTips Future PossibilitiesFuture Possibilities ExercisesExercises DiscussionDiscussion

33

Original IVP written around 2000-2001 for old HP based verify softwareOriginal IVP written around 2000-2001 for old HP based verify software Verification moved to archive database for OB4Verification moved to archive database for OB4

– IVP Batch Program software writtenIVP Batch Program software written– Old IVP GUI still ran, but used pairs files created during batch processing; Old IVP GUI still ran, but used pairs files created during batch processing;

it did not access the archive database directlyit did not access the archive database directly New IVP GUI written for OB7.2New IVP GUI written for OB7.2

– Provided ability to build verification products via GUI, modify components Provided ability to build verification products via GUI, modify components of a chart and save modifications, create batch file to reconstruct graphics of a chart and save modifications, create batch file to reconstruct graphics via IVP Batch Programvia IVP Batch Program

– Accessed archive database directlyAccessed archive database directly– Limitations:Limitations:

Designed for stage dataDesigned for stage data Accessed only three data tables of the archive databaseAccessed only three data tables of the archive database Limited number of statisticsLimited number of statistics No quality control of pairsNo quality control of pairs Inflexibility in building batch files from GUIInflexibility in building batch files from GUI JClass Desktop Views used to render charts (limited charting capabilities; JClass Desktop Views used to render charts (limited charting capabilities;

upgrade cost too much money)upgrade cost too much money)

IntroductionIntroductionHistory of IVPHistory of IVP

44

IntroductionIntroductionNew for OB8.2New for OB8.2

Designed for verification of stage, flow, temperature, and precipitation Designed for verification of stage, flow, temperature, and precipitation forecastsforecasts– Can do other data types as long as data is stored within accessible tables Can do other data types as long as data is stored within accessible tables – Accesses 10 data tables of the archive databaseAccesses 10 data tables of the archive database

Uses all data tables except pempsep, pemrsep, and pairedvaluesUses all data tables except pempsep, pemrsep, and pairedvalues Additional statisticsAdditional statistics

– Moments, CDF Plot, ROC, etc.Moments, CDF Plot, ROC, etc. Quality control of pairs via GUIQuality control of pairs via GUI

– View pairs in a time series plot, in addition to existing scatter plotView pairs in a time series plot, in addition to existing scatter plot– Mark pairs as ‘bad’ so that verification ignores themMark pairs as ‘bad’ so that verification ignores them– Change forecast or observed values within a pair and save changes to Change forecast or observed values within a pair and save changes to

databasedatabase More flexible batch file constructionMore flexible batch file construction

– Batch file can be built that will construct current chart for other Batch file can be built that will construct current chart for other combinations of locations and analysis periodscombinations of locations and analysis periods

Chart Director used for renderingChart Director used for rendering– Relatively quick, memory efficient chart renderingRelatively quick, memory efficient chart rendering– Used previously for HydroGen, so a run-time license already existsUsed previously for HydroGen, so a run-time license already exists

55

IVP TerminologyIVP Terminology

Verification LocationVerification Location– A forecast point or location and a data type for which verification is A forecast point or location and a data type for which verification is

to be performedto be performed– Uniquely specified by location id (lid within database) and SHEF Uniquely specified by location id (lid within database) and SHEF

pedtsep code.pedtsep code.– Includes following information:Includes following information:

Forecast and observed categoriesForecast and observed categories Observation type (raw or processed)Observation type (raw or processed) Active flag (yes or no)Active flag (yes or no) National flag (yes or no)National flag (yes or no) Sensor preferences (ordered list of observed type source, representing Sensor preferences (ordered list of observed type source, representing

preferences for which observed values to pair with a forecast)preferences for which observed values to pair with a forecast)

66


Verification GroupVerification Group– A set of locations and parameters that define the data to load and A set of locations and parameters that define the data to load and

other parameters of verificationother parameters of verification Locations must have the same number of categories and must be of Locations must have the same number of categories and must be of

the same type: stage (height), flow (discharge), temperature, the same type: stage (height), flow (discharge), temperature, precipitationprecipitation

– Includes following parameters:Includes following parameters: Analysis period and intervalsAnalysis period and intervals Lead time period and intervalsLead time period and intervals Issuance time-of-day (0-24 hours) period and intervalsIssuance time-of-day (0-24 hours) period and intervals Restrictions on location response time, active flag, and SHEF Restrictions on location response time, active flag, and SHEF

components (forecast type source, physical element, duration, and components (forecast type source, physical element, duration, and extremum)extremum)

Whether or not to compute statistics independently for each lid or Whether or not to compute statistics independently for each lid or forecast type sourceforecast type source

77


Forecast-Observed Data PairForecast-Observed Data Pair– A forecast value paired with an observed valueA forecast value paired with an observed value

Observed value can be “raw” or “processed” depending on if the value Observed value can be “raw” or “processed” depending on if the value was posted to the database by the raw or processed SHEF decoderwas posted to the database by the raw or processed SHEF decoder

Each forecast can have one raw pair and one processed pair Each forecast can have one raw pair and one processed pair associated with itassociated with it

– ““Best” candidate observation is chosen for each forecast based on Best” candidate observation is chosen for each forecast based on following criteria:following criteria: SHEF qualifier code (G, M, P, V, S, Z, T, F, and Q, in order)SHEF qualifier code (G, M, P, V, S, Z, T, F, and Q, in order) sensor preference (observation type source) ranksensor preference (observation type source) rank difference between forecast validtime and observation obstimedifference between forecast validtime and observation obstime

– A pair can be “good” or “bad”A pair can be “good” or “bad” Bad pairs are not used in verification computationsBad pairs are not used in verification computations

88

Software ComponentsSoftware Components

Vfyruninfo Editor GUIVfyruninfo Editor GUI– Manage vfyruninfo tableManage vfyruninfo table

IVPIVP– Data quality control and define verification products graphicallyData quality control and define verification products graphically

IVP Batch ProgramIVP Batch Program– Build pairs and verification products via a batch languageBuild pairs and verification products via a batch language

IVP Batch BuilderIVP Batch Builder– Edit IVP Batch Program batch files graphicallyEdit IVP Batch Program batch files graphically

Verify Pairs IngestorVerify Pairs Ingestor– Ingest pairs directly from text files of specific format into the Ingest pairs directly from text files of specific format into the

vfypairs or vfyprocpairs tablesvfypairs or vfyprocpairs tables del_bad_pairsdel_bad_pairs

– Removes bad pairs from the vfypairs and vfyprocpairs tableRemoves bad pairs from the vfypairs and vfyprocpairs table

99

Database UsageDatabase Usage

Verification Location InformationVerification Location Information– Vfyruninfo tableVfyruninfo table

Defines locations for which verification can be performed.Defines locations for which verification can be performed. Individual entries for each verification location and sensor Individual entries for each verification location and sensor

preference component (observed type source)preference component (observed type source) Edited via Vfyruninfo Editor GUIEdited via Vfyruninfo Editor GUI

– Location table Location table Information used in building labels, includes RFC name, full Information used in building labels, includes RFC name, full

location name, etc.location name, etc.

– Rivercrit table Rivercrit table Flood stages and flows (if location is of appropriate data type).Flood stages and flows (if location is of appropriate data type).

1010


Building PairsBuilding Pairs– Tables of data for pairing:Tables of data for pairing:

Forecasts: pedfsep, pehfsep, peqfsep, pedcsepForecasts: pedfsep, pehfsep, peqfsep, pedcsep Raw Observations: pecrsep, pedrsep, peoosepRaw Observations: pecrsep, pedrsep, peoosep Processed Observations: pedpsep, pehpsep, peqpsep, peoosepProcessed Observations: pedpsep, pehpsep, peqpsep, peoosep

– Vaddadjust tableVaddadjust table Adjust observations prior to pairing with a forecastAdjust observations prior to pairing with a forecast Only affects forecast-observed pairsOnly affects forecast-observed pairs Is not dated; there can be only one entry per locationIs not dated; there can be only one entry per location

– Sensok tableSensok table Allows for specifying time ranges during which sensors should Allows for specifying time ranges during which sensors should

not be used (i.e. observations should not be used for pairing)not be used (i.e. observations should not be used for pairing)

1111


Storing PairsStoring Pairs– Vfypairs tableVfypairs table

Stores pairs constructed using raw observationsStores pairs constructed using raw observations Field quality_code defines if the pair is good or badField quality_code defines if the pair is good or bad Can be very large (millions of records)Can be very large (millions of records)

– Vfyprocpairs tableVfyprocpairs table Stores pairs constructed using processed observationsStores pairs constructed using processed observations Field quality_code defines if the pair is good or badField quality_code defines if the pair is good or bad Can be very large, but will likely be a fraction of the size of Can be very large, but will likely be a fraction of the size of

vfypairsvfypairs

– A forecast can have up to two pairs built for it: one A forecast can have up to two pairs built for it: one stored in vfypairs and one stored in vfyprocpairsstored in vfypairs and one stored in vfyprocpairs Only one of the two pairs can be analyzed at one timeOnly one of the two pairs can be analyzed at one time

1212

Output FilesOutput Files

Output Statistics File (batch only)Output Statistics File (batch only)– Verification statistics output in an ASCII tabular formatVerification statistics output in an ASCII tabular format

Pairs FilePairs File– Pairs used to compute statisticsPairs used to compute statistics– ASCII tabular formatASCII tabular format

Output Image File (.png, .jpg)Output Image File (.png, .jpg)– PNG or JPG files displaying a plot of verification PNG or JPG files displaying a plot of verification

statisticsstatistics Image Data File (.dat)Image Data File (.dat)

– Statistic values comprising an output image plotStatistic values comprising an output image plot– ASCII tabular formatASCII tabular format

1313

DesignDesignFoundationFoundation

Archive Database

Archive database provides the following:Archive database provides the following:– Forecasts and observationsForecasts and observations

– Verification location informationVerification location information

– Forecast-observed data pairsForecast-observed data pairs

User

1414

DesignDesignVfyruninfo EditorVfyruninfo Editor

Archive Database

Vfyruninfo Editor

User

Vfyruninfo rows

Vfyruninfo Editor is used to specify verification Vfyruninfo Editor is used to specify verification locations and saves them to the vfyruninfo locations and saves them to the vfyruninfo table of the archive databasetable of the archive database

1515

PairingProcessor

Archive Database

User

pairs

Design Design IVP Batch ProgramIVP Batch Program

Vfyruninfo Editor

Vfyruninfo rows

forecasts,observations,location info

IVP Batch Program provides the user the IVP Batch Program provides the user the ability to:ability to:

– Pair forecasts with observed values according Pair forecasts with observed values according to user specified parametersto user specified parameters

parameters

Batch

Batch file

1616

DesignDesignIVP Batch ProgramIVP Batch Program

VerificationProcessor

PairingProcessor

Archive Database

User

pairs

ASCII

pairs, location info


Vfyruninfo Editor

Vfyruninfo rows

pairs, statistics


– Calculate statistics and output results to tabular Calculate statistics and output results to tabular ASCII filesASCII files

parameters

Batch

Batch file

1717

DesignDesignIVP Batch ProgramIVP Batch Program


PairingProcessor

Archive Database

User

ASCII

pairs

GraphicsProcessor

Graphics

statisticsparams

Vfyruninfo Editor

Vfyruninfo rows



pairs, statistics


– Generate user customizable graphical Generate user customizable graphical verification products displaying statisticsverification products displaying statistics

parameters

Batch

Batch file

1818

params


PairingProcessor

Archive Database

User

GUIGraphics

ASCII

pairs

pairs, statistics

DesignDesignIVP GUIIVP GUI

Vfyruninfo Editor

Vfyruninfo rows



IVP provides the user the ability to:IVP provides the user the ability to:– Specify, build, and view graphicsSpecify, build, and view graphics

GraphicsProcessor

statisticsparams

parameters

Batch

Batch file

1919

params


PairingProcessor

Archive Database

User

GUIGraphics

ASCII

pairs

pairs, statistics


Vfyruninfo Editor

Vfyruninfo rows



IVP provides the user the ability to:IVP provides the user the ability to:– Modify forecast-observed data pairsModify forecast-observed data pairs

GraphicsProcessor

statisticsparams

pairs

parameters

Batch

Batch file

2020

params


PairingProcessor

Archive Database

User

GUIGraphics

ASCII

pairs

parameters

pairs, statistics


Batch

Vfyruninfo Editor

Vfyruninfo rows



IVP provides the user the ability to:IVP provides the user the ability to:– Create batch files to produce graphics via IVP Create batch files to produce graphics via IVP

Batch ProgramBatch Program

GraphicsProcessor

statisticsparams

pairs

Batch file batchtokens

2121

DesignDesignIVP Window LayoutIVP Window Layout

1. Load Pairs 2. View and QC Pairs

3. Define Products4. View Products

2222

DemonstrationDemonstration

Feature availability:Feature availability:– Both OB7.2 and OB8.2Both OB7.2 and OB8.2– OB8.2 only*OB8.2 only*

2323

Start Vfyruninfo EditorStart Vfyruninfo Editor Filter ingestfilter table contents*Filter ingestfilter table contents* Select locations from ingestfilter table*Select locations from ingestfilter table* Create locations from scratchCreate locations from scratch Specify location parametersSpecify location parameters

– Active*Active*– National*National*– Response timeResponse time– Sensor preferencesSensor preferences

Save to archive databaseSave to archive database

DemonstrationDemonstrationVfyruninfo EditorVfyruninfo Editor

ref: Vfyruninfo Editor User’s Manual

2424

Commands:Commands:– Time window: START_TIME, END_TIMETime window: START_TIME, END_TIME– Specify locations: LOCATION, Specify locations: LOCATION, DUR*DUR*, , EXTREMUM*EXTREMUM*, ,

FCST_TS*FCST_TS*, , PE*PE*– Pairing window: PAIRING_WINDOWPairing window: PAIRING_WINDOW– Observation type: Observation type: OBS_TYPE (replaces OBS_TABLE)*OBS_TYPE (replaces OBS_TABLE)*– Persistence flag: PERSISTENCEPersistence flag: PERSISTENCE

Action:Action:– Build the pairs: BUILD_PAIRSBuild the pairs: BUILD_PAIRS

DemonstrationDemonstrationPairing Batch FilePairing Batch File

ref: IVP Batch Program User’s Manual for Pairing

2525

Example (buildpairs_template.bat):Example (buildpairs_template.bat):

DemonstrationDemonstrationPairing Batch FilePairing Batch File

# Pair all locations. Use default settings of # ALL for all other location specifying commandslocation = ALL

# Build pairs for a two week window prior to # current system timestart_time = "* - 14 days"end_time = "*"

# Parameters pairing_window = 1obs_type = RAW

build_pairs = true

ref: IVP Batch Program User’s Manual for Pairing

2626

Commands:Commands:– Time Window: START_TIME, END_TIMETime Window: START_TIME, END_TIME– Specify Locations: Specify Locations: DUR*DUR*, , EXTREMUM*EXTREMUM*, FCST_TS, PE, FCST_TS, PE

Action:Action:– Compute statistics: NATLSTATSCompute statistics: NATLSTATS

Recommendation*:Recommendation*:– Set the national flag for all locations that are part of the Set the national flag for all locations that are part of the

national program to ‘Y’ (use Vfyruninfo Editor)national program to ‘Y’ (use Vfyruninfo Editor)– Give the NATLSTATS action a value of NATIONALGive the NATLSTATS action a value of NATIONAL

DemonstrationDemonstrationNational Statistics Batch FileNational Statistics Batch File

ref: IVP Batch Program User’s Manual for Verification

2727

Example (natlstats_template.bat):Example (natlstats_template.bat):

DemonstrationDemonstrationNational Statistics Batch FileNational Statistics Batch File

# Compute stats for September, 2003.START_TIME = “2003-09-01 00:00:00”END_TIME = “2003-09-30 23:59:59”

# Default Values are ‘ALL’# FCST_TS = ALL# PE = ALL# DUR = ALL# EXTREMUM = ALL

NATLSTATS = NATIONAL

ref: IVP Batch Program User’s Manual for Verification

2828

Start IVPStart IVPgo vsys_scriptsgo vsys_scripts

ivpivp

Specify parameters of a verification groupSpecify parameters of a verification group– Analysis period and intervalsAnalysis period and intervals– Lead time period and intervalsLead time period and intervals– Issuance time period and intervals*Issuance time period and intervals*– Miscellaneous parametersMiscellaneous parameters

DemonstrationDemonstrationIVP: Verification Group ManagerIVP: Verification Group Manager

ref: IVP User’s Manual, Section 9

2929

Specify verification locations that comprise a Specify verification locations that comprise a groupgroup– Filter available locations*Filter available locations*– Choose locationsChoose locations– Specify categoriesSpecify categories– Specify observation type (from which pairs table Specify observation type (from which pairs table

to acquire data)*to acquire data)*

DemonstrationDemonstrationIVP: Verification Location ManagerIVP: Verification Location Manager


3030

Create IVP Data DisplayCreate IVP Data Display– Loading data progress windowLoading data progress window

Open the Verification Group Manager and Open the Verification Group Manager and Verification Location ManagerVerification Location Manager

Save the imageSave the image

DemonstrationDemonstrationIVP: IVP Data DisplayIVP: IVP Data Display


3131

Examine data through two views:Examine data through two views:– Scatter Plot ViewScatter Plot View

Select a location*Select a location* – In OB7.2, use Show button in Verification Location ManagerIn OB7.2, use Show button in Verification Location Manager

Display info about a location*Display info about a location* Line x = yLine x = y

– Time Series View*Time Series View* Specify time series to viewSpecify time series to view Emphasize a time seriesEmphasize a time series


ref: IVP User’s Manual, Sections 15.1, 15.2

3232

Draw a selected region rectangleDraw a selected region rectangle Zoom and the navigator panelZoom and the navigator panel

– Zoom in on scatter viewZoom in on scatter view– Zoom in on time series view*Zoom in on time series view*

Panning through the time series by zoomingPanning through the time series by zooming

– Shift the zoomed region via the navigator panel Shift the zoomed region via the navigator panel and keyboard*and keyboard*

– Zoom outZoom out


ref: IVP User’s Manual, Sections 15.6, 15.7

3333

View Pairs via the Verify Pairs ViewerView Pairs via the Verify Pairs Viewer– Select pairsSelect pairs– Sort tableSort table– Create a pairs fileCreate a pairs file

WARNING: Do not view very large WARNING: Do not view very large (30,000+) sets of pairs… the table rendering (30,000+) sets of pairs… the table rendering may be too slowmay be too slow



3434

Mark pairs as good or bad via IVP Data Display*Mark pairs as good or bad via IVP Data Display* Edit pairs via the Verify Pairs Editor*Edit pairs via the Verify Pairs Editor*

– Select pairsSelect pairs– Mark pairs as good or badMark pairs as good or bad– Edit forecast or observed value manuallyEdit forecast or observed value manually– Focus on a pairFocus on a pair– Edit forecast or observed value graphicallyEdit forecast or observed value graphically

Scatter viewScatter view Time series viewTime series view

Change the mouse mode*Change the mouse mode*

WARNING: Do not select large sets (1000+) of pairs for WARNING: Do not select large sets (1000+) of pairs for editing… the table rendering speed may be too slowediting… the table rendering speed may be too slow


ref: IVP User’s Manual, Sections 15.11, 17

3535

Open the Plot Definition ManagerOpen the Plot Definition Manager Select statistics Select statistics

– IVP Statistic Chooser ManagerIVP Statistic Chooser Manager Select plot typeSelect plot type Select x-axis variableSelect x-axis variable Select comparison variable*Select comparison variable* Edit statistics parameters*Edit statistics parameters* Edit group parameters*Edit group parameters*

– Spawns a window nearly identical to Verification Group ManagerSpawns a window nearly identical to Verification Group Manager

Note how variable selection affects analysis, lead time, and issuance Note how variable selection affects analysis, lead time, and issuance time-of-day intervalstime-of-day intervals

Note how variable selection affects category choice boxesNote how variable selection affects category choice boxes

DemonstrationDemonstrationIVP: Verification Plot Definition ManagerIVP: Verification Plot Definition Manager

ref: IVP User’s Manual, Sections 18, 19

3636

ErrorsErrors Categorical StatisticsCategorical Statistics Quantiles and ExtremesQuantiles and Extremes

– Computed relative to whichever variable whose category used is Computed relative to whichever variable whose category used is set to “Do Not Use”set to “Do Not Use”

Moments*Moments* Sample SizeSample Size

– The number of pairs used in the computations The number of pairs used in the computations ––or–or–

– Number of pairs within the category (if x-axis or comparison Number of pairs within the category (if x-axis or comparison variable is forecast or observed categories)variable is forecast or observed categories)

Misc.*Misc.* Special Plots*Special Plots*

DemonstrationDemonstrationIVP: The StatisticsIVP: The Statistics

ref: IVP Batch Builder User’s Manual, Appendix B

3737

Categorical Statistics use 3x3 Contingency TableCategorical Statistics use 3x3 Contingency Table

Examples:Examples:– HFAR = D/(D + E + F)HFAR = D/(D + E + F)– TFAR = (D + F)/(D + E + F)TFAR = (D + F)/(D + E + F)– CSI = (A+B+D+E)/(A+B+D+E+G+H+C+F) CSI = (A+B+D+E)/(A+B+D+E+G+H+C+F)

In a 2x2 contingency tableIn a 2x2 contingency table– HFAR is always 0 for the lowermost categoryHFAR is always 0 for the lowermost category– TFAR and HFAR are equivalent for uppermost categoryTFAR and HFAR are equivalent for uppermost category

POD + UFR + OFR = 1 in all casesPOD + UFR + OFR = 1 in all cases

DemonstrationDemonstrationIVP: The StatisticsIVP: The Statistics

ref: IVP Batch Builder User’s Manual, Appendix B

Observed was...

Forecast was... below within above

below A B C

within D E F

above G H I

3838


x-axis

primary y-axis

comparisonvariable

secondary y-axis

Statisticshort name(matches outputfiles)

3939

Choose categories usedChoose categories used– Disabled if x-axis or comparison variable is forecast or Disabled if x-axis or comparison variable is forecast or

observed categoriesobserved categories– Select “Do Not Use”Select “Do Not Use”

Only one of the two choice boxes can be “Do Not Use”Only one of the two choice boxes can be “Do Not Use”

– Impact of selections on how statistics are computedImpact of selections on how statistics are computed Category used restricts pairs used in computations to only Category used restricts pairs used in computations to only

those within the categorythose within the category Some statistics are calculated relative to the variable not used Some statistics are calculated relative to the variable not used

to categorize data (e.g. quantiles, to categorize data (e.g. quantiles, cdf plot*cdf plot*, , pdf plot*pdf plot*))

– Special case: ROC plot*Special case: ROC plot* Category used defines observation thresholdCategory used defines observation threshold


ref: IVP User’s Manual, Section 18.7

4040

Create IVP Statistic DisplayCreate IVP Statistic Display– Data is always loaded from memory even if the Data is always loaded from memory even if the

overridden group parameters imply additional data is overridden group parameters imply additional data is needed (such as a wider analysis interval, for example)needed (such as a wider analysis interval, for example)

– Allows for relatively quick plot generationAllows for relatively quick plot generation

Save the imageSave the image View the data via the IVP Statistics Data ViewerView the data via the IVP Statistics Data Viewer

– Create an ASCII format image data fileCreate an ASCII format image data file

DemonstrationDemonstrationIVP: IVP Statistic DisplayIVP: IVP Statistic Display


4141

Create a batch fileCreate a batch file– Change the analysis period*Change the analysis period*– Change the included locations*Change the included locations*– Create other groups for which to generate Create other groups for which to generate

products*products*– Specify output and template file namesSpecify output and template file names

Replacement strings are needed if images are to be Replacement strings are needed if images are to be produced for multiple groups in order to avoid produced for multiple groups in order to avoid overwriting output files*overwriting output files*

– Specify batch file nameSpecify batch file name

DemonstrationDemonstrationIVP: IVP Statistics DisplayIVP: IVP Statistics Display


4242

Open up the Chart Property ManagerOpen up the Chart Property Manager– Click on menu item or tool bar itemClick on menu item or tool bar item– Click on plot component*Click on plot component*

Change labelsChange labels Change axis limits or chart dimensionsChange axis limits or chart dimensions Change legend componentsChange legend components Change x-axis labels (for IVP Statistic Display)Change x-axis labels (for IVP Statistic Display) Create a template fileCreate a template file

– Don’t modify manually unless absolutely necessaryDon’t modify manually unless absolutely necessary Apply changes from a template fileApply changes from a template file

DemonstrationDemonstrationIVP: Chart Property ManagerIVP: Chart Property Manager

ref: IVP User’s Manual, Sections 24, Appendix D

4343

DemonstrationDemonstrationIVP: Chart Property ManagerIVP: Chart Property Manager

X-Axis Title

Y-Axis 1Title

Legend

Y-Axis 2Title

LegendTitle

Tick Mark Font

Plot Title

4444

Start IVP Batch BuilderStart IVP Batch Builder Load batch fileLoad batch file Insert, append, delete command linesInsert, append, delete command lines Edit command linesEdit command lines Comment and uncomment command linesComment and uncomment command lines Check for errorsCheck for errors Run batch fileRun batch file

DemonstrationDemonstrationIVP Batch Builder and the IVP Batch ProgramIVP Batch Builder and the IVP Batch Program

ref: IVP Batch Builder User’s Manual

4545

Create one batch file to generate products Create one batch file to generate products for many locationsfor many locations– @FILE Command to refer to other files@FILE Command to refer to other files– Replacement strings to avoid overwriting output Replacement strings to avoid overwriting output

filesfiles



4646

Organize output by using subdirectories of $Organize output by using subdirectories of $(vsys_output)*(vsys_output)*– If producing graphics for each location, use If producing graphics for each location, use

@ONELID or @ONELIDSHEF to define @ONELID or @ONELIDSHEF to define directories: directories:

OUTPUT_FILE = “@ONELID/output.txt,c”OUTPUT_FILE = “@ONELID/output.txt,c”– If producing graphics for time periods, use If producing graphics for time periods, use

@STARTTIME or @ENDTIME:@STARTTIME or @ENDTIME:OUTPUT_FILE = OUTPUT_FILE =

“@STARTTIME{[CCYY]}/output.txt,c”“@STARTTIME{[CCYY]}/output.txt,c”



4747

To minimize time spent querying database, load all needed To minimize time spent querying database, load all needed pairs at oncepairs at once– Only specify the overall analysis period and choose the locations Only specify the overall analysis period and choose the locations

within the Verification Group Manager and Verification Location within the Verification Group Manager and Verification Location ManagerManager Specify other parameters via the Plot Definition version of the Specify other parameters via the Plot Definition version of the

Verification Group Manager (click on Edit Group Parameters in the Plot Verification Group Manager (click on Edit Group Parameters in the Plot Definition Manager)Definition Manager)

– Important when generating many productsImportant when generating many products To minimize the chart rendering time for the IVP Data To minimize the chart rendering time for the IVP Data

Display, keep the number of pairs smallDisplay, keep the number of pairs small– Do one location at a time for a couple yearsDo one location at a time for a couple years– Do several locations at a time for one monthDo several locations at a time for one month– Important when quality controlling pairs dataImportant when quality controlling pairs data

See the Memory Limitations Tips later for recommended See the Memory Limitations Tips later for recommended numbers of pairs to loadnumbers of pairs to load

Tips:Tips:Verification Group Manager UsageVerification Group Manager Usage

4848

Use OBS_TYPE “RAW” for paired observations Use OBS_TYPE “RAW” for paired observations if…if…– Observed data was posted to archive database via raw Observed data was posted to archive database via raw

SHEF decoder, SHEF decoder, ANDAND– First letter of type source is ‘R’.First letter of type source is ‘R’.

Use OBS_TYPE “PROCESSED” if…Use OBS_TYPE “PROCESSED” if…– Observed data was posted to archive database via Observed data was posted to archive database via

processed SHEF decoder, processed SHEF decoder, OROR– First letter of type source is not ‘R’First letter of type source is not ‘R’

Example:Example:– If processed observations were posted via raw SHEF If processed observations were posted via raw SHEF

decoder (to peoosep table), then specify PROCESSEDdecoder (to peoosep table), then specify PROCESSED

Tips:Tips:Deciding Between PROCESSED or RAWDeciding Between PROCESSED or RAW

4949

Specifies parameters of graphical user interfacesSpecifies parameters of graphical user interfaces– FontsFonts– Window SizesWindow Sizes– ColorsColors– OthersOthers

LocationLocation– Default:Default:

$(get_apps_defaults vsys_dir)/app-defaults/*_SYSTEM_FILE.txt$(get_apps_defaults vsys_dir)/app-defaults/*_SYSTEM_FILE.txt

– Directory can be overridden by exporting environment variable Directory can be overridden by exporting environment variable SYSTEM_SETTINGS_FILE_DIRSYSTEM_SETTINGS_FILE_DIR Can allow for user specific system settingsCan allow for user specific system settings

FormatFormat– ESPADP/IVP batch language standard “<token> = <value>” formatESPADP/IVP batch language standard “<token> = <value>” format

Tips:Tips:System Settings FilesSystem Settings Files

ref: IVP User’s Manual, IVP Batch Builder User’s Manual, Vfyruninfo Editor User’s Manual, Appendix A

5050

Tips: Tips: Memory LimitationsMemory Limitations

Recommended number of pairs:Recommended number of pairs:

– Limits can be changed by using the ivp and ivpbatch script –m and –s Limits can be changed by using the ivp and ivpbatch script –m and –s options (run “ivp –hh” or “ivpbatch –hh” to read about these options)options (run “ivp –hh” or “ivpbatch –hh” to read about these options)

– Limits above are for IVP GUI; IVP Batch Program can deal with more Limits above are for IVP GUI; IVP Batch Program can deal with more pairs because not all pairs need to be stored in memory at one timepairs because not all pairs need to be stored in memory at one time

If IVP GUI runs out of memory, an error message will pop-up and If IVP GUI runs out of memory, an error message will pop-up and the pairs will be discarded (in most cases)the pairs will be discarded (in most cases)– The software may crash in other casesThe software may crash in other cases

Verify Pairs Viewer window is a memory hogVerify Pairs Viewer window is a memory hog– Add about 25% to the recommended value if not using the ViewerAdd about 25% to the recommended value if not using the Viewer

ref: IVP User’s Manual, IVP Batch Builder User’s Manual, Vfyruninfo Editor User’s Manual, Appendix A

System pairs QC product generation Approx. MaximumAX 20,000 20,000 40-60,000LX 100,000 400,000 600-800,000

Recommendation

5151

Debug problemsDebug problems– Export vsys_debug token prior to execution to see additional outputExport vsys_debug token prior to execution to see additional output

Useful levels are 0 (no output), 1, 20, 40, 60, 80, 100, 120, 128.Useful levels are 0 (no output), 1, 20, 40, 60, 80, 100, 120, 128. ““export vsys_debug=20” to see additional progress outputexport vsys_debug=20” to see additional progress output ““export vsys_debug=40” to see database query where clauses and export vsys_debug=40” to see database query where clauses and

timing results for chart renderingtiming results for chart rendering ““export vsys_debug=128” to see everything (lots and lots of output)export vsys_debug=128” to see everything (lots and lots of output)

PerformancePerformance– If IVP Data Display chart rendering is slow…If IVP Data Display chart rendering is slow…

Reduce number of pairs viewedReduce number of pairs viewed Zoom in on regions (points outside current zoom area are not Zoom in on regions (points outside current zoom area are not

rendered)rendered) Run on faster machines (port code to LX and execute software there)Run on faster machines (port code to LX and execute software there)

– If database query speed is slow…If database query speed is slow… Turn off SHEF decoders for a short time (if you can)Turn off SHEF decoders for a short time (if you can) Reduce number of locations or width of analysis periodReduce number of locations or width of analysis period

Tips:Tips:Debugging and PerformanceDebugging and Performance


5252

Future PossibilitiesFuture Possibilities Confidence IntervalsConfidence Intervals

– Algorithms need to be developedAlgorithms need to be developed– Ability to produce CI values and plot them needs to be addedAbility to produce CI values and plot them needs to be added

GUI redesignGUI redesign– Reduce the number of windowsReduce the number of windows– Reorganize window so that the Verification Group Manager does not need Reorganize window so that the Verification Group Manager does not need

to be in two placesto be in two places Add ability to handle multiple comparison variablesAdd ability to handle multiple comparison variables Add ability to modify other properties of a chartAdd ability to modify other properties of a chart

– Tick spacing, line style (solid, dashed, etc), line width, etc.Tick spacing, line style (solid, dashed, etc), line width, etc.– Requires additions to Chart Property ManagerRequires additions to Chart Property Manager

Create baseline batch files requiring no changes in order to produce Create baseline batch files requiring no changes in order to produce standard graphics standard graphics – Standard graphics need to be determinedStandard graphics need to be determined

Add ability to put changed forecasts and observed values back into Add ability to put changed forecasts and observed values back into original data tables and rebuild all affected pairs on-the-flyoriginal data tables and rebuild all affected pairs on-the-fly– IVP would become a more powerful, general QC tool for archive dataIVP would become a more powerful, general QC tool for archive data

5353

DocumentationDocumentation

Available on the webAvailable on the web

http://www.nws.noaa.gov/oh/hrl/verification/verification_doc_ob82.phphttp://www.nws.noaa.gov/oh/hrl/verification/verification_doc_ob82.php

5454

ExercisesExercises

Described in hand-outsDescribed in hand-outs– The data sets are very large, so while the pairs are The data sets are very large, so while the pairs are

loading, discuss the plots that are needed in order to loading, discuss the plots that are needed in order to answer the exercise questionsanswer the exercise questions What statistics are to be displayed?What statistics are to be displayed? What are the x-axis and comparison variables?What are the x-axis and comparison variables? Are there any observed or forecast category restrictions?Are there any observed or forecast category restrictions?

Group assignmentsGroup assignments

Group LX Machine Exercise 1 Location Exercise 2 Location1 lx1 BLKO2 TIFM72 lx3 GLOO2 WTTO23 lx4 MORA4 TIFM74 lx5 QUAO2 WTTO25 lx6 WTTO2 TIFM7

5555

DiscussionDiscussion

Review exercisesReview exercises Pre-workshop questionsPre-workshop questions

– Have they been answered?Have they been answered?

General comments on softwareGeneral comments on software

Interactive Verification Program Training for AWIPS OB8.2

Documents

Transcript of Interactive Verification Program Training for AWIPS OB8.2