Post on 18-Dec-2015
WMP Technical Working Group – 14 Feb 2012
…and the Watershed Management Program!
WMP Technical Working Group – 14 Feb 2012
Last meeting we discussed the difficulties handling large numbers of ArcHydro-generated catchments, specifically including:
1- processing time to get the “micro-catchments”,2- time to review “micro-catchments”, and3- difficulty constructing the HydroNetwork for “micro-catchments”, and4- problems with Preferential Flow Path and catchment grouping.
Recent Developments:
1- Some empirical data on processing times and some surprising findings,2- Ideas to cut down on review time,3- A fix for the HydroNetwork construction, and4- no real progress on catchment grouping (yet.)
Well… three out of four isn’t bad!
WMP Technical Working Group – 14 Feb 2012
1 – Some Empirical Data
Experimental Conditions:
Dell T3500 – Win7-64 SP1; Intel Xeon Dual Core (3.2GHz); 4GB RAM ArcGIS 10: Builds 2414, 2800 (SP1), or 3200 (SP2)ArcHydro 2: Builds 2.1.0.16, 2.1.0.19, 2.1.0.22, 2.1.0.27)
Notes:
Some combinations of ArcGIS 10 service packs and ArcHydro 2 demonstrated incompatibilities. ArcGIS 3200 andArcHydro 2.1.0.16 and ArcHydro 2.1.0.19 produced unexpected errors. This was corrected with 2.1.0.22.
We find the most stable combinations to be ArcHydro 2.1.0.22 and .27with ArcGIS 2414 or 3200. We have not tested ArcGIS SP3
Time trials included processing up to HydroNetwork (Drainage ConnectivityCharacterization tools)
WMP Technical Working Group – 14 Feb 2012
Brooker Creek (Hillsborough County portion)Basin Area ~23.5 sq. milesDEM Size = 56,000,000 cells
SWF13023 SWF13023Sink Fill Area (in US Acres) 4356 sq ft NoFIllNumber of Catchments/Sinks 1,703 418,542
Sink PreScreening 0:04:07 0:04:00Sink Evaluation 0:05:55 0:48:45Fill Sinks 0:00:34 0:00:55Flow Direction with Sinks 0:04:20 0:12:30Flow Accumulation 0:02:13 0:01:16CatchmentGridDelineation 0:00:53 0:00:53Catchment Polygon Processing 0:00:21 0:13:34Drainage Point Processing 0:00:28 0:16:22Drainage Boundary Definition 0:00:48 1:24:18Sub-Total Processing Time 0:19:39 3:02:33
Drainage Connectivity (HydroNet) 0:03:26 7:08:47Preferential Flow 0:03:11 0:00:00
TOTAL PROCESSING TIME 0:26:16 10:11:20ArcHydro statistics:
BoundaryDrainage lines – 5,248 vs. 1,231,134 Sink Points – 1703 vs. 418,542Hydro Edges – 10,059 vs. 2,453,996HydroJunctions – 6,934 vs. 1,600,863
WMP Technical Working Group – 14 Feb 2012
Just a real quick calculation…
1 Day =
24 hours =
1440 minutes =
86,400 seconds
So…to review (just look at) each catchment inBrooker Creek for just 1 second each….
418,542 catchments (seconds) / 86,400 seconds/day =
4.85 DAYS
That is non-stop, doing ABSOLUTELY NOTHING else!
WMP Technical Working Group – 14 Feb 2012
Pemberton Creek – Hillsborough CountyPemberton Creek
Basin Area ~65 sq. miles
DEM Size = 156,000,000
ArcGIS Build 2414 SP0 2414 SP0
ArcHydro Build 2.1.0.19 2.1.0.19
Computer SWF13325 SWF13325
Sink Fill Area (in US Acres) 0.1 acre fill No Fill
Number of Catchments/Sinks 1158 965,076
Sink PreScreening 0:13:51 0:13:06
Sink Evaluation 0:18:38 1:50:21
Fill Sinks 0:02:49 0:02:50
Flow Direction with Sinks 0:10:24 8:06:46
Flow Accumulation 0:10:30 0:03:17
CatchmentGridDelineation 0:03:50 0:03:05
Catchment Polygon Processing 0:00:23 0:30:25
Drainage Point Processing 0:00:59 0:34:32
Drainage Boundary Definition 0:00:41 3:15:02
Sub-Total Processing Time 1:02:05 14:39:24
Drainage Connectivity (HydroNet) 0:03:55 17:03:00
Preferential Flow
TOTAL PROCESSING TIME 1:06:00 31:42:24
WMP Technical Working Group – 14 Feb 2012
Lines Time
North1 228663 1:21:04North2 290735 1:41:32North3 211685 1:15:01Central4 254676 1:32:49Central5 236254 1:24:37Central6 287534 1:45:03Central7 166714 1:02:16Central8 98580 0:37:25Central9 159914 0:58:55Central10 208352 1:15:44Central11 154406 0:58:30Central12 146900 0:56:46South13 153296 0:57:53South14 200880 1:15:25
Total 17:03:00
Pemberton Creek – ArcHydro HydroNetwork “Chunks”
The Drainage Connectivity Characterization Tool works on a selected set of Catchments(= ArcHydro “Chunk”)
WMP Technical Working Group – 14 Feb 2012
Pemberton Creek Catchments
WMP Technical Working Group – 14 Feb 2012
Raleigh-Rogers-Starvation Lake Watershed
Basin Area ~ 69.5 sq. miles
DEM Size = 156,000,000 cells
SWF13023 SWF13023 SWF13023 SWF13023 SWF13023
Sink Fill Area (in US Acres) 1 acre 0.5 acre 0.25 acre 4356 Sq. Ft. No Fill
Number of Catchments/Sinks 1,916 2,097 3,785 80,147 865,418
Sink PreScreening 0:12:06 0:11:54 0:11:54 0:11:53 0:11:59
Sink Evaluation 0:05:47 0:05:57 0:06:47 0:14:48 1:38:35
Fill Sinks 0:02:03 0:02:40 0:02:37 0:03:16 0:03:44
Flow Direction with Sinks 0:12:04 0:11:10 0:11:10 0:09:02 13:01:00
Flow Accumulation 0:06:39 0:06:10 0:06:01 0:04:40 0:03:45
CatchmentGridDelineation 0:02:30 0:02:29 0:02:23 0:02:01 0:02:42
Catchment Polygon Processing 0:00:34 0:00:34 0:00:44 0:04:07 0:28:05
Drainage Point Processing 0:00:55 0:00:55 0:00:58 0:03:27 0:37:51
Drainage Boundary Definition 0:01:11 0:01:15 0:01:42 0:17:21 3:44:39
Drainage Connectivity (HydroNet) 0:05:25 0:05:28 0:08:01 1:33:11 16:01:12
Preferential Flow 0:00:00 0:00:00 0:00:00 0:00:00 0:00:00
Total Processing Time 0:49:14 0:48:32 0:52:17 2:43:46 35:53:32
Raleigh-Rogers-Starvation Lake Watershed
WMP Technical Working Group – 14 Feb 2012
Pinellas-Anclote Watershed
Pinellas-Anclote Watershed
Watershed Size (sq. Miles) = 63.3
DEM size (number of Cells) = 180,325,425
Processing Machine SWF13322 SWF13322 SWF13023 SWF13023
ArcGIS Build/Service Pack 2800 SP0 2800 SP0 2414 SP0 2414 SP0
ArcHydro Build 2.1.0.16 2.1.0.16 2.1.0.19 2.1.0.19
Sink Fill Area (in US Acres) Half acre Fill Quarter Acre Fill 4356 SqFt 625 Sq Ft
Number of Catchments/Sinks 1,999 3,312 1,659 2,332
Sink PreScreening 0:17:56 0:17:50 0:16:54 0:15:38
Sink Evaluation 0:09:24 0:09:56 0:18:20 1:33:18
Fill Sinks 0:04:27 0:03:28 0:03:39 0:03:48
Flow Direction with Sinks 0:14:17 0:13:36 0:17:07 0:16:01
Flow Accumulation 0:09:52 0:09:33 0:12:44 0:10:24
CatchmentGridDelineation 0:05:49 0:05:04 0:06:07 0:05:25
Catchment Polygon Processing 0:00:34 0:00:42 0:00:31 0:00:37
Drainage Point Processing 0:00:59 0:01:01 0:01:07 0:01:53
Drainage Boundary Definition 0:01:10 0:01:33 0:01:01 0:01:15
Drainage Connectivity (HydroNet) 0:06:07 0:08:22 0:05:31 0:06:54
Preferential Flow 0:00:00 0:00:00 0:00:00 0:00:00
Total Processing Time 1:10:35 1:11:05 1:23:01 2:35:13
WMP Technical Working Group – 14 Feb 2012
Pinellas-Anclote Watershed
Watershed Statistics (625 sq. ft. fill): 2,332 Sinks/Catchments 6,971 Drainage Boundary Lines 9,281 HydroJunctions13,495 HydroEdges
WMP Technical Working Group – 14 Feb 2012
Pinellas-Anclote Watershed
Anclote Watershed
AncloteBasin HydroJunction
HydroEdgeCatchment
1:30,000 – 625 sq. ft. fill
WMP Technical Working Group – 14 Feb 2012
Pinellas-Anclote Watershed
Anclote Watershed
AncloteBasin HydroJunction
HydroEdgeCatchment
1:30,000 – No Fill
WMP Technical Working Group – 14 Feb 2012
Pinellas-Anclote Watershed
1:1200 No Fill
Watershed Statistics (No Fill) 1,146,540 Sinks/Catchments 3,371,256 Drainage Boundary Lines 3,733,211 HydroJunctions 5,156,619 HydroEdges
ArcGIS Build/Service Pack 2414 SP0 Chunk Lines TimeArcHydro Build 2.1.0.19
North1 430288 2:34;59Sink Fill Area (in US Acres) No Fill North2 333308 1:59:40Number of Catchments/Sinks 1,146,540 North3 147605 0:55:26
North4 122112 0:45:18Sink PreScreening 0:16:59 Central5 341814 2:08:15Sink Evaluation 2:11:28 Central6 230942 1:28:11Fill Sinks 0:04:00 Central7 240075 1:28:59Flow Direction with Sinks 14:37:55 Central8 227590 1:25:44Flow Accumulation 0:03:47 Central9 150866 0:57:18CatchmentGridDelineation 0:03:58 South10 140568 0:53:31Catchment Polygon Processing 0:36:22 South11 213308 1:20:04Drainage Point Processing 0:44:59 South12 281958 1:45:03Drainage Boundary Definition 3:59:51 South13 510882 3:10:05Drainage Connectivity (HydroNet) 18:17:34Preferential Flow 0:00:00
Total Processing Time 40:56:53
That is the equivalent of 17.7 days of Catchment evaluation!
WMP Technical Working Group – 14 Feb 2012
Conclusions: 1- Some fill is necessary to produce a “reasonable” number of ArcHydro
catchments for human evaluation. Results indicate that between0.1 acre-fill (4360 sq. ft.) and 5x5 DEM cell (625 sq. ft) fill seemsappropriate,
2- Filling to one of these thresholds yields ~ 1000 – 3000 ArcHydro Catchments for human evaluation,
3- As the Drainage Connectivity Characterization tool now operates onselected sets, smaller areas of interest (AOIs) can be selectedfrom the DEM and micro-catchments can be generated for those AOIs. Chunking and appropriate workflows can be usedto re-integrate AOIs into the HydroNetwork,
4- There are several relationships between the number of elements andprocessing times. Adding catchments/sinks increases the timefor Flow Direction computation but decreases the time for FlowAccumulation computation,
5- The “chunking” workflow permits computing HydroNetworks for LARGEwatersheds. Empirical data suggest that the processing takes~ one hour per 150000 DrainageBoundary lines.
WMP Technical Working Group – 14 Feb 2012
Part 2 – ArcGIS Data Reviewer (formerly known as Data ReViewer)
WMP Technical Working Group – 14 Feb 2012
Current StatusSince 2008, the District has been using a third-party ESRI extension, DataQE, forinsuring consistent Data Quality.
Advantages:
1- Works in the ArcGIS Environment,2- User customizable dictionary and rule set,3- Ruleset can be easily distributed to multiple users,4- Reasonable learning curve and cost point.
Disadvantages:
1- Doesn’t make very good use of the geodatabase,2- Doesn’t include any topological/geometric checks,3- Doesn’t track edits and audits made to the database,4- Doesn’t include any comment functionality (or tracking),5- Doesn’t record information regarding reviewers and editors,6- Doesn’t indicate severity of errors.
WMP Technical Working Group – 14 Feb 2012
Status under Evaluation for Adoption
Prior to ArcGIS 10, ESRI bundled several products into the Production Line Tool Set (PLTS) package, including:
Workflow ManagementJob TrackingVersion ManagementData Quality Management
Although these are still available as a bundled product, in 2011, ESRI beganoffering the Data Quality Management portion as either a “stand alone” or concurrently licensed extension as:
ArcGIS Data Reviewer
WMP Technical Working Group – 14 Feb 2012
ArcGIS Data Reviewer
Advantages: 1- Tightly integrated into ArcGIS
2- Reviewer Checks include,
Database Validation ChecksPolygon ChecksPolyLine ChecksTable ChecksTopology ChecksZ-Value ChecksSpatial Parameter EvaluationFeature on Feature ChecksDuplicate Geometry Checks
WMP Technical Working Group – 14 Feb 2012
ArcGIS Data Reviewer
Advantages continued,3- Maintains QC calls in feature classes and geodatabase tables,
Feature Dataset Reviewer Tables
WMP Technical Working Group – 14 Feb 2012
ArcGIS Data Reviewer
Advantages continued,4- Maintains Detailed information in the REVTABLEMAIN, such as:
Check types, who made it, severity of error, who repaired it,when/who checked the repair, etc.
WMP Technical Working Group – 14 Feb 2012
ArcGIS Data Reviewer
Advantages continued,5- Easy to construct and distribute the rule set. User can select all or a subset of rules to run.
WMP Technical Working Group – 14 Feb 2012
ArcGIS Data Reviewer
Advantages continued,6- Facilitates manual review by incorporating a user-defined grid
for review,7- User formatted, flexible report generator, and8- Insures that corrections are made and QCed!9- ESRI Maintained and managed software; will always be
“in step” with ArcGIS.
Disadvantages,
1- Higher cost point than DataQE (a standard ESRI Extension)for a single-use license ($2500 vs $899)
2- Steeper learning curve than DataQE (additional flexibilityand more options than DataQE)
3- Requires modifications to current workflows (1st reviewer, editor, 2nd reviewer workflow)
4- Additional data tables (and feature classes) are added to GWISto manage.
WMP Technical Working Group – 14 Feb 2012
Moving Forward
1- District Staff have developed rule sets for GWIS including:
2- District Staff are developing and testing workflows for implementation on WMP Projects,
3- District Staff have attended an ESRI Training Session and are developingGWIS-centric Training Materials,
4- Mapping & GIS has deployed ArcGIS Data Reviewer for ArcGIS 10
5- Mapping & GIS will phase out support of DataQE, starting in January 2012 and ending in December 2012. Mapping & GIS will not supportDataQE after January 2013.
WMP Technical Working Group – 14 Feb 2012
QUESTIONS AND/OR COMMENTS ???