Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... •...
Transcript of Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... •...
![Page 1: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/1.jpg)
Inventory Your GeoData and Map ServicesUSING SQL SERVER AND PYTHON
![Page 2: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/2.jpg)
Metropolitan Transportation Commission
• MPO for 9-County San Francisco Bay Area
• Regional Transportation Plan • A blueprint for the development of mass transit, highway, freight, bicycle and
pedestrian facilities
• GIS Team of 4 supports Planning, Finance and Operations• Cartography• Geographic analysis• Web development• Data collection• DB Replication/Management
![Page 3: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/3.jpg)
• Types of data:• Census• Tom Tom MultiNet• Transit Route and Stop• Bike routes• Parking• Managed Lanes• Electric Vehicle Fueling• Freeway Service Patrol• Vehicle Miles Traveled• Clipper Card• Inrix TMC
GIS at MTC
![Page 4: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/4.jpg)
• System Architecture(All AWS Cloud Servers):
GIS at MTC
G-DriveLayer Files
Copies of GISDB1 data for Web Services
Main data repository Production Maps Server
DevelopmentMaps Server
Open Data (official data)
Custom Applications
AGOL (unofficial temp maps)
GISDB1 DB Server RDS
![Page 5: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/5.jpg)
Problems
• Open Data bulk uploads
• Integrating Metadata Updates into a fast-paced environment
• Open up Metadata to employees outside GIS to access/search/contribute
• Track feature class updates through workflow
• Identify feature classes tied to Map Services
• Search for data on all servers (Search metadata)
![Page 6: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/6.jpg)
Integrate Metadata Updates into WorkflowUSING PYTHON AND ARCPY
![Page 7: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/7.jpg)
Integrate Metadata Updates into Workflow
PART 1
• Gatekeeper for adding data to main repository
• Narrow metadata fields to our minimum standards• Title• Tags• Purpose• Abstract• Data Credits• Use Limitations• Contact Info
![Page 8: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/8.jpg)
Integrate Metadata Updates into Workflow
PART 2
• Metadata Toolkit Python/ArcPy
• Create tool to import feature data that enforces metadata updates
• Create Python Script tool to update metadata outside of Catalog’s editing interface
• Uses Validation
• Uses XSLT: arcpy.XSLTransform_conversion
![Page 9: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/9.jpg)
Searching for Data and Expanding Access to MetadataUSING SQL SERVER AND PYTHON
![Page 10: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/10.jpg)
Searching and Metadata
PART 1: SQL/SDE Solutions
• Search for data on all servers
• Determine which servers a dataset is on
• Compare row counts of the same dataset on different servers/dbs
• Search metadata
• SQL Server and XPath
gdb.Documentation.value('(*/dataIdInfo/idCitation/resTitle)[1]', 'nvarchar(max)') AS Title
![Page 11: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/11.jpg)
Metadata View
![Page 12: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/12.jpg)
Dataset Locator View
![Page 13: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/13.jpg)
Searching and Metadata
PART 2: Python Solutions
• Map Service Inventory
• Layer Files Inventory
• Python:• os module:
• walk
• path
• arcpy package:• InsertCursor
• Mapping
• Layer
• dataSource/datasetName
• MapDocument
• ListDataFrames
• ListLayers
• dataSource/datasetName
![Page 14: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/14.jpg)
Map Services in SQL
![Page 15: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/15.jpg)
Layer Library Table in SQL
![Page 16: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/16.jpg)
Searching and Metadata
PART 3: Inventory
• Search Data, Layer files, and Map Services
• View Metadata
• Edit/Update Metadata
• Add data to ArcMap or AGOL
DEMO
![Page 17: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/17.jpg)
Inventory Application
![Page 18: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/18.jpg)
Inventory Application
![Page 19: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/19.jpg)
Inventory Application
![Page 20: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/20.jpg)
Inventory Application
![Page 21: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/21.jpg)
Inventory Application
![Page 22: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/22.jpg)
Inventory Application
![Page 23: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/23.jpg)
View Map Service in AGOL
![Page 24: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/24.jpg)
Inventory Application
• Data and Services• SQL Server• ESRI
• ArcGIS Server, ArcSDE, ArcMap, AGOL, ArcPy/Python
• Backend Code• Node.js
• Frontend Code• Bootstrap, HTML5, CSS, jQuery, Kendo UI
Technology:
![Page 25: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/25.jpg)
Open Data Bulk Upload
• ESRI Github AGOL Admin Tools
• Bulk load map services to AGOL• Python
• auto-calculate values for CSV file
![Page 26: Inventory Your GeoData and Map Services - Esri · Inventory Your GeoData and Map Services ... • Inrix TMC. GIS at MTC • System Architecture (All AWS Cloud Servers): GIS at MTC.](https://reader031.fdocuments.net/reader031/viewer/2022012918/5e965252d8478731062bf061/html5/thumbnails/26.jpg)
Solutions
• Open Data bulk uploads
• Integrating Metadata Updates into a fast-paced environment
• Open up Metadata to employees outside GIS to access/search/contribute
• Track feature class updates through workflow
• Identify feature classes tied to Map Services
• Search for data on all servers (Search metadata)