Automation Using Python for WVDOT GIS Data Management
description
Transcript of Automation Using Python for WVDOT GIS Data Management
Automation Using Python for WVDOT GIS Data Management
Yueming Wu, Ph.D., GISPGIS Manager
May 9, 2012
Agenda
IntroductionWVDOT Geospatial Data RepositoryAutomation ImplementationKey Issues & SolutionsFuture WorkQuestions & Answers
Introduction
HistoryHistorically the West Virginia Department of Transportation (WVDOT) has been utilizing geospatial (GIS, Remote Sensing, & GPS) technology for planning, analysis, and mapping purposes ever since the technology became available.
In 2005 the WVDOT GIS Section was created to meet the growing needs for geospatial data and services in transportation.
In 2007 the GIS Section was renamed as the GIS Unit and merged with the Highway Data Services (HDS) Unit, a program responsible for processing changes in roadway information, and the update and maintenance of the Roadway Inventory Log, to form the current Geospatial Transportation Information (GTI) Section.
- Roadway Inventory Log
-Roadway Statistics
-Public Certified Mileage
- Geospatial Data Management
-Geospatial Services
Geospatial Transportation Information (GTI) Section
Program Planning & Administration Division
Highway Data Services UnitGIS Unit
GTI Mission & Vision Statements
Mission StatementTo improve customer service to the citizens of West Virginia by supplying the latest geospatial transportation information.
Vision StatementThe West Virginia Department of Transportation Geospatial Transportation Information (GTI) Section is committed to managing an enterprise Geospatial Information System that will improve the productivity of the West Virginia Department of Transportation, meet growing customer needs, and play a leading role in West Virginia’s geospatial community.
GIS Implementation
Esri Shop Enterprise License Agreement (ELA) Esri Technology
Staff 7 GIS Professionals 2 Cartographers 1 GIS Coordinator at District 7
Enterprise GIS WVDOT Enterprise GIS Roadmap Enterprise Resource Planning (ERP)/Asset Management
System
Major GIS Projects in 2012 Enterprise GIS Implementation
New Transportation GIS Data Model District Level GIS Implementation Workflow Manager
Transportation GIS Data Collection(MS4, Railroad Crossings, Landslides, Outdoor Advertising, Right-of-Way, Trails, Meta Data, etc.) Highway Map Reproduction Web Mapping Applications (STIP, Park & Ride, Functional Classifications, etc.) Mobile GIS Applications (MS4, etc.) 3D GIS Application Development GIS Portal Upgrade ProjectWise Implementation ERP/Asset Management System Implementation
WVDOT Geospatial Data Repository
Export
ReprojectUpload
Update
Store
Download &Add
Metadata
Version - Edit GTI_EDITING
TRANSPORTATIONGIS DATA
• Commissioner Orders• Paper Maps• Field Data• Consultant Reports• Etc.
GTI_PUB_WM
GTI_PUB_UTM
UploadREFERENCE
GIS DATA • WV GIS Tech Center• WVGES• USGS• Esri• Etc.
IMAGERY• Aerial Photos• Shade Relief• Hillshade• Topo• Etc.
Editing
Publication
Source Data
Shapefiles
Imagery GIS Services
Shapefiles for
Download
Store
Download & Add
Metadata
Version - Edit Update
Geospatial Analysis
Map Production
WebApps
Reproject
Export
WVDOT Geospatial Data
Repository
Automation Implementation
Automation ImplementationFour Automated Data Management Operations
GTI_EDITING GDB GTI_PUB_UTM GDB Shapefiles GTI_PUB_UTM GDB GTI_PUB_UTM GDB GTI_PUB_WM GDB GTI_PUB_UTM GDB Shapefiles for Publications
GP1
GP2
GPn
GP …
Data Model
ArcGIS Model Builder
Python Script
Pythonwin
Task
Windows Scheduler
Export Deploy
Improve
Key Issues & Solutions
Database Lock
When a GDB is updated, it’s been accessed by clients (ArcGIS Desktop, ArcGIS Server, etc.). Database locks are created accordingly.
#Remove Database Locks on GTI_PUB_WMImport pyodbccnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=dotarcgis02;DATABASE=GTI_PUB_UTM;UID=xxxxx;PWD=xxx')cursor = cnxn.cursor()cursor.execute("delete FROM [GTI_PUB_UTM].[dbo].[SDE_process_information]" )cnxn.commit()
Scheduled Task
After a Python script passes test, use Windows Scheduler to create a task so the script would run on a regular basis. Make the task independent to individual’s login
Email Notification?
If a task fails, notify the data repository manager by email
#Emailimport smtplib, MIMETexttry:
…except:
sender = '[email protected]‘receivers = ['[email protected]', '[email protected]']msg = MIMEText("The GTI_PUB_UTM_2_Web_Shapefiles script failed. Please check!")
msg['Subject'] = 'Python Script Failed' msg['From'] = "[email protected]" msg['Reply-to'] = "GTI Core Team" msg['To'] = "[email protected]" smtpObj = smtplib.SMTP('localhost') smtpObj.sendmail(sender, receivers, msg.as_string())
or Log
After a task is created, use a log file to track. #LogImport os, sys, timerootFolder = r"C:\Geoprocessing Scripts\\" # This is where the log file is locatedtext_file = open(rootFolder + "PScriptLogFile.txt", "a")text_file.write("\n\n")text_file.write("Running GTI_EDITING_2_GTI_PUB_UTM.py began at " + str(time.strftime('%X %x %Z')) + ".\n\n")try:
… text_file.write("Running GTI_EDITING_2_GTI_PUB_UTM.py successfully ended at " +
str(time.strftime('%X %x %Z')) + ".")except: text_file.write("Running GTI_EDITING_2_GTI_PUB_UTM.py failed at " + str(time.strftime('%X %x %Z')) + ".") text_file.close()
Future Work
Export
ReprojectUpload
Update
Store
Download &Add
Metadata
Version - Edit GTI_EDITING
TRANSPORTATIONGIS DATA
• Commissioner Orders• Paper Maps• Field Data• Consultant Reports• Etc.
GTI_PUB_WM
GTI_PUB_UTM
UploadREFERENCE
GIS DATA • WV GIS Tech Center• WVGES• USGS• Esri• Etc.
IMAGERY• Aerial Photos• Shade Relief• Hillshade• Topo• Etc.
Editing
Publication
Source Data
Shapefiles
Imagery GIS Services
Shapefiles for
Download
Store
Download &Add
Metadata
Version - Edit Update
Geospatial Analysis
Map Production
WebApps
Reproject
Export
WVDOT Geospatial Data
Repository
Replicate
ReprojectUpload
UpdateGTI_EDITING
GTI_PUB_WM
GTI_PUB_UTM
Upload
Editing
Publication
Shapefiles
Update
Reproject
Headquarters
Replicate
Districts 1- 10
GTI_PUB_UTMFGDB
Publication
Replicate
WVDOT Geospatial Data
Repository
Imagery