Harry Halpin: Artificial Intelligence versus Collective Intelligence
Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology...
-
Upload
hope-mcdaniel -
Category
Documents
-
view
220 -
download
0
Transcript of Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology...
![Page 1: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/1.jpg)
Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin
Duke University Marine Geospatial Ecology Lab4-Mar-2009
![Page 2: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/2.jpg)
Talk outlineOverview of MGETQuick tour of the MGET tool collectionExample application: habitat modeling
![Page 3: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/3.jpg)
What is MGET?A collection of geoprocessing tools for marine ecology
Oceanographic data management and analysisHabitat modeling, connectivity modeling, statisticsHighly modular; designed to be used in many scenariosEmphasis on batch processing and interoperability
Free, open source softwareWritten in Python, R, MATLAB, C#, and C++Minimum requirements: Win XP, Python 2.4 ArcGIS 9.1 or later currently needed for many toolsArcGIS and Windows are only non-free requirements
Also useful for terrestrial problems!
![Page 4: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/4.jpg)
Set goals & priorities
Make & implement EBM decisions
Monitor &
assess
Develop conceptual
models
Analyze data & develop models and scenarios
Collect physical, biological, and
socioeconomic data
Visualize scenarios
![Page 5: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/5.jpg)
Set goals & priorities
Make & implement EBM decisions
Monitor &
assess
Develop conceptual
models
Analyze data & develop models and scenarios
Collect physical, biological, and
socioeconomic data
Visualize scenarios
Monitoring & Monitoring & assessment toolsassessment tools
Project Project management toolsmanagement tools
Data collection Data collection and management and management
toolstools
Conceptual Conceptual modeling modeling
toolstools
Data processing Data processing toolstools
Modeling toolsModeling tools-Model development tools
-Watershed models-Dispersal and habitat models
-Marine ecosystem models-Social science models
Sector-specific decision support toolsSector-specific decision support tools-Conservation and restoration site selection
-Coastal zone management tools-Fisheries management tools
-Hazard assessment and resiliency planning tools-Land use planning tools
Stakeholder Stakeholder communication communication & engagement & engagement
toolstoolsScenario Scenario
visualization visualization toolstools
![Page 6: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/6.jpg)
Set goals & priorities
Make & implement EBM decisions
Monitor &
assess
Develop conceptual
models
Analyze data & develop models and scenarios
Collect physical, biological, and
socioeconomic data
Visualize scenarios
Monitoring & Monitoring & assessment toolsassessment tools
Project Project management toolsmanagement tools
Conceptual Conceptual modeling modeling
toolstools
Sector-specific decision support toolsSector-specific decision support tools-Conservation and restoration site selection
-Coastal zone management tools-Fisheries management tools
-Hazard assessment and resiliency planning tools-Land use planning tools
Stakeholder Stakeholder communication communication & engagement & engagement
toolstoolsScenario Scenario
visualization visualization toolstools
Data collection Data collection and management and management
toolstools
Data processing Data processing toolstools
Modeling toolsModeling tools-Model development tools
-Watershed models-Dispersal and habitat models
-Marine ecosystem models-Social science models
Data collection Data collection and management and management
toolstools
Data processing Data processing toolstools
Modeling toolsModeling tools-Model development tools
-Watershed models-Dispersal and habitat models
-Marine ecosystem models-Social science models
MGET
![Page 7: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/7.jpg)
MGET’s software architectureMGET “tools” are really just Python functions, e.g.:
pythoncom2x.dll
IDispatchIMyTool
MyToolCOM class
MyTool.py
MGET ArcGIS Toolbox
Python programs
ArcGIS 9.xEarly-bound COM clients (e.g. C++)
Late-bound COM clients
(e.g. VBScript)
MGET
External callers
win32com module
MGET COM module
MGET exposes them to several types of external callers:
def MyTool(input1, input2, input3, output1)
![Page 8: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/8.jpg)
MGET interface in ArcGISThe MGET toolbox appears in the ArcToolbox window
![Page 9: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/9.jpg)
MGET interface in ArcGISDrill into the toolbox to find the toolsDouble-click tools to execute directly, or drag
to geoprocessing models to create a workflow
![Page 10: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/10.jpg)
Integration The Python functions can invoke C++, MATLAB, R, ArcGIS, and COM classes.
R interpreter
MyTool.m MyTool.r
MyTool.py
Python extension DLL
MyTool.cpp
C++
MyTool.pyd
Python extension DLL
MyToolMatlab.pyd
MATLAB Component Runtime (MCR)
rpy module
MGET COM module
win32com module
R packagesMATLAB toolboxes
IDispatch
COM Automation
classes
MGET ArcGIS module
arcgisscripting or win32com
module
ArcGIS geoprocessor
C libraries
ArcGIS toolboxes
Python packages
MGET R module
![Page 11: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/11.jpg)
MGET utilizes a lot of other softwareInterpreters / Runtimes
PythonMATLAB Component RuntimeR
R Packages
gamMASSmgcvrgdalROCR
Applications
ArcGISNOAA CoastWatch Utilities
Python Packages
docutilshttplib2lxmlnetcdf4numpyosgeopydappyparsingpyprojpywin32rpysetuptools
C Libraries
GDAL/OGRgziphdflibxmllibxsltnetcdfproj4zlib
All but one of these (pywin32) are installed automatically
![Page 12: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/12.jpg)
Quick tour of the tools
![Page 13: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/13.jpg)
Analyzing larval connectivityCoral reef ID and % cover maps Ocean currents data
Tool downloads data for the region and dates you specify
Larval density time series rasters
Edge list feature class representing dispersal network
Original research by Eric A. Treml
![Page 14: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/14.jpg)
Converting data
![Page 15: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/15.jpg)
Batch processingCopy one raster at a time
![Page 16: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/16.jpg)
Batch processingCopy rasters that you list in a table
![Page 17: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/17.jpg)
Batch processingCopy rasters from a directory tree
![Page 18: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/18.jpg)
Tools for specific products
Downloads sea surface height data from http://opendap.aviso.oceanobs.com/thredds
![Page 19: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/19.jpg)
Identifying SST fronts
~120 km
AVHRR Daytime SST 03-Jan-2005
28.0 °C
25.8 °C
Mexico
Front
Cayula and Cornillion (1992) edge detection algorithm
Freq
uenc
y
Temperature
Optimal break 27.0 °C
Strong cohesion front present
Step 1: Histogram analysis
Step 2: Spatial cohesion test
Weak cohesion no front
Bimodal
Example output
Mexico
ArcGIS model
![Page 20: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/20.jpg)
Identifying geostrophic eddies
Aviso DT-MSLA 27-Jan-1993 Red: Anticyclonic Blue: Cyclonic
Negative W at eddy core
SS
H a
nom
aly
Available in MGET 0.8
Example output
![Page 21: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/21.jpg)
Mapping species biodiversity
![Page 22: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/22.jpg)
Invoking R from ArcGIS
![Page 23: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/23.jpg)
Example application: habitat modeling
ChlorophyllSST
Bathymetry
Presence/absence observations
Sampled environmental data
Multivariate statistical model
Probability of occurrence predicted from environmental covariates
Binary classification
Warning: Habitat modeling is complicated! This simplified example is meant to briefly illustrate tools. Consult the literature for best practices!
![Page 24: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/24.jpg)
Focal species: Stenella frontalis
Photo: Garth Mix
Map: OBIS-SEAMAP
Common name:Atlantic Spotted Dolphin
Distribution:Tropical and warm temperate Atlantic
Study area: Eastern U.S.
![Page 25: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/25.jpg)
Species observation data
The Ocean Biogeographic Information System (OBIS) is a global database of marine species observations.The OBIS-SEAMAP system at Duke University holds the records for seabirds, marine mammals, and sea turtles, including records gathered during NOAA cruises.
![Page 26: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/26.jpg)
Environmental predictor variables
Bathymetry:ETOPO2V2 from NOAA NGDC
SST:Monthly climatological 4km AVHRR Pathfinder from NOAA NODC
Chlorophyll:Monthly climatological SeaWiFS chlorophyll-a from NASA GSFC
Images shown above are for month of March
![Page 27: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/27.jpg)
Step 1: Download species pointsDownload points using MGET tool:Presence: Records of Stenella frontalisAbsence: Records of other cetaceans
The tool uses the DiGIR protocol to retrieve data from OBIS servers
![Page 28: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/28.jpg)
Red: PresenceGreen: Absence
![Page 29: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/29.jpg)
Step 2: Convert oceanography to Arc rasters1.Download with FTP from NOAA and NASA:
ETOPO2 bathymetry – 1 binary fileAVHRR Pathfinder monthly climatological SST – 12 HDF filesSeaWiFS monthly climatological chlorophyll – 12 HDF files
2.Convert to ArcGIS rasters using MGET tools:
![Page 30: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/30.jpg)
Step 3: Sample oceanography at points
• Need to sample rasters and populate fields
• Must sample SST and chlorophyll by date
![Page 31: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/31.jpg)
Step 3: Sample oceanography at points
Sampling bathymetry is easy because it is static
To sample dynamic data such as SST and chlorophyll, you must first calculate the paths to rasters to sample from the points’ dates
Then use an MGET batch sampling tool
![Page 32: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/32.jpg)
Step 4: Create exploratory plots
Best predictors: SST and Chl
![Page 33: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/33.jpg)
Step 5: Fit, evaluate, and predict model
Presence ~ s(SST) + s(log10(Chlorophyll))Presence ~ s(SST) + s(log10(Chlorophyll))
![Page 34: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/34.jpg)
Partial plots produced by the Fit GAM tool
SST SSTlog10(Chlorophyll)
s(S
ST
,8.9
7)
s(lo
g10
(Ch
loro
ph
yll)
,5.6
)
Presence more likely at higher SST Presence more likely at lower Chl
![Page 35: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/35.jpg)
Plotting a receiver operating characteristic curve
![Page 36: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/36.jpg)
The ROC plot
False positive rate
Tru
e p
ositi
ve r
ate Cutoff = 0.020
By default, tool selects the cutoff closest to the point of perfect classification (0, 1)
Model summary statistics:
Area under the ROC curve (auc) = 0.960779Mean cross-entropy (mxe) = 0.030566Precision-recall break-even point (prbe) = 0.001866Root-mean square error (rmse) = 0.087781
Contingency table for cutoff = 0.019638:
Actual P Actual N TotalPredicted P 287 3541 3828Predicted N 26 32408 32434 Total 313 35949 36262
Accuracy (acc) = 0.901633Error rate (err) = 0.098367Rate of positive predictions (rpp) = 0.105565Rate of negative predictions (rnp) = 0.894435
True positive rate (tpr, or sensitivity) = 0.916933False positive rate (fpr, or fallout) = 0.098501True negative rate (tnr, or specificity) = 0.901499False negative rate (fnr, or miss) = 0.083067
Positive prediction value (ppv, or precision) = 0.074974Negative prediction value (npv) = 0.999198Prediction-conditioned fallout (pcfall) = 0.925026Prediction-conditioned miss (pcmiss) = 0.000802
Matthews correlation coefficient (mcc) = 0.246384Odds ratio (odds) = 101.026394SAR = 0.650065
ROC summary stats for cutoff:
![Page 37: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/37.jpg)
Predicting presence for oceanographic rasters
![Page 38: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/38.jpg)
Rasters output by the Predict GAM tool
Predicted presence:Range: 0 - 0.25
Standard errors:Range: 0 - 0.11
Binary classification:Species range map produced by classifying presence into 0 or 1 according to ROC cutoff
Similar to OBIS-SEAMAP range map?
Predictions for October
![Page 39: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/39.jpg)
AcknowledgementsA special thanks to the many developers of the open source software that MGET is built upon, including:
Guido van Rossum and his many collaborators; Mark Hammond; Travis Oliphant and his collaborators; Walter Moreira and Gregory Warnes; Peter Hollemans; David Ullman, Jean-Francois Cayula, and Peter Cornillon; Stephanie Henson; Tobias Sing, Oliver Sander, Niko Beerenwinkel, and Thomas Lengauer; Frank Warmerdam and his collaborators, Howard Butler; Timothy H. Keitt, Roger Bivand, Edzer Pebesma, and Barry Rowlingson; Gerald Evenden; Jeff Whitaker; Roberto De Almeida and his collaborators; Joe Gregorio; David Goodger and his collaborators; Daniel Veillard and his collaborators; Stefan Behnel, Martijn Faassen, and their collaborators; Paul McGuire and his collaborators; Phillip Eby, Bob Ippolito, and their collaborators; Jean-loup Gailly and Mark Adler; the developers of netCDF; the developers of HDFThanks to our funders:
![Page 40: Jason Roberts, Ben Best, Dan Dunn, Eric Treml, Pat Halpin Duke University Marine Geospatial Ecology Lab 4-Mar-2009.](https://reader036.fdocuments.net/reader036/viewer/2022062314/56649e8e5503460f94b91b3e/html5/thumbnails/40.jpg)
For more informationDownload MGET:
http://code.env.duke.edu/projects/mget
Email us:
[email protected], [email protected]
Learn more about habitat modeling:Guisan, A., Zimmermann, N.E. (2000) Predictive habitat distribution models in ecology. Ecological Modelling 135, 147–186.
Thanks for attending!