Community for Data Integration Conference Call September 8, 2010 Dave Briar The USGS National Water...
-
Upload
benjamin-johnston -
Category
Documents
-
view
225 -
download
5
Transcript of Community for Data Integration Conference Call September 8, 2010 Dave Briar The USGS National Water...
Community for Data IntegrationConference Call
September 8, 2010Dave Briar
The USGSThe USGS
National Water Information SystemNational Water Information System
Design, Uses, and FutureDesign, Uses, and Future
Community for Data IntegrationConference Call
September 8, 2010Dave Briar
The USGSThe USGS
National Water Information ServicesNational Water Information Services
Design, Uses, and FutureDesign, Uses, and Future
slide 3
• core-NWIS / NWISWeb systemscore-NWIS / NWISWeb systems– Different functions and architecture – same philosophyDifferent functions and architecture – same philosophy
• Public and internal usesPublic and internal uses– Nontraditional and automated accessNontraditional and automated access
• Universal discovery and accessUniversal discovery and access– The promise of interoperable web servicesThe promise of interoperable web services
Topics:Topics:
slide 4
• Uses a data model optimized for its prime Uses a data model optimized for its prime functions -- the acquisition, processing, and long-functions -- the acquisition, processing, and long-term archiving of data.term archiving of data.
• Data processing (computation of derivatives, Data processing (computation of derivatives, review, qualification, and approvalreview, qualification, and approval) is highly interactive favoring local access to server.
• core-NWIS is developing its own internal services core-NWIS is developing its own internal services so that desktop-processing tools can be so that desktop-processing tools can be contracted for development by outside vendors.contracted for development by outside vendors.
core-NWIS:core-NWIS:
slide 5
• Uses a data model that is optimized for its prime Uses a data model that is optimized for its prime function – to provide highly available and efficient function – to provide highly available and efficient public access to USGS water data.public access to USGS water data.
• NWISWeb receives selected data sent from the 45 NWISWeb receives selected data sent from the 45 core-NWIS systems and creates a nationally core-NWIS systems and creates a nationally seamless view of those data for public access.seamless view of those data for public access.
• NWISWeb national architecture and design NWISWeb national architecture and design philosophy reflects history and USGS culture of philosophy reflects history and USGS culture of local control and “ownership” of their data.local control and “ownership” of their data.
NWISWeb:NWISWeb:
slide 6
-- NWIS database installation
slide 7
http://waterdata.usgs.gov/
Secure RepositorySecure RepositorySecure RepositorySecure Repository Public “View”Public “View”Public “View”Public “View”45 NWIS hosts
http://waterservices.usgs.gov/
slide 8
NNationalational AAggregateggregate DDatasetataset
Total sitesReal-time sites
Daily valuesGround-water levels
QW samplesQW values
Peaks
1,541,987 12,320324,567,200 8,437,274 4,836,150 88,893,776 699,583
http://waterdata.usgs.gov/
Secure RepositorySecure RepositorySecure RepositorySecure Repository Public “View”Public “View”Public “View”Public “View”45 NWIS hosts
http://waterservices.usgs.gov/
slide 9
-- NWIS database installation
-- NatWeb Node with Firewall
slide 10
NWISWeb Data DeliveryNWISWeb Data Delivery
Menlo Park CA
Sioux Falls SD
Reston VA
Five redundant copies of all timeseries data for all sites
nationwide
Two copies of all data discrete / historical
for all sites nationwide
NAD1
NAD2
slide 11
core-NWISDatabase
LRGS
GOES
DOMSAT
slide 12
NWISDatabase
LRGS
GOES
DOMSAT
slide 13
NWISWeb Data TransferNWISWeb Data Transfer
NAD
Menlo Park CA
Sioux Falls SD
Reston VA
HUB1
NWIS Database
slide 14
NADHUB1
TS dataTS data TS data
All data
Sioux Falls SD
Reston VA
Menlo Park CA
NWISWeb Data TransferNWISWeb Data Transfer
slide 15
NAD
HUB2
HUB1
Sioux Falls SD
Reston VA
Menlo Park CA
NWISWeb Data TransferNWISWeb Data Transfer
slide 16
HUB2
TS data
TS data
TS data
All data
NAD
Sioux Falls SD
Reston VA
Menlo Park CA
NWISWeb Data TransferNWISWeb Data Transfer
slide 17
HUB2
TS data
TS data
TS data
All data
NAD
Sioux Falls SD
Reston VA
Menlo Park CA
NWISWeb Data TransferNWISWeb Data Transfer
slide 18
NWIS Database is single point of failureNWIS Database is single point of failure
NADHUB1
NWISDatabase
slide 19
NWISDatabase
LRGS
GOES
DOMSAT
slide 20
NADHUB1NWIS-RT in
“Hot Standby”
NWIS – NWIS-RT BackupNWIS – NWIS-RT Backup
Replication
NWISDatabase
DOMSAT
slide 21
NADHUB1NWIS-RT Serving WSC Data
NWIS – NWIS-RT BackupNWIS – NWIS-RT BackupDOMSAT
slide 22
Louisiana Data -- Hurricane KatrinaLouisiana Data -- Hurricane Katrina
slide 23
Louisiana Data -- Hurricane KatrinaLouisiana Data -- Hurricane Katrina
slide 24
2004 Tropical Storm Tracks2004 Tropical Storm Tracks
slide 25
• core-NWIS / NWISWeb systemscore-NWIS / NWISWeb systems– Different functions and architecture – same philosophyDifferent functions and architecture – same philosophy
• Public and internal usesPublic and internal uses
– Nontraditional and automated accessNontraditional and automated access
• Universal discovery and accessUniversal discovery and access– The promise of interoperable web servicesThe promise of interoperable web services
Topics:Topics:
slide 26
Testimonial:
Principal of a School: "Our school floods quite frequently. I have calibrated our response and reactions to this great data source. I deeply appreciate the availability, as do the 300+ families who benefit from the benefit our school gains from the data."
Principal of a School: "Our school floods quite frequently. I have calibrated our response and reactions to this great data source. I deeply appreciate the availability, as do the 300+ families who benefit from the benefit our school gains from the data."
slide 27
Testimonial:
"I live on lakefront property at Canyon Lake, Texas. Your website was like a bright light in a dark cave during the flood of 2002. You gave us the essential and critical information needed to make major decisions which would effect my family and home. Thank you for providing a portal to the big picture, enabling us to make timely decisions with up to date information. Making accurate decisions is difficult when threatened with flooding, but it would have been impossible without your website. Keep up the good work."
"I live on lakefront property at Canyon Lake, Texas. Your website was like a bright light in a dark cave during the flood of 2002. You gave us the essential and critical information needed to make major decisions which would effect my family and home. Thank you for providing a portal to the big picture, enabling us to make timely decisions with up to date information. Making accurate decisions is difficult when threatened with flooding, but it would have been impossible without your website. Keep up the good work."
slide 28
NWISWeb Usage, in millions of successful requests per month
0
10
20
30
40
50
60
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
slide 29
Public and Internal UsesPublic and Internal Uses
• NWISWeb allows 4 levels of accessNWISWeb allows 4 levels of access– Public, cooperator, USGS, and local Public, cooperator, USGS, and local
office onlyoffice only
slide 30
Public and Internal UsesPublic and Internal Uses
• NWISWeb allows 4 levels of accessNWISWeb allows 4 levels of access– Public, cooperator, USGS, and local Public, cooperator, USGS, and local
office onlyoffice only
• NWISWeb uses http GETNWISWeb uses http GET– All pages are “bookmarkable”All pages are “bookmarkable”– All data are retrievable in ASCII tab-All data are retrievable in ASCII tab-
delimited (RDB) formatdelimited (RDB) format
slide 31
Where What Output formatWhere What Output format
slide 32
Public and Internal UsesPublic and Internal Uses
• NWISWeb allows 4 levels of accessNWISWeb allows 4 levels of access– Public, cooperator, USGS, and local Public, cooperator, USGS, and local
office onlyoffice only
• NWISWeb uses http GETNWISWeb uses http GET– All pages are “bookmarkable”All pages are “bookmarkable”– All data are retrievable in ASCII tab-All data are retrievable in ASCII tab-
delimited (RDB) formatdelimited (RDB) format– ~50+% of total hits are from automated ~50+% of total hits are from automated
programs (not a user at a browser)programs (not a user at a browser)
slide 33
slide 34
slide 35
slide 36
slide 37
slide 38
slide 39
slide 40
slide 41
slide 42
• core-NWIS / NWISWeb systemscore-NWIS / NWISWeb systems– Different functions and architecture – same philosophyDifferent functions and architecture – same philosophy
• Public and internal usesPublic and internal uses
– Nontraditional and automated accessNontraditional and automated access
• Universal discovery and accessUniversal discovery and access– The promise of interoperable web servicesThe promise of interoperable web services
Topics:Topics:
slide 43
Web Service DefinitionWeb Service Definition
• A Web service is defined by the W3C as:A Web service is defined by the W3C as: “ “A software system designed to support interoperable A software system designed to support interoperable
machine-to-machine interaction over a network“. machine-to-machine interaction over a network“.
• Web services are frequently just Web APIs that can be Web services are frequently just Web APIs that can be accessed over a network, such as the Internet, and accessed over a network, such as the Internet, and executed on a remote system hosting the requested executed on a remote system hosting the requested services.services.
• By this definition, NWISWeb has been a web service from By this definition, NWISWeb has been a web service from the beginning, however, discovery by and interoperability the beginning, however, discovery by and interoperability with other services/portals has been limited.with other services/portals has been limited.
slide 44
NWISWeb Data Discovery ServicesNWISWeb Data Discovery Services
• EPA “Window to my environment”EPA “Window to my environment”– March 2001March 2001– Site location and type (no catalog)Site location and type (no catalog)
• EDC ARC IMS map feature serviceEDC ARC IMS map feature service– March 2001 March 2001 – Expanded Site Info (basic local catalog)Expanded Site Info (basic local catalog)
• CUAHSI Hydrologic Information System (HIS)CUAHSI Hydrologic Information System (HIS)– January 2007 January 2007 – Expanded site Information including data period-of-Expanded site Information including data period-of-
record (extensive catalog)record (extensive catalog)
slide 45
Question: How much “discovery” information Question: How much “discovery” information should be cached at a Portal catalog verses should be cached at a Portal catalog verses supplied dynamically via its own service?supplied dynamically via its own service?
• No portal catalog: No portal catalog: – EPA “Window to my environment”EPA “Window to my environment”– Site location and type Site location and type
• Basic portal catalog: Basic portal catalog: – EDC IMS map feature serviceEDC IMS map feature service
– Expanded Site InfoExpanded Site Info
• Extensive portal catalog: Extensive portal catalog: – CUAHSI HISCUAHSI HIS– Expanded site Information including data period-Expanded site Information including data period-
of-recordof-record
slide 46
EPA “Window to my Environment”EPA “Window to my Environment”
• EPA WME does not maintain any USGS site information EPA WME does not maintain any USGS site information (catalog) local to the WME portal but uses NWISWeb as a (catalog) local to the WME portal but uses NWISWeb as a service to populate that information dynamically when a user service to populate that information dynamically when a user requests it.requests it.
• EPA WME submits a Lat/Long and Site-type request to EPA WME submits a Lat/Long and Site-type request to NWISWeb and then plots the site locations on-the-fly with links NWISWeb and then plots the site locations on-the-fly with links back to waterdata.usgs.gov to get the data.back to waterdata.usgs.gov to get the data.
slide 47
EPA “Window to my Environment”EPA “Window to my Environment”
• EPA WME does not maintain any USGS site information EPA WME does not maintain any USGS site information (catalog) local to the WME portal but uses NWISWeb as a (catalog) local to the WME portal but uses NWISWeb as a service to populate that information dynamically when a user service to populate that information dynamically when a user requests it.requests it.
• EPA WME submits a Lat/Long and Site-type request to EPA WME submits a Lat/Long and Site-type request to NWISWeb and then plots the site locations on-the-fly with links NWISWeb and then plots the site locations on-the-fly with links back to waterdata.usgs.gov to get the data.back to waterdata.usgs.gov to get the data.
• Advantages: USGS site information is always up-to-date and Advantages: USGS site information is always up-to-date and no operational overhead is incurred by having to maintain a no operational overhead is incurred by having to maintain a local copy of a catalog.local copy of a catalog.
• Disadvantages: Can only “Ask questions” of NWISWeb that Disadvantages: Can only “Ask questions” of NWISWeb that NWISWeb supports/understands.NWISWeb supports/understands.
slide 48
slide 49
CUAHSI HIS PortalCUAHSI HIS Portal
• CUAHSI HIS portal maintains an extensive catalog local to the CUAHSI HIS portal maintains an extensive catalog local to the portal detailing the location of measurement sites from many portal detailing the location of measurement sites from many academic and Federal agencies and what data are available at academic and Federal agencies and what data are available at each site allowing “data discovery”.each site allowing “data discovery”.
• HIS portal then provides web services for “data delivery” from HIS portal then provides web services for “data delivery” from the source entity/agency.the source entity/agency.
slide 50
CUAHSI HIS PortalCUAHSI HIS Portal
• CUAHSI HIS portal maintains an extensive catalog local to the CUAHSI HIS portal maintains an extensive catalog local to the portal detailing the location of measurement sites from many portal detailing the location of measurement sites from many academic and Federal agencies and what data are available at academic and Federal agencies and what data are available at each site allowing “data discovery”.each site allowing “data discovery”.
• HIS portal then provides web services for “data delivery” from HIS portal then provides web services for “data delivery” from the source entity/agency.the source entity/agency.
• Advantages: An extensive catalog local to the portal Advantages: An extensive catalog local to the portal potentially allows the portal to offer more sophisticated data potentially allows the portal to offer more sophisticated data discovery options if those options are not available directly discovery options if those options are not available directly from the source system.from the source system.
• Disadvantages: Once the catalog is cached at the portal it is Disadvantages: Once the catalog is cached at the portal it is instantly out-of-date and there is a continual maintenance instantly out-of-date and there is a continual maintenance overhead in refreshing the catalog information.overhead in refreshing the catalog information.
slide 51
CUAHSI initiativesCUAHSI initiatives
slide 52
CUAHSI HIS PortalCUAHSI HIS Portal
NWISWeb dump files “currently” supplied to populate NWISWeb dump files “currently” supplied to populate the CUAHSI portal catalog:the CUAHSI portal catalog:
618248 Apr 14 18:29 USGS_DV_DD_POR_list.gz618248 Apr 14 18:29 USGS_DV_DD_POR_list.gz
9127817 Apr 14 18:29 USGS_POR_list_0000-00-00_removed.gz9127817 Apr 14 18:29 USGS_POR_list_0000-00-00_removed.gz
10337889 Apr 14 18:29 USGS_POR_list.gz10337889 Apr 14 18:29 USGS_POR_list.gz
66484565 Apr 14 18:30 USGS_QW_PARAM_POR_list.gz66484565 Apr 14 18:30 USGS_QW_PARAM_POR_list.gz
40765012 Apr 14 18:29 USGS_QW_PGROUP_POR_list.gz40765012 Apr 14 18:29 USGS_QW_PGROUP_POR_list.gz
71660218 Apr 14 18:29 USGS_SITEFILE_list.gz71660218 Apr 14 18:29 USGS_SITEFILE_list.gz
109992 Apr 14 18:29 USGS_UV_DD_list.gz109992 Apr 14 18:29 USGS_UV_DD_list.gz
slide 53
NWISWeb Site Information ServiceNWISWeb Site Information Service
• The proposed NWISWeb “Site Information” service for The proposed NWISWeb “Site Information” service for FY11 will return this same site and data period-of-record FY11 will return this same site and data period-of-record information in XML directly -- either to populate/ refresh a information in XML directly -- either to populate/ refresh a portal catalog or dynamically respond to a portal user’s portal catalog or dynamically respond to a portal user’s request.request.
• The NWISWeb Site Information service will also output The NWISWeb Site Information service will also output KML and so can be used directly by the public (and USGS) KML and so can be used directly by the public (and USGS) to support mapping applications.to support mapping applications.
• The “goal” is to make the Site Information service OGC The “goal” is to make the Site Information service OGC “Web Feature Service” as compliant as possible.“Web Feature Service” as compliant as possible.
slide 54
slide 55
Data Integration / InteroperabilityData Integration / Interoperability
• The interoperability of services depends both on the way The interoperability of services depends both on the way specific services are called as well as the format / schema of specific services are called as well as the format / schema of the information returned. the information returned.
• The Open Geospatial Consortium (OGC) has become the lead The Open Geospatial Consortium (OGC) has become the lead international organization for the development of geospatial international organization for the development of geospatial standards and data-sharing protocols.standards and data-sharing protocols.
• The recently released (Aug 2010) NWISWeb Instantaneous The recently released (Aug 2010) NWISWeb Instantaneous Values service returns data in the CUAHSI developed Values service returns data in the CUAHSI developed WaterML1.1 schema but does not fully comply with the CUAHSI WaterML1.1 schema but does not fully comply with the CUAHSI “Water One Flow” services naming conventions.“Water One Flow” services naming conventions.
slide 56
http://waterservices.usgs.gov/WOF/InstantaneousValues?location=06018500&variable=00060&period=P1D
slide 57
Data Integration / InteroperabilityData Integration / Interoperability
• The interoperability of services depends both on the way The interoperability of services depends both on the way specific services are called as well as the format / schema of specific services are called as well as the format / schema of the information returned. the information returned.
• The Open Geospatial Consortium (OGC) has become the lead The Open Geospatial Consortium (OGC) has become the lead international organization for the development of geospatial international organization for the development of geospatial standards and data-sharing protocols.standards and data-sharing protocols.
• The recently released (Aug 2010) NWISWeb Instantaneous The recently released (Aug 2010) NWISWeb Instantaneous Values service returns data in the CUAHSI developed Values service returns data in the CUAHSI developed WaterML1.1 schema but does not fully comply with the CUAHSI WaterML1.1 schema but does not fully comply with the CUAHSI “Water One Flow” services naming conventions.“Water One Flow” services naming conventions.
• The NWISWeb IV service is NOT presently compliant with the The NWISWeb IV service is NOT presently compliant with the OGC “Sensor Observation Service” protocol but that is a goal OGC “Sensor Observation Service” protocol but that is a goal over time.over time.
slide 58
Data Integration / InteroperabilityData Integration / Interoperability
• NWISWeb services will move towards supporting OGC NWISWeb services will move towards supporting OGC protocols (WaterML 2, WFS, SOS) but will likely need to also protocols (WaterML 2, WFS, SOS) but will likely need to also support other protocols as well.support other protocols as well.
• The Center for Data Integration and Analysis headed by Nate The Center for Data Integration and Analysis headed by Nate Booth is taking the lead on collaboration with OGC including Booth is taking the lead on collaboration with OGC including working on OGC “Interoperability experiments” with working on OGC “Interoperability experiments” with international partners and actively contributing to standards international partners and actively contributing to standards and protocol development.and protocol development.
• CIDA also developed the USGS/EPA Water Quality web CIDA also developed the USGS/EPA Water Quality web services portal and is heavily involved in portal development services portal and is heavily involved in portal development efforts with other agencies as well.efforts with other agencies as well.
slide 59
slide 60
Questions?Questions?
http://waterdata.usgs.gov/
http://waterservices.usgs.gov/http://waterservices.usgs.gov/
[email protected]@usgs.gov
slide 61
slide 62
History of other NWIS “Web services”History of other NWIS “Web services”
• Interim USGS Web ServicesInterim USGS Web Services– August 2009August 2009
– All IV timeseries (public data only)All IV timeseries (public data only)
– Site information to service NWIS mappersSite information to service NWIS mappers
• Production USGS Instantaneous-values serviceProduction USGS Instantaneous-values service– August 2010August 2010
– All IV timeseries All IV timeseries
– WaterML 1.1, RDB, and JSON output formatsWaterML 1.1, RDB, and JSON output formats
• Production USGS Site-information serviceProduction USGS Site-information service– November(?) 2010November(?) 2010
– Service NWIS mappers and public KML/XML Service NWIS mappers and public KML/XML
– Service/refresh/update portal catalogsService/refresh/update portal catalogs
– OGC Web Feature ServiceOGC Web Feature Service
slide 63