The Fourth Paradigm - Deltares Data Science Day, 31 October 2014
-
Upload
microsoft-azure-for-research -
Category
Science
-
view
587 -
download
4
description
Transcript of The Fourth Paradigm - Deltares Data Science Day, 31 October 2014
![Page 1: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/1.jpg)
![Page 2: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/2.jpg)
![Page 3: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/3.jpg)
šš·š£
š·š”= āš»š + š» ā šÆ + š
![Page 4: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/4.jpg)
Data
Acquisition &
modelling
Collaboration
and
visualisation
Analysis &
data mining
Dissemination
& sharing
Archiving and
preserving
fourthparadigm.org
Data-intensive Research
![Page 5: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/5.jpg)
X-Info
ā¢ Data ingest
ā¢ Managing a petabyte
ā¢ Common schema
ā¢ How to organize it
ā¢ How to reorganize it
ā¢ How to share with others
ā¢ Query and Vis tools
ā¢ Building and executing models
ā¢ Integrating data and Literature
ā¢ Documenting experiments
ā¢ Curation and long-term
preservation
The Generic Problems
Experiments &Instruments
Simulations
Literature
Other Archives
facts
facts
facts
facts
Questions
Answers
![Page 6: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/6.jpg)
All Scientific Data Online
ā¢ Many disciplines overlap and use
data from other sciences.
ā¢ Internet can unify all literature and
data
ā¢ Go from literature to computation to
data back to literature.
ā¢ Information at your fingertips ā
For everyone, everywhere
ā¢ Increase Scientific Information
Velocity
ā¢ Huge increase in Science
Productivity
(From Jim Grayās last talk)
Literature
Derived and recombined data
Raw data
![Page 8: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/8.jpg)
![Page 9: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/9.jpg)
Manual Measurement
Automated Measurement
Sample Collection
Historical Photographs
Counting
Ubiquitous
Motes
Aircraft SurveysModel Output
Typing
![Page 10: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/10.jpg)
Monitoring
Collation
Quality assurance
Aggregation
Analysis
Reporting
Forecasting
Distribution
Done poorly,but a few notablecounter-examples
Done poorly to moderately,not easy to find
Sometimes done well,generally discoverable and available,
but could be improved
Integration
(I. Zaslavsky & CSIRO, BOM, WMO)
![Page 11: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/11.jpg)
![Page 12: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/12.jpg)
Web search:
āopen weather
data azureā
![Page 13: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/13.jpg)
![Page 14: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/14.jpg)
Water depth map of London(~130km2). Storm event of 60
minutes and 100 years return period
http://www.ncl.ac.uk/ceser/researchprogramme
/informatics/citycaturbanfloodmodel/
![Page 15: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/15.jpg)
![Page 17: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/17.jpg)
![Page 18: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/18.jpg)
![Page 19: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/19.jpg)
![Page 20: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/20.jpg)
![Page 21: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/21.jpg)
![Page 22: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/22.jpg)
Parker MacCready: Univ. of Washington
Rob Fatland:, Wenming Ye, Nels Oscar, Microsoft Research
![Page 23: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/23.jpg)
![Page 24: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/24.jpg)
Numerical model of 3-D ocean currents and water properties
ā¢ salinity,
ā¢ temperature,
ā¢ biogeochemistry
Relies on external data sources:ā¢ Bathymetry
ā¢ Wind and heating
ā¢ Open Ocean BCās
ā¢ Tides
ā¢ Rivers
![Page 25: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/25.jpg)
Model Validation Comparisons are done to an extensive suite of in-situ observations
ā¢ sea surface height
12 NOAA tide gauges
ā¢ salinity and temperature
over 2000 CTD casts from ECOHAB, RISE,
DOE, NANOOS, Hood Canal, IOS, King
County, and NOAA
ā¢ velocity and moored S,T
7 coastal ADCP / CTD moorings from the
ECOHAB and RISE projects, 2 moorings
from IOS
![Page 26: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/26.jpg)
Interactive 3-D Model Visualization using WorldWide Telescope, Narwhal and Layerscape
www.layerscape.org
![Page 27: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/27.jpg)
EH4 32 m
Figure from SA Siedlecki, UW/JISAO; Observations from Connolly et al., 2010
Validation: Dissolved Oxygen & Temperature
![Page 28: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/28.jpg)
LiveOcean: System Architecture
HPClinux 150 cores
ForecastNetCDF files
LiveOcean
Serverā¢ Post Processing
ā¢ Pre-make .png āviewsā
ā¢ Archive NetCDF files
ā¢ API for web sites
ā¢ Admin.js
ā¢ Client.jsBlob Storage:
Forecast Copy
Science UserpythonAzure Table:
Log Info
Admin
Website
Client Websitehttp://mappable.azurewebsites.
net/liveocean/
Rivers
USGS
Atmosphere
UW WRFOcean
HYCOM
![Page 29: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/29.jpg)
http://mappable.azurewebsites.net/liveocean
![Page 30: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/30.jpg)
![Page 31: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/31.jpg)
CloudBig data
Aggregation
MachineLearning Analytics
![Page 32: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/32.jpg)
The Cloud
democratizes
access to scale &
economies of scale
![Page 33: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/33.jpg)
Commodity at Scale
![Page 36: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/36.jpg)
![Page 37: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/37.jpg)
![Page 38: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/38.jpg)
![Page 39: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/39.jpg)
Research Cloud Ecosystem
![Page 41: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/41.jpg)
Use laptops &
desktop computers
Overwhelmed by
data
Finding analysis
ever more difficult;
sharing even
harder
www.azure4research.com
![Page 42: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/42.jpg)
![Page 43: The Fourth Paradigm - Deltares Data Science Day, 31 October 2014](https://reader034.fdocuments.net/reader034/viewer/2022052602/559de75d1a28aba75f8b472f/html5/thumbnails/43.jpg)