Continuous Integration and Deployment Best Practices on AWS - AWS Symposium 2014 - Washington D.C.
Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014
-
Upload
amazon-web-services -
Category
Technology
-
view
369 -
download
1
description
Transcript of Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014
![Page 1: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/1.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Big Open Data: Transformation through Public Data Sets
Ariel [email protected]
![Page 2: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/2.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Agenda• Open Data Overview (Ariel Gold, AWS)• NASA NEX (Dr. Tsengdar Lee, NASA)• Data Distribution Model (Ariel Gold, AWS)• Q&A
![Page 3: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/3.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
![Page 4: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/4.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
What is Open Data?
Data released to the public in ways that make it easy to
discover, access, and use.
![Page 5: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/5.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Why are governments making their data open?
Five main goals seen across different policies and programs.
![Page 6: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/6.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Reduced backlog in manual submissions while improving accuracy and reducing costs
Enabled ingestion of new open data sources to improve mission delivery
Shifted from quarterly reports to developer-focused APIs, raw data, and documentation
![Page 7: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/7.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Release Definition & Policy Launch Open Data CatalogDo Things w/ Open Data
Release New (Big) Open Data
Repeat w/ Increased Focus
Impact
Gather Customer Feedback
Open Data Adoption Curve
![Page 8: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/8.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
How do you drive impact with (big) (open) data?
Build communities and tools around sustainable open data sets.
![Page 9: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/9.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
OpenNEX as an Open Data Analytical Service
Tsengdar Lee, [email protected]
![Page 10: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/10.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
NASA EARTH EXCHANGE (NEX)
Virtual collaborative-for-global-change scienceCOLLABORATION
Web & System
COMPUTINGNASA HECC
Pleiades 180k+ coresEndeavour 1.5k+ cores, 6TBD-wave Quantum 512 Qbit
DATA REPOSITORY800 TB, 126 PB
network links
![Page 11: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/11.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
NEX Provides a Complete Work Environment “Science as a Service”
COLLABORATION
Over 400 Members
CENTRALIZED DATA REPOSITORY
Over 1300 TB of Data
COMPUTINGScalable Diverse Secure/Reliable
KNOWLEDGE
WorkflowsMachine ImagesRe-useable software
![Page 12: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/12.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
• 33 CMIP5 models available
• All 4 RCPs (2.6, 4.5, 6.0, 8.5) and historical runs
• 30 arc-second (800m) spatial resolution, monthly time-step, 1950-2099
• Max/min temperature and precipitation
• Statistical downscaling (bias-corrected spatial disaggregation; Maurer et al., 2007)
Thrasher et al., 2013, Eos
Creating NEX-DCP30 Downscaled Climate Projections at 30 Arc-Second Resolution for the National Climate Assessment
![Page 13: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/13.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
• Sam Goward, PI
• Detecting 30m
resolution forest disturbance annually • National runs complete• Data
distribution being worked
North American Forest Dynamics (NAFD)
13
![Page 14: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/14.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
April 2010
October 2010
• David Roy, PI• Completed processing 140,000 scenes for first year of global Landsat (monthly and annual)• Public release later in June
Web-Enabled Landsat Data (WELD)
14
![Page 15: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/15.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
California Drought Monitoringat 250m Scale
Changes in High Latitude
Ecosystems - Greening
![Page 16: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/16.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
A tale of two droughts/Amazon 2005 & 2010
TRMM MODIS
Workflows Improving Research Productivity
Faster (24 months vs. 6 months), consistent (same analytical methods, quality flags) and reproducibleSamantha et al., GRL, 2011
Xu et al., GRL, 2011
![Page 17: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/17.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
NEX and AWS
Sandbox on Public Cloud- Prototyping - Development
NAS HECC- Large-Scale processing- NCA Example
COLLABORATION
(more public members)
COMPUTINGHECC + AWS1
DATA REPOSITORY(AWS OpenData, with
“Market driven” Cache)
AWS
![Page 18: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/18.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
2014 International Space Apps Challenge
46 countries participated 735 virtual participants 69 virtual team projects
Space Apps kicked off Thursday night in Doha, Qatar at midnight and ended in Seattle, WA at 6 p.m. Pacific—76 hours of around-the-world
hacking on NASA data.
![Page 19: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/19.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
2014 International Space Apps: Most Popular Challenges
Earth Watch: Where on Earth (44 projects), Climate & Neighborhood (22), Cool It (21)
Robotics: Exomars Rover is My Robot (42)Asteroids: Asteroid Prospector (38)Space Tech: Space Wearables (25), Alert-Alert (24)Space Flight: Growing Food for A Martian Table (22), SpaceT (20)
![Page 20: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/20.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
2014 International Space Apps:OpenNEX Data Challenges
Climate and My Neighborhood: 22 projects
Climate Alert: People’s Choice Local Award Winner
Climatehood: Best Use of Data Local Award Winner
Ways to Die: Most Inspiring Local Award Winner
NASA-NEX-Climate-Changes: People’s Choice Local Award Winner
![Page 21: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/21.jpg)
21
OpenNEXWorkshops and Challenges
7/1 – 11/15/2014• Virtual Workshop teaches how to use OpenNEX
AMIs and Public Data sets on AWS.• Collaborate within arms length
- Users work with vm templates including prepackaged tools, they instantiate, extend and own accordingly.
- Core data is read only public data sets on S3• Bring citizen scientists together around data,
services and knowledge• Extract more value out of assets, i.e. data and
knowledge, via the group • http://nex.nasa.gov/OpenNEX
![Page 22: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/22.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Thank You
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Thank YouTsengdar Lee, Ph.D.
![Page 23: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/23.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
How do you drive impact with (big) (open) data?
Build communities and tools around sustainable open data sets.
![Page 24: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/24.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
National Agriculture Imagery Program
![Page 25: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/25.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Before• Pay $5k • Wait weeks to get data on disks • Technical and GIS expertise to
make data usable
After• No or minimal cost• Get access immediately • No technical or GIS expertise to
start using
Significantly increase openness of a 48TB aerial imagery data set at a low and sustainable cost ($1,600/month)
National Agriculture Imagery Program
![Page 26: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/26.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Managing the Cost of Data Distribution
• Amazon Simple Storage Service (S3) “requestor pays”
• Eliminate variable costs• Multiple configuration options
depending on your use case
![Page 27: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/27.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Insert NAIP Demo Slides (or link to live pages)
![Page 28: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/28.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Q & AFor more information on AWS and open data, visit:
http://aws.amazon.com/government-education/open-data/
![Page 29: Big Open Data Transformation Through Public Data Sets - AWS Washington D.C. Symposium 2014](https://reader037.fdocuments.net/reader037/viewer/2022103109/540cd5c07bef0ad7288b6d33/html5/thumbnails/29.jpg)
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
AWS Government, Education, and Nonprofits Symposium Washington, DC | June 24, 2014 - June 26, 2014
Thank YouAriel Gold