EVALUATION OF DIFFERENT CLOUD COMPUTING RESOURCES … · cloud-computing platform, provided access...
Transcript of EVALUATION OF DIFFERENT CLOUD COMPUTING RESOURCES … · cloud-computing platform, provided access...
E VA L U AT I O N O F D I F F E R E N T C L O U D C O M P U T I N G R E S O U R C E S F O R L A R G E S C A L E S AT E L L I T E D ATA P R O C E S S I N G
P O T E N T I A L A N D L I M I TAT I O N
D R . T U O N G - T H U Y V U
B I G D ATA
E A R T H O B S E R VAT I O N
S AT E L L I T E I M A G E D ATA
C L O U D C O M P U T I N G
B I G D ATA
Volume Variety Velocity
analysis, capture, data curation, search, sharing, storage, transfer, visualisation, querying,
updating and information privacy.
CHALLENGE
E A R T H O B S E R VAT I O N S AT E L L I T E D ATA
L A N D S AT 5
L A N D S AT 7
L A N D S AT 8
160 MB (compressed) 366 MB (uncompressed)
900 MB (compressed) 1.6 GB (uncompressed)
1 GB (compressed) 2 GB (uncompressed)
1 scene
E A R T H O B S E R VAT I O N S AT E L L I T E D ATA
RapidEye
PlanetLab
Landsat
Sentinel-2
Multi-sensors
GeoEye
E A R T H O B S E R VAT I O N S AT E L L I T E D ATA
RapidEye
PlanetLab
Landsat
Sentinel-2
Multi-sensors
Ground-truth
GeoEye
E A R T H O B S E R VAT I O N S AT E L L I T E D ATA
RapidEye
PlanetLab
Landsat
Sentinel-2
Multi-sensors
Ground-truth
Modelling
GeoEye
E A R T H O B S E R VAT I O N S AT E L L I T E D ATA
RapidEye
PlanetLab
Landsat
Sentinel-2
Multi-sensors
Ground-truth
Modelling
Spatial analysis
….
GeoEye
Building Petabyte Data Production Systems with the NASA Earth Exchange
(http://www.nas.nasa.gov/SC15/demos/demo22.html)
• Production pipeline for Landsat and NASA Global Imagery and Browse Service – 37 steps, 7 petabytes of data, and 670,000 processor hours.
• NEX Downscaled Climate Projections (NEX-DCP30) and NEX Global Daily Downscaled Projections (NEX-GDDP) pipeline – 40 terabytes (TB) of data and 320,000 processor-hours.
• Estimating biomass by counting trees across the United States by using machine learning and computer vision – 40 TB of data/100,000 compute hours.
C L O U D C O M P U T I N G
… a service over the Internet
I N F R A S T R U C T U R E ( I A A S )
P L AT F O R M ( PA A S )
S O F T WA R E ( S A A S )
R E M O T E S E N S I N G D A T A S T O R E D O N T H E C L O U D - C O M P U T I N G P L A T F O R M , P R O V I D E D A C C E S S V I A W E B P O R TA L , D O W N L O A D A B L E
C U R R E N T T R E N D
S O F T W A R E C O M PA N I E S P R O V I D E D P R O C E S S I N G S E R V I C E S O N T H E C L O U D
A G E N C I E S P R O V I D E C L O U D - B A S E D P R O C E S S I N G T O O L S
L A N D S AT O N AW S
D I G I TA L G L O B E
E S R I C L O U D
E N V I C L O U D
E S A C L O U D T O O L B O X
N A S A E A R T H E X C H A N G E
R E M O T E S E N S I N G D A T A S T O R E D O N T H E C L O U D - C O M P U T I N G P L A T F O R M , P R O V I D E D A C C E S S V I A W E B P O R TA L , D O W N L O A D A B L E
C U R R E N T T R E N D
S O F T W A R E C O M PA N I E S P R O V I D E D P R O C E S S I N G S E R V I C E S O N T H E C L O U D
A G E N C I E S P R O V I D E C L O U D - B A S E D P R O C E S S I N G T O O L S
I N C R E A S I N G R E S E A R C H O N S PAT I A L D ATA P R O C E S S I N G S E R V I C E S O N T H E C L O U D
L A N D S AT O N AW S
D I G I TA L G L O B E
E S R I C L O U D
E N V I C L O U D
E S A C L O U D T O O L B O X
N A S A E A R T H E X C H A N G E
The most open platform so far …
N A S A E A R T H E X C H A N G E
• A collaborative compute platform • Data • Tools & Libraries • Models • Disk storage
• Pleiades 1.3 PB total • Sandboxes: lnxsrv105: 163 TB
total and externally accessible sandbox 305TB
D ATA S E T
Landcover Climate Datasets
MODIS Land MODIS Atmosphere
Landsat 5 Landsat 7 Landsat 8
Landsat GLS Web-Enabled Landsat
ASTER TRMM AVHRR
Ecosystem Modeling Datasets DEM Soils Aerial
M O D E L S
Ecosystem Models • Terrestrial Observation and Prediction System (TOPS) North America • Biome-BGC USA • Lund-Potsdam-Jena Dynamic Global Model (LPJ) Global
Climate Models • The Weather Research and Forecasting (WRF) Model Regional
T O O L S
Analytical Tools MATLAB, IDL/ENVI,
R for statistical computing and graphics Numpy and SciPy for Python
Matplotlib for plotting and graphics GNU Octave
Database Management Systems MySQL, SQLite, MongoDB
Data Manipulation Tools and Libraries MODIS Reprojection Tool (MRT)
MODIS Reprojection Tool for Swath (MRTSwath) HDF-EOS To GeoTIFF Conversion Tool (HEG)
The Land Data Operational Products Evaluation (LDOPE) Tools PROJ.4 Cartographic Projections Library
GDAL - Geospatial Data Abstraction Library NetCDF (Network Common Data Form) tools and libraries
….
• a computing platform
• for petabyte-scale scientific analysis and visualisation of geospatial datasets
C O D E E D I T O R E X P L O R E R C L I E N T L I B R A R I E S
web-based IDE for writing and running scripts
a lightweight web app for
exploring our data catalog and running simple
analyses
Python and JavaScript
wrappers around our web API
D ATA S E T
Landsat
Sentinel
MODIS
1m resolution aerial image data US National Agriculture Imagery Program
(NAIP)
Other sub-meter resolution sample images
SRTM 30-m
Landcover MODIS, MERIS
Surface temperature
Climate & Weather
World population
T O O L S
Most of tools you need are here
Image Math operation, convolutions, edge detection, spectral transformation, object-based methods, compositing & mosaicking, etc.
Reducer Image reductions, statistics, raster-to-vector and vector-to-raster conversion, linear regression
Supervised classification Landsat algorithms Sentinel-1 algorithms
Chart Array …..
A C C E S S I B I L I T Y & L I C E N S E
NASA/NDC or OpenID account, need to signup and be approved
NEX sandbox AWS, GEE, Azure, etc.
A C C E S S I B I L I T Y & L I C E N S E
https://earthengine.google.com/signup. Google evaluates the request
Free for research, education, and nonprofit use.
• Can import your own data, download data & images to work with your own tools
• You own your written algorithms and results on Google EE
NASA/NDC or OpenID account, need to signup and be approved
NEX sandbox AWS, GEE, Azure, etc.
• Cloud-computing platforms/resources are currently available
• Google Earth Engine is growing very fast, both basic and advanced remote sensing processing tools are available, free access
• NEX is more than just a computing platform, is a community and knowledge based system
• Cloud-computing platforms/resources are currently available
• Google Earth Engine is growing very fast, both basic and advanced remote sensing processing tools are available, free access
• NEX is more than just a computing platform, is a community and knowledge based system
Ground truth?
Regional, local context?
Users (ready?)
Crowd-2-Cloud
CROWD-SOURCINGREMOTE SENSING
PROCESSING SERVICES ON CLOUD COMPUTING
Training
Validation
Applications
URBAN
AGR ICULTURE
E C O L O G Y
PUBL IC HEALTH
D ISASTER
• Remote sensing processing services on the cloud
• Disaster recovery monitoring • Damage mapping
• Urban green space
• Urban construction monitoring
• Megafauna index
BIG DATA ANALYTICS
• Crowd-sourcing data quality assessment
• Oil palm plantation management
• Health care delivery • Disaster Health
on-demand, scalable processing capabilities efficiently co-located with large image databases rapid deployment of new algorithms sharing, collaborative platform
CLOUD COMPUTING PLATFORM
on-demand, scalable processing capabilities efficiently co-located with large image databases rapid deployment of new algorithms sharing, collaborative platform
CLOUD COMPUTING PLATFORM
Promote open resources (data, tools, etc.) NEX and GEE
on-demand, scalable processing capabilities efficiently co-located with large image databases rapid deployment of new algorithms sharing, collaborative platform
CLOUD COMPUTING PLATFORM
Potential or limitation depends on user’sOBJECTIVES
TECHNICAL LEVELS
Promote open resources (data, tools, etc.) NEX and GEE
on-demand, scalable processing capabilities efficiently co-located with large image databases rapid deployment of new algorithms sharing, collaborative platform
CLOUD COMPUTING PLATFORM
Potential or limitation depends on user’sOBJECTIVES
TECHNICAL LEVELS
Promote open resources (data, tools, etc.) NEX and GEE
PaaS and SaaS
potential to deploy to various SaaS on demand
on-demand, scalable processing capabilities efficiently co-located with large image databases rapid deployment of new algorithms sharing, collaborative platform
CLOUD COMPUTING PLATFORM
Potential or limitation depends on user’sOBJECTIVES
TECHNICAL LEVELS
Promote open resources (data, tools, etc.) NEX and GEE
PaaS and SaaS
potential to deploy to various SaaS on demand
Many other infrastructures, platforms were not discussed here including ESA, AIST (Japan), other national programmes
T H A N K Y O U