Status: For Information Only ESA UNCLASSIFIED - For Official Use Cloud Processing at ESA [EO Payload...

Status: For Information Only

ESA UNCLASSIFIED - For Official Use

Cloud Processing at ESA [EO Payload Ground Segment]

Cristiano Lopes, ESACEOS WGISS-4030/09/2015

Cristiano Lopes | CEOS WGISS-40 | 30/09/2015 | Slide 2

Overview – Cloud Computing Uses

• Cloud computing and Earth Observation:

Processing,

Archiving,

Distribution,

Discovery…

• Cloud computing uses in ESA EO Ground-Segment

Past activities

On-going activities

Future activities


Overview – Cloud Computing

• Cloud Computing definition: Cloud Computing, Wikipedia article

"Emerging Technologies/WGISS" Presentation, Slides 10-15, CEOS Plenary 28th

Cloud is simply outsourcing in a new form, Joe McKendrick, Forbes, Oct 18th 2014

• Traditional cloud service models: Infrastructure As a Service (IaaS)

Platform As a Service (PaaS)

Software as a Service (SaaS)

https://en.wikipedia.org/wiki/Cloud_computing

https://en.wikipedia.org/wiki/Cloud_computing

http://ceos.org/document_management/Meetings/Plenary/28/Side%20Mtg%20Emerging%20Technologies%20Side%20Meeting_Final.pdf

http://ceos.org/document_management/Meetings/Plenary/28/Side%20Mtg%20Emerging%20Technologies%20Side%20Meeting_Final.pdf

http://www.forbes.com/sites/joemckendrick/2014/10/18/cloud-may-be-the-new-outsourcing-but-the-same-due-diligence-must-apply/


Overview – Cloud

“Cloud services” are simply ICT resources that are hosted and managed by someone else and offered as a service (XaaS).


Distribution – Content Delivery Network

EO Data Distribution over Content Delivery Network (CDN):• CDN provides dynamic geographical load balancing (local caching);

• GOCE Gravity field data, distributed over CDN with good success (5 releases);

• ENVISAT AND ERS SAR data, also distributed with low success (due to data specificities – on-demand data)

Plus:• Very high user satisfaction – download performance,

• Transparent caching of data at different locations,

• Peak loads not affecting ESA infrastructure

Minus:• Expensive,

• Only for “stable” and consolidated data

Currently not in use – replaced by a managed hosting service.


Infrastructure – Cloud Deployment

EO Data Handling Infrastructure deployment in Cloud Provider:• Full deployment of existing infrastructure in a Cloud/Virtualised Environment;

• “Porting” / Deployment of existing applications as-is (templates): Archive, Processing Management, Distribution, Circulation, Discovery, etc.

• Initial activities pilots led to full reference and integration platforms deployed in the “Cloud”.

Plus:• Very easy to deploy existing applications;

• No noticeable functional issues;

• Very Flexible – One environment per need (one mission, one project);

• Pay-per-use (when needed) – minimises costs (resources are only used when needed).

Minus:• IPR / License issues;

• Performance impacts (when compared with physical infrastructure, 10x in some cases);

• Security (Policy, very difficult to demonstrate full compliance for production platforms).

Results led to follow-up activities.


Processing – Cloud Deployment

EO Data Processing Pilots/Trials with commercial Cloud Provider:• Virtual servers allocated in the cloud – Scheduler and Processing Algorithm. Data

transferred off-line via media;

• ENVISAT MIPAS Reprocessing – 200 VM, 400 CPUs for 5 weeks. 8TiB input, 600GB output.

• ENVISAT ASAR NRT Processing – 10 VM, 20 CPU; runs with 20 and 50 input products taking ~20 and ~45 minutes.

Plus:• Very easy to deploy existing processing algorithms;

• No noticeable data quality issues;

• Easily scalable (linked to Provided availability).

Minus:• Relative Expensive (compared to owning hardware);

• Static infrastructure (full environment started and stopped manually – templates used for processing algorithms);

• Data transfers cumbersome (Disk In, Network out – costs).



Infrastructure–Managed Hosted Services

DISSHARM Project and Platform:• Managed hosting service for ESA EO Data Distribution;

• Single-Tenant virtualised environment with dedicated storage;

• Located within the ESA EO WAN Backbone and close to its data entry;

• Currently in Production.

Plus:• Integrated in ESA EO WAN (data travels locally);

• Managed as a service (Built-to requirements; Provider responsible for architecture, security);

• Bandwidth (currently 2Gbps, scalable to 10Gbps contractually).

Minus:• Limited resources (VMs, storage);

• No local data redundancy (cost reduction choice);



Infrastructure–Managed Hosted Services Evolution

ELVIS Platform DISSHARM Evolution:• Managed hosting service for ESA EO Data Distribution & Processing;

• Single-Tenant virtualised environment (in this case with more processing resources) with dedicated storage, with a multi-tenant cloud connection (Cloud Bursting);

• Access to the Provider’s commercial cloud via internal LAN;

• Currently on early stages of production (cloud access still not present).

Example use case - SAR On-the-fly Processing:

• Process SAR data (ERS, ENVISAT, ALOS) on download request by user;

• Perform processing on the back-end transparently to the user;

• Cache output product for next user ;

• Cloud bursting as needed by demand (to be implemented).


Processing - Hosted Processing

Super Sites Exploitation Platform and Project (SSEP) :• Collaborative environment for end-user’s data exploitation, through

coordinated access to data, tools and processing power;

• Virtual workstation on a cloud platform;

• Integrated information sharing and support tools;

• Demonstrated on Helix-Nebula.

Plus:• Change in paradigm (bring users to data);

• Abstract ICT from users (transparently multi-sourced resources);

Minus:• IPR / License issues (data, tools);

• Cloud-API Nightmare (different cloud providers using different APIs).


http://www.helix-nebula.eu/


EO Exploitation Platforms

Earth Observation Exploitation Platforms• Follow to the SSEP concepts

• On-going activities for Thematic Exploitation Platforms (TEP):

Geohazards (SSEP follow-up),

Coastal, Forestry, Hydrology, Polar and Urban

• TEP activities are industry driven, with input from ESA on overall architecture:

Open source,

Standards,

Infrastructure Independence

Earth Observation Platform - Test bed (Pilot):

• Open Tender: Solution defined by provider and offered as a service;

• Hosted processing only (no download of source data);

• Hosting Sentinel-2, Landsat and ENVISAT MERIS data;

• ESA provide free (pre-paid) access to limited test and science users


Lessons learned, challenges, opportunities

Cloud Services/Technology

• Lessons learned:

New solutions for old problems (e.g. Processing, Distribution):

• Challenges:

Technology maturity (Numerous APIs, different SLAs, reliability of suppliers)

ESA Industrial Policy Contract placing;

Security Compliance with ESA Security Rules.

• Opportunities

Neutral Environment with open access to anyone, one place to develop, prototype and run applications;

Potentially unlimited resources allowing for new ways to exploit EO data (big data);

Space to define an EO Cloud Architecture.


Cloud Processing Framework (1)

Processing of EO Data in the Cloud.

• Challenge:

Multiple data sources;

Multiple processing algorithms;

Multiple possibilities of joining processing algorithms;

Quasi-Unlimited resources to control.

• Opportunity for defining standards and best practices:

• From existing multiple solutions;

• From existing or new standards;

• Tailored to the EO Data Specific Needs.


Cloud Processing Framework (2)

Processing of EO Data in the Cloud.• Potential domains:

• Data Discovery

Where Opensearch?

• Data access

How OpenDap?, WCS?, HTTP GET [from Object Storage]?

• Resource management

How to provision Cloud Resources Cloud Scheduler?, Apache jClouds/libCloud?

How to do VM Management Condor?, Torque?, Grid(any)?

• Workflow management (Algorithms)

How to “chain” algorithms BEPL?, XPDL?


Cloud Processing – CERN ATLAS.

Learning from others The ATLAS experiment driven by CERN.

“The Evolution of Cloud Computing in ATLAS”, Taylor et all, ATL-SOFT-PROC-2015-049, 27/05/2015.


Summary

• In ESA Earth Observation, a number of activities related to cloud technology and services have been performed.

• The Cloud is an enabler that allows ESA to perform is data exploitation activities in a more efficient way.

• We believe there is room for collaboration and cooperation in the use of the Cloud with other partners.

• One such possibility of collaboration is the definition of standards, best-practices and an architecture for EO Data Exploitation.

• The Cloud Processing Framework is a first example.


Thank you!

Cristiano Lopes

Ground Segment System Engineer

Earth Observation Programmes

Phone: +39 06 941 80735

Email: [email protected]

Status: For Information Only ESA UNCLASSIFIED - For Official Use Cloud Processing at ESA [EO Payload...

Documents

Transcript of Status: For Information Only ESA UNCLASSIFIED - For Official Use Cloud Processing at ESA [EO Payload...