Post on 31-Dec-2015
www.egi.eu
European Grid Initiative
www.egi.eu
Federated Cloud update
Peter solagnapeter.solagna@egi.eu
Pre-GDB Workshop
04/19/23 .... 1
www.egi.eu
Principles of federation
.... 2
HardwareHardwareHardwareHardware
HardwareHardwareHardwareHardware
HardwareHardware
Cloud ManagementCloud ManagementCloud ManagementCloud Management
Cloud ManagementCloud ManagementCloud ManagementCloud Management
Cloud ManagementCloud Management
User CommunitiesUser Communities
User CommunitiesUser Communities
User CommunitiesUser Communities
Federated interfaces Federated interfaces
Federated services
Federated services
• Standards and validation: Recommended and common open standards for the interfaces and images – OCCI, CDMI, OVF, GLUE2.
• Resource integration: Cloud Computing to be integrated into the existing production infrastructure.
• Security coordination, operations framework
• Heterogeneous implementation: no mandate on the cloud technology. Operational integration of the interfaces.
www.egi.eu
Services offered
• Service deployment
• HTC over cloud
• Heavy memory applications
• Different access levels: IaaS,PaaS, SaaS• There are already several tools integrated• Continuous activity of integrating more platforms
in the federated cloud
.... 3
www.egi.eu
EGI Cloud infrastructure
.... 4
EGI Core Platform
Federated AAI
ServiceRegistry
Monitoring Accounting
EGI Cloud Infrastructure Platform
Instance Mgmt
Information
Discovery
Cloudinit
Cloudinit
Storage Manageme
ntCDMICDMI
Cloud Management Stack(OpenStack, OpenNebula, Synnefo, …)
Help and Support
Security Co-
ordination
Training and
Outreach
EG
I App
licat
ion
DB
Imag
e R
epos
itory
EG
I Clo
ud S
ervi
ce M
arke
tpla
ce
Sustainable
Business Models
OCCIOCCI
www.egi.eu
VM Management
• Uniform VM Management through OCCI• Plugins/interfaces for Openstack
Opennebula and Synnefo
• Cloud init• Contextualization tool supported Extending
this capability with other configuration tools such as puppet
.... 5
www.egi.eu
Expanding the interfaces supported
• Standard interfaces allow uniform behavior across providers• EGI invested on the common interfaces from the very beginning and is
committed to continue the support to those interfaces also in EGI-Engage
• Top down approach, but this built momentum for the federation
• EGI technological evolution is also user-driven• User communities are already using native interfaces in their
workflows• EGI will support user requirements for the operational integration of
the interfaces• This can be hard work (easy for O.Stack, less easy for O.Nebula), need for
clear use cases from the user communities
• Steps for operational integration• Monitoring of availability• Accounting of usage• Support user authn/authz in the federation
.... 6
www.egi.eu
VMI Management
• A single point for control for the management and control of VO specific application packages across clouds
• Catalogue of available Software Appliances (AppDB)• Support for VO-wide image lists following the HEPiX format • Support requirement for endorsed images only to be distributed• Integration with the EGI Information system• Extended authentication system to support federated AAI through
eduGAIN• Integration with external metadata repositories to provide broader
support to communities and engage with other activities• Automated processes for VMI distribution
• Security policy for VMI endorsement being updated by EGI SPG
.... 7
www.egi.eu
Accounting
• New version of usage record for cloud resources under implementation• Including also VM Images ID
• Publishing scripts have been debugged to report correct usage values
• Accounting probes can report also cloud usage not performed through OCCI interfaces• Depending on the middleware the information can be
more or less complete (user information, VO, …)
• Normalization is still an open issue: collaboration is more than welcome!
.... 8
www.egi.eu
Monitoring
• Currently monitored capabilities:• Standard interfaces: OCCI/CDMI• Accounting• Integration with AppDB: vmcatcher
• Availability calculation engine ARGO allows flexible profiles • Production A/R calculation engine from mid 2015• Will allow easier introduction of custom A/R profiles
• Introduce monitoring for native interfaces• Pros: a lot of probes already available from other
communities. Allow certification for sites exposing only native interfaces
• Cons: manage the credentials used for monitoring
.... 9
www.egi.eu
X509 support
• X509 is still the federated AuthN infrastructure that works™
• X509 support implemented in Keystone, to be used with native OpenStack interfaces and OCCI
• OpenNebula supports X509 only through OCCI
• In EGI-Engage, Cloud services will lead the extension of EGI AAI framework to support new authentication technologies
.... 10
www.egi.eu
Improve user tracking
.... 11
• Most of the science gateways are using robot certificates to generate short-term proxies for the users
• Use of robot certificates proxies can be extended to improve accounting and user tracking
Robot Certificate info VO Information
The same for every user of the gateway
User UIDUser UID
The UID is provided by the science gateway. The user will have the same UID using
different science gateways. It’s a per-user sub-proxy
The UID is provided by the science gateway. The user will have the same UID using
different science gateways. It’s a per-user sub-proxy
www.egi.eu
How big research infrastructures can benefit from the EGI Fedcloud experience?
.... 12
Full integration in the EGI Production infrastructure
Accounting
GOCDB
Monitoring
VMI Management