EGI: European Grid Infrastructure
description
Transcript of EGI: European Grid Infrastructure
![Page 1: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/1.jpg)
EGI: European Grid InfrastructureSteven Newhouse
Interim EGI.eu DirectorEGI-InSPIRE Project DirectorTechnical Director EGEE-III
1
![Page 2: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/2.jpg)
European e-Infrastructure• European Data Grid (EDG)
– Explore concepts in a testbed• Enabling Grid for E-sciencE (EGEE)
– Moving from prototype to production• European Grid Infrastructure (EGI)
– Routine usage of a sustainable e-infrastructure
2
![Page 3: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/3.jpg)
What is e-Infrastructure?• Resources linked by high speed networks
– Compute, Storage, Instruments, ...• Controlled access to shared resources
– Authentication, Authorisation, Accounting, ...• Dependable services for others to use
– Driven by availability and reliability metrics• Services that are there for the long-term
– Supporting experiments lasting decades
Swiss Grid Day 3
![Page 4: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/4.jpg)
The EGEE-III Project17,000 users139,000 LCPUs (cores)25Pb disk39Pb tape
12 million jobs/month+45% in a year
268 sites+5% in a year
48 countries+10% in a year
162 Virtual Organisations+29% in a year
Over 20 active communities in 112 VOs
![Page 5: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/5.jpg)
EGICollaboration
NGI
NGI
NGI
NGI
ResearchCommunity
ResearchCommunity
ResearchCommunity
ResearchCommunity
EIRO
EIRO
EGI.eu
ResearchCommunity
5
![Page 6: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/6.jpg)
EGI.eu• Coordination for European Grid resources
– Roadmap to integrate HTC, HPC, Data, Instruments, ...– Policy & services needed to run a grid
• Governance & ownership by its stakeholders– EGI Council votes proportional to GDP– EGI Council fees proportional to votes– Sustainable small coordinating organisation (EGI.eu)– Builds on resources from within its stakeholders
• Location selected to be Amsterdam– Distributed staff with a core (~50%) in Amsterdam– Approximately 40 staff, €3.5M/year
Swiss Grid Day 6
![Page 7: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/7.jpg)
EC Funding
National Funding
Relationship betweenNGIs & EGI
Swiss Grid Day 7NGI National Tasks (NGI)
NGIInternational
Tasks (EC & NGI)NGI National Tasks (NGI)
NGIInternational
Tasks (EC & NGI)NGI National Tasks (NGI)
NGIInternational
Tasks (EC & NGI)
EGI.euEGI Global Tasks (EC, NGI, EGI.eu)
![Page 8: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/8.jpg)
EGI means Innovation• Deploy Technology Innovation
– Distributed Computing continues to evolve• Grids Desktops Virtualisation Clouds ?
• Enable Software Innovation– Provide reliable persistent technology platform
• Today: Tools built on gLite/UNICORE/ARC• Support Research Innovation
– Infrastructure for data driven research• Support for international research (e.g. ESFRI)
Swiss Grid Day 8
![Page 9: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/9.jpg)
Technology Innovation• Will come from outside EGI
– EGI runs DCI technologies in production• Partnership with technology projects
– EMI (European Middleware Infrastructure)• Continued evolution of gLite, ARC, UNICORE
– RESERVOIR• Public cloud functionality to ERA
– StratusLab (Proposed)• Use RESERVOIR to support gLite environments
Swiss Grid Day 9
![Page 10: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/10.jpg)
Swiss Grid Day 10
Prototype
Requirements
Implementation
Usage
Release
Evaluate
Deployment
Feedback
Collaboration
Technology innovations taking place outside the production infrastructure
Innovating technology being deployed within the production infrastructure
NGIs & EIROs
EC &National
ResearchProjects
Release
![Page 11: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/11.jpg)
Software Innovation• Will come from outside EGI
– EGI is a neutral platform for applications• EGI cannot support all services in its core
– Every community needs something different• Foster innovation within ‘sectors’
– High Throughput Computing– Digital Libraries– ...
Swiss Grid Day 11
![Page 12: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/12.jpg)
EGEE Maintained Components External Components
Example of Deployed Services
12Physical Resources
General Services
LHC FileCatalogue
HydraWorkload
Management Service
File TransferService
Logging &Book keeping
Service
AMGA
Storage Element
Disk Pool Manager
dCache
Information S
ervices
BDII
MON
User InterfaceUser Access
SecurityServices
Virtual Organisation Membership
Service
Authz. Service
SCAS
Proxy Server
LCAS & LCMAPS
Compute Element
CREAM LCG-CE
gLExec
BLAH
Worker Node
User Interface
BDII
CORE UMD SITE SERVICES (TBC)
COMMUNITY SERVICES
RESPECT USER
Deployed by EGIMaintained & Supported
through EGI
Deployed by EGIMaintained &
Supported by the community
![Page 13: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/13.jpg)
Digital Library Community• Community specific software stack
– gCube from D4Science project• VREs: Manage access to libraries
– Big focus on usability
Swiss Grid Day 13Physical Resources
Core UMD Services
Information Service
Virtual Research
Environment (VRE)
Manager Service Container
VREServices
VREServices
![Page 14: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/14.jpg)
High Throughput Computing• Managing large collections of jobs & files
– HEP, LS, ...• Problem specific Applications and Tools
– Maintained & supported by that community
Swiss Grid Day 14Physical Resources
Core UMD Services
GANGA DIANE
HTC Community Services (e.g. WMS, FTS, LFC , AMGA, ...)
User ApplicationsWISDOM
![Page 15: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/15.jpg)
Research Innovation• An infrastructure to support the ERA
– Within the EU27– Geographical Europe– Interoperability worldwide for collaboration
• Work with Virtual Research Communities– Groupings of aligned Virtual Organisations– Provide community specific:
• Support, training, consultancy, requirements etc.
Swiss Grid Day 15
![Page 16: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/16.jpg)
Swiss Grid Day 16
The ESFRI Projects• First roadmap 2006 and updated in 2008
with 44 projects• Preparatory phase funding for most with
second round soon• Big push in FP8?
ESFRI: European Strategy Forum on Research Infrastructures
![Page 17: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/17.jpg)
ESFRI Environmental Sciences
LIFEWATCH
EMSO
IAGOS-ERI
AURORA BOREALIS
EUFAR-COPAL
ICOS
EURO-ARGO
EPOS
EISCAT-3D
SIOS
Status 2009
17Swiss Grid Day
![Page 18: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/18.jpg)
EGI User SupportU
SE
RS VOs
VOs
Virtual Research
Community
UserForum
SteeringCommitteeU
SE
RS VOs
VOs
Virtual Research
Community
US
ER
S
NGI
NGIHelpdesk
EGI.eu
Training Events
Trainers
Apps.DB
EGIHelpdesk
VRCHelpdesk
OtherHelpdesk
ESFRI Project
VOsVirtual
ResearchCommunity
18
![Page 19: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/19.jpg)
Swiss Grid Day 19
Moving from EGEE to EGI• What is different?
– EGEE did ‘everything’– EGI focuses just on infrastructure operations
• What is the same?– Running a 24/7 production quality infrastructure– Providing a support framework for the users– Collaboration to drive European DCI forward
![Page 20: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/20.jpg)
The EGI-InSPIRE Project Integrated Sustainable Pan-European Infrastructure for Researchers in Europe
• Submitted to INFRA-2010 1.2.1• Targeting call objectives:
– 1.2.1.1: European Grid Initiative– 1.2.1.2: Service deployment for heavy users
• A 4 year project• Seeking a total €25M EC contribution
– Project cost €70M
![Page 21: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/21.jpg)
How much does it cost?
24
25
Annual Running Costs EGEE-III (M€)Staff (EC & National)
Hardware (National)
5127
14
8
% Staff Effort in EGEE-III
Operations User CommunityMiddleware Coordination
Annual CostsEGEE EC Contribution: €16M
(EGI Equivalent: ~€11M)Infrastructure Cost: ~ €80M
![Page 22: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/22.jpg)
EGI Project Ecosystem
EGI
SoftwareProviders
SoftwareProviders
SoftwareProviders
Comm
unityCom
munity
VRC
VRC
SSC
SSC
EC Projects
VRC
SpecialisedSupportCentres
SupportingProjects
Virtual ResearchCommunities
22
![Page 23: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/23.jpg)
Project Structure
23Project Management
External Relations: Policy, Dissemination and Standards
SA3SoftwareSupport
SA2Middleware
Unit
SA1Operations
JRA1Operational
Tools
SA4Services for
HUCs
NA3User
SupportSoftwareProvisioning
andSupport
Operations and Tools
User Support and
Services
![Page 24: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/24.jpg)
Project Structure
EGI-InSPIRE
Managerial Collaborations (e.g. EC, DEISA, OSG, ...)
Technical Collaborations(e.g. CHAIN, GISELR, SIENA, ...)
VRC & SSCProjects
(e.g. ROSCOE, SAFE, CUE, TAPAS, ...)
SoftwareProjects
(e.g. EMI, IGE, SGI...)
![Page 25: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/25.jpg)
User Support & Services• Support User Communities
– Researchers in International Collaborations– Scale up from the single VO to a community
• Provide a federated Helpdesk linking:– Discipline specific support (e.g. Bio Apps)– National infrastructure support (e.g. NGS)– Generic services (e.g. Training)
• Provide core services to support users– Manage VOs, Application DB, Training DB
Swiss Grid Day 25
![Page 26: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/26.jpg)
Possible Virtual Research Communities (FP7 funding)
• High Energy Physics (ROSCOE)• Life Sciences (ROSCOE)• Computational Chemistry & Material Science (ROSCOE)• Grid Observatory [Computer Science] (ROSCOE)• Complexity Science (ROSCOE)• Photon Science (ROSCOE)• Humanities (ROSCOE)• Astronomy & Astrophysics (SAFE)• Fusion (SAFE)• Earth Science (SAFE)• Molecular Biodiversity (SIMBIOME)• NMR in Structural Biology (WeNMR)• Hydro-Meteorology (DRIHME)
Swiss Grid Day 26
![Page 27: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/27.jpg)
Training Events
![Page 28: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/28.jpg)
Registry of Trainers
![Page 29: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/29.jpg)
Applications Database
29
http://appdb.eu-egee.org
Alt. link: http://grid.ct.infn.it/egee_applications/
![Page 30: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/30.jpg)
EGEE RESPECT EGI• Identify third-party software that works well with Grid
– gLite: http://technical.eu-egee.org/index.php?id=290 – Expand to include UNICORE, ARC, Globus, …
• Simplified Access– P-GRADE, Ganga, Migrating Desktop,
g-Eclipse, i2glogin, Virtual Control Room• Workload Management
– GridWay Metascheduler, DIANE• New Resources
– GRelC, Instrument Element• Infrastructure Services
– StoRM
Swiss Grid Day 30
![Page 31: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/31.jpg)
Other Activities• Dissemination
– With NGIs, VRCs, SSCs and other projects• Support for Heavy User Communities
– General & community specific services• Events
– Two Annual meetings: Users & Technology• Technology
– Liaison with software providers– Definition and verification of requirements
Swiss Grid Day 31
![Page 32: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/32.jpg)
Resources• EGI.eu does not own or control resources• Resources owned by individual organisations
– They manage access for their user communities• Middleware enables secure authorised access
for external collaborators– EGI provided Unified Middleware Distribution (UMD)– EGI defined software & interfaces– UMD compatible software must be deployed– Interoperation within your country and internationally
Swiss Grid Day 32
![Page 33: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/33.jpg)
Transition to EGI Model• Has already started within EGEE-III• Will continue in EGI-InSPIRE
– Integrated Sustainable Pan-European Infrastructure for Researchers in Europe
– 4 year project, seeking €25M EC contribution• User facing changes will be gradual
– Focused communication from EGI & EGEE• Main public points of contact have few changes
– Helpdesk & software
Swiss Grid Day 33
![Page 34: EGI: European Grid Infrastructure](https://reader035.fdocuments.net/reader035/viewer/2022062305/5681641d550346895dd5d8ce/html5/thumbnails/34.jpg)
Summary• Move away from a project based model
– Sustainable e-Infrastructure• User services will evolve to this new model
– Coordination of services provided through NGIs– Expect little change in the delivered services
• Start coordinating European e-Infrastructure– 1st May 2010
• Contact: [email protected]
Swiss Grid Day 34