EGEE is a project co-funded by the European Commission under contract INFSO-RI-508833
EGEE is a project co-funded by the European Commission under contract INFSO-RI-508833 EU EGEE...
-
Upload
kathryn-preston -
Category
Documents
-
view
219 -
download
0
Transcript of EGEE is a project co-funded by the European Commission under contract INFSO-RI-508833 EU EGEE...
EGEE is a project co-funded by the European Commission under contract INFSO-RI-508833
EU EGEE project – status and plans Bob Jones
EGEE Technical [email protected]
UK eScience All Hands MeetingNottingham, September 2004
AHM2004, Nottingham, September 2004 - 2
Contents
• EGEE - what is it and why is it needed?
• Grid operations – providing a stable service
• Grid middleware – current and future
• How to join – for new applications
• Summary
The material for this talk has been contributed by many colleagues in the EGEE & LCG projects
Despite its name EGEE is an International project involving in particular Israel, Russia and the US
AHM2004, Nottingham, September 2004 - 3
The next generation of grids:EGEE Enabling Grids for E-science in Europe
Build a large-scale production grid service to:
• Underpin European science and technology
• Link with and build on national, regional and international initiatives
• Foster international cooperation both in the creation and the use of the e-infrastructure Network
infrastructure(GÉANT )
Op
era
tio
ns
, S
up
po
rt a
nd
tr
ain
ing
Collaboration
Pan-European Grid
AHM2004, Nottingham, September 2004 - 4
EGEE Activities
• 48 % service activities (Grid Operations, Support and Management, Network Resource Provision)
• 24 % middleware re-engineering (Quality Assurance, Security, Network Services Development)
• 28 % networking (Management, Dissemination and Outreach, User Training and Education, Application Identification and Support, Policy and International Cooperation)
32 Million Euros EU funding over 2 years starting 1st April 2004
Emphasis in EGEE is on operating a productiongrid and supporting the end-users
AHM2004, Nottingham, September 2004 - 5
In 2 years EGEE will:
• Establish production quality sustained Grid services • 3000 users from at least 5 disciplines• over 8,000 CPU's, 50 sites• over 5 Petabytes (1015) storage
• Demonstrate a viable general process to bring other scientific communities on board
• Propose a second phase in mid 2005 to take over EGEE in early 2006
Pilot New
AHM2004, Nottingham, September 2004 - 6
EGEE and LCG
EGEE builds on the work of LCG to establish a grid operations service
• LCG (LHC Computing Grid) - Building and operating the LHC Grid
• A collaboration between:• The physicists and computing
specialists from the LHC experiment
• The projects in Europe and the US that have been developing Grid middleware
• The regional and national computing centres that provide resources for LHC
• The research networks
AHM2004, Nottingham, September 2004 - 7
EGEE pilot application: BioMedical
• BioMedical• Bioinformatics (gene/proteome databases
distributions)
• Medical applications (screening, epidemiology, image databases distribution, etc.)
• Interactive application (human supervision or simulation)
• Security/privacy constraints Heterogeneous data formats - Frequent
data updates - Complex data sets - Long term archiving
• BioMed applications deployed and going live in September
• GATE - Geant4 Application for Tomographic Emission• GPS@ - genomic web portal • CDSS - Clinical Decision Support System
http://egee-na4.ct.infn.it/biomed/applications.html
AHM2004, Nottingham, September 2004 - 8
production grid service
Launched Sept’03 with 12 sites, now more than 70 sites and continues to grow
Live updateshttp://goc.grid-support.ac.uk/lcg2
AHM2004, Nottingham, September 2004 - 9
Current production mware: LCG-2
• Regular updates (latest is LCG-2.2.0 August 2004)• short term developments driven by operational priorities
Computing cluster Network resources Data storage
Operating system Local schedulerFile system
User access SecurityData transferInformation schema
Resource Broker Data managementApp monitoring system
User interfaces Applications
Hardware
System software
“Basic” services
“Collective” services
Application level services
HPSS, CASTOR…HPSS, CASTOR…
RedHat LinuxRedHat Linux NFS, …NFS, … PBS, Condor, LSF,…PBS, Condor, LSF,…
VDT (Condor, Globus, GLUE)VDT (Condor, Globus, GLUE)
EU DataGridEU DataGrid
Information system
AHM2004, Nottingham, September 2004 - 10
Running the Production Service
Grid deployment has entered a new phase• Basic middleware is working
• responsible now for a small fraction of the problems
• Outstanding performance/functionality issues• RLS, RB / little modularity & lack of consistent interfaces …• some solutions are being developed but many cannot be addressed in current
software/architecture - set priorities for new middleware (gLite)
• Many operational issues• mis-configuration, out of date mware, single points of failure, failover, mgmt interfaces …• resources unsuitable for applications needs (e.g. insufficient disk space)• slow response by sites to problems (holiday periods, security concerns)• new middleware will not help for many of these issues - grid partners must think Service
The grid still does not appear as a single coherent facilityapplications must adapt to the current service to gain maximum profit but result has been very effective for LHCb - ~3000 concurrent jobs (August)
AHM2004, Nottingham, September 2004 - 11
• Intended to replace LCG-2
• Starts with existing components from AliEN, EDG, VDT etc.
• Aims to address LCG-2 shortcoming and advanced needs from applications
• Prototyping short development cycles for fast user feedback
• Initial web-services based prototypes being tested with representatives from the application groups
Future EGEE Middleware - gLite
Globus 2 based Web services based
gLite-2gLite-1LCG-2LCG-1
Application requirements http://egee-na4.ct.infn.it/requirements/
AHM2004, Nottingham, September 2004 - 12
Architecture Guiding Principles
• Lightweight (existing) services • Easily and quickly deployable• Use existing services where possible as
basis for re-engineering
• Interoperability• Allow for multiple implementations
• Resilience and Fault Tolerance
• Co-existence with deployed infrastructure• Reduce requirements on site components• Co-existence (and convergence) with LCG-2 and Grid3 are essential for the EGEE
Grid service
• Service oriented approach• Follow WSRF standardization• No mature WSRF implementations exist to date so start with plain WS (WS-I)• Provide framework to others so higher-level services can be developed quickly
Architecture: https://edms.cern.ch/document/476451
AHM2004, Nottingham, September 2004 - 13
gLite Approach
• Exploit experience and components from existing projects
• AliEn, VDT, EDG, LCG, and others
• Design team works out architecture and design
• Feedback and guidance from EGEE PTF & applications; Operations, LCG GAG & ARDA
• Components are initially deployed on a prototype infrastructure• Small scale (CERN & Univ. Wisconsin)• Get user feedback on service semantics and interfaces
• After internal integration and testing, components are delivered to grid operations group and deployed on the pre-production service
EDGVDT . . .
LCG . . .AliEn
Draft Design - https://edms.cern.ch/document/487871/PTF – Project Technical Forum (http://egee-ptf.web.cern.ch/egee-ptf/default.htm)GAG – Grid Application Group (http://project-lcg-gag.web.cern.ch/project-lcg-gag/)ARDA - A Realisation of Distributed Analysis for LHC (http://lcg.web.cern.ch/LCG/peb/arda/Default.htm)
AHM2004, Nottingham, September 2004 - 14
Deployment considerations
• Interoperability and co-existence• Exploit different service implementations
E.g. Castor and dCache SRM implementations
• Flexible service deployment Multiple services running on the same physical machine (if possible)
• Platform support• Goal is to have portable middleware
• Building & Integration on RHEL 3 and windows
• Initial testing (at least 3 sites) using different Linux flavors (including free distributions)
• Service autonomy• User may talk to services directly or through other services (like access
service)
AHM2004, Nottingham, September 2004 - 15
gLite security
Aims at being• Modular – add new modules later• Agnostic – modules will evolve• Standard – start with transport-level security but intend to move to WS-Security when it matures • Interoperable - at least for AuthN & AuthZ
Applied to Web-services hosted in containers and applications (Apache Axis & Tomcat) as additional modules
Draft security architecture: https://edms.cern.ch/document/487004/
AHM2004, Nottingham, September 2004 - 16
gLite Services
Legend:Service foreseen in release 1See release plan:https://edms.cern.ch/document/468699
VOMS
PKI / GSI / Myproxy
EDG RGMA
ALiEN FileCatalog /EDG/Globus RLS
EDG HLR
File I/OSRMGlobus gridFTP
Globus GateKeeper / CondorG-C EDG RB, L&B
ALiEN TaskQueue /Condor DAG
implemtaton
ALiEN shell
AHM2004, Nottingham, September 2004 - 17
• LCG-2• Current base for production services• Evolves with certified new or improved services
from the preproduction
• Pre-production Service• Early application access for new developments• Certification of selected components from gLite• Starts with LCG-2
• Migrate new mware in 2005• Organising smooth/gradual transition from LCG-2
to gLite for production operations
EGEE Middleware Migration
LCG-2 (=EGEE-0)
prototyping
prototyping
product
20042004
20052005
LCG-3 (=EGEE-x?)
product
AHM2004, Nottingham, September 2004 - 18
Intellectual Property
• The existing EGEE grid middleware (LCG-2) is distributed under an Open Source License developed by EU DataGrid• Derived from modified BSD - no restriction on
usage (academic or commercial) beyond acknowledgement
• Same approach for new middleware (gLite)
• Application software maintains its own licensing scheme• Sites must obtain appropriate licenses before
installation
AHM2004, Nottingham, September 2004 - 19
Who else can benefit from EGEE?
• EGEE Generic Applications Advisory Panel:• 4 applications presented
• 3 applications (comp. chemistry, earth science, astro-particle) recommended for deployment with allocation of NA4 resources
• EU GRACE project already tested
• EU projects: MammoGrid, Diligent, SEE-GRID …
• Expression of interest: Planck/Gaia (astroparticle), SimDat (drug discovery)
http://agenda.cern.ch/age?a042351 Next meeting at EGEE conference (November)
AHM2004, Nottingham, September 2004 - 20
Bringing new applications to the grid
1. Outreach events inform people about the grid / EGEE
2. Application experts discuss specific characteristics with the users
3. Migrate application to EGEE infrastructure with the help of EGEE experts
4. Initial deployment for testing purposes
5. Production usage - user community contributes computing resources for heavy production demands - “Canadian dinner party”
AHM2004, Nottingham, September 2004 - 21
Private vs Federated ResourcesFor applications that must operate in a closed environment, EGEE
middleware can be downloaded and installed on closed infrastructuresApproach being used by MammoGrid
EGEE sites are administered/owned by different organisationsSites have ultimate control over how their resources are usedLimiting the demands of your application will make it acceptable to more sites and hence make more resources available to you
AHM2004, Nottingham, September 2004 - 22
User training and induction
• Training material and courses from introductory to advanced level
• Train a wide variety of users both internal to the EGEE consortium and external groups from across Europe
• ~20 courses/presentations already held and many more planned (see roadmap)
• Experience with GENIUS portal and GILDA testbed
• Courses inline with the needs of the projects and applications
Training: http://www.egee.nesc.ac.uk/Roadmap: http://www.egee.nesc.ac.uk/schedreg/index.html
AHM2004, Nottingham, September 2004 - 23
Dissemination
• 1st project conference• Over 300 delegates came to the 4 day
event during April in Cork Ireland• Kick-off meeting bringing together
representatives from the 70 partner organisations
• 2nd conference scheduled• 22-26 November in The Hague• http://public.eu-egee.org/conferences/2nd/
• Websites, Brochures and press releases• For project and general public www.eu-
egee.org• Information packs for the general public,
press and industry
AHM2004, Nottingham, September 2004 - 24
EGEE Industry Forum
• EGEE Industry Forum• raise awareness of the project in industry to
encourage industrial participation in the project
• foster direct contact of the project partners with industry
• ensure that the project can benefit from practical experience of industrial applications
• For more info: http://public.eu-egee.org/industry/
AHM2004, Nottingham, September 2004 - 25
EGEE Plans for the coming year
• SeptemberFirst non-HEP applications running on LCG-2 production serviceSecurity architecture/ Grid services design for new mwareDeployment of 2nd gLite prototype
• November2nd EGEE conference (Den Hague) in common with DEISA, SEE-GRID,
DILIGENT etc.
• DecemberApplication migration reports
• February 20051st EU review
• March 2005Large-scale deployment of gLite softwareAnnual report
42 d
eliv
erab
les
in 1
st y
ear
AHM2004, Nottingham, September 2004 - 26
Summary
• EGEE is the first attempt to build a worldwide Grid infrastructure for data intensive applications from many scientific domains
• A large-scale production grid service is already deployed and being used for HEP and BioMed applications with new applications being ported
• Resources & user groups will rapidly expand during the project
• A process is in place for migrating new applications to the EGEE infrastructure
• A training programme has started with events already held
• Prototype “next generation” middleware is being tested (gLite)
• Plans for a follow-on project are being discussed
AHM2004, Nottingham, September 2004 - 27
EGEE www.eu-egee.orgLCG lcg.web.cern.ch/LCG/CERN www.cern.chThe Grid Cafe www.gridcafe.org
Further Information