Post on 06-Feb-2016
description
EGEE-III INFSO-RI-222667
Enabling Grids for E-sciencE
www.eu-egee.org
EGEE and gLite are registered trademarks
C. Loomis (CNRS/LAL)NA4 Activity Manager
EGEE-III First Review, 24-25 June, 2009
NA4: User CommunitySupport and Expansion
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 2
Activity OverviewCountry PM FTE
Austria 6 0.3
Belgium 12 0.5
CERN 162 6.8
Cyprus 12 0.5
Czech Republic
17 0.7
France 257 10.7
Germany 47 2.0
Greece 92 3.8
Hungary 126 5.3
Israel 12 0.5
Italy 209 8.7
Netherlands 29 1.2
Norway 30 1.3
Poland 42 1.8
Russia 66 2.8
Slovakia 24 1.0
Spain 211 8.8
Sweden 16 0.7
UK 41 1.7
TOTAL 1421 59.2
44 Partners
287 People
19 Countries
NA4: 19%
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 3
Tasks• TNA4.1: Support
– Virtual Organization Support (VOS)– Application Porting Support (APS)– Direct User Support (DUS)
• TNA4.2: Strategic Discipline Clusters– High Energy Physics (HEP)– Life Sciences (LS)– Earth Sciences (ES)– Grid Observatory (GO, CS)– Computational Chemistry (CC)– Astronomy & Astrophysics (AA)– Fusion (F)
• TNA4.3: Activity Coordination– Activity Management– Regional Coordination
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 4
User Community Organization
User
User
UserUser
User
User User
User
User
User
VO VO VO
Domain Domain
User C
omm
unityG
rid Auth.
Clusters
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 5
Community & Use
Domain
VOs
Users
AA 20 373
CC 4 347
CS 4 21
ES 7 142
F 2 68
HEP 36 8577
LS 9 379
MV 26 1658
OTH 28 1816
TOTAL 136 13381
Around 13000 Registered Users
Consistent doubling every 12-18 months.
EGEE-III = Y2EGEE-II = Y1
Accounting Portal: http://www3.egee.cesga.es/ CIC Portal: http://cic.gridops.org/
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 6
CPU Utilization by Domain
Domain
VOs(>0%
)
VOs(>10%
)VO Names
(>10%)AA 11 3 astro.vo.eu-egee.org, auger,
virgo
CC 4 2 compchem, trgrida
CS 2 1 imath.cesga.es
ES 4 1 esr
F 1 1 fusion
HEP 32 4 alice, atlas, cms, lhcb
LS 7 1 biomed
MV 12 5 aegis, balticgrid, see, seegrid, vo.gear.cern.ch
OTH 19 2 geant4, theophys
UNK 79 3 bg, litgrid, vo.nanocmos.ac.uk
112 Registered VOs 171 “Visible” VOs23 “Core” VOs4167 “Core” Users
Domain Y1 Y2
Y2/Y1
AA 2520 15726 6.2
CC 17400 29206 1.7
CS 1 52 42.6
ES 286 3301 11.5
F 3681 1428 0.4
HEP 202794
444399
2.2
LS 14343 19698 1.4
MV 12797 24164 1.9
OTH 3013 14528 4.8
UNK 4135 5524 1.3
TOTAL 260970
558025
2.12x increase overallHEP largest users / contributorsAA/ES/OTH show strong increase
CPU Use: 1K-SI2K-Month
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 7
Applications
http://appdb.eu-egee.org
Alt. link: http://grid.ct.infn.it/egee_applications/
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 8
Virtual Organization Support• VO Management Developments
– Improving the VO registration information– Integration of collaborative tools with VO information– Expansion of SAM testing framework for non-LHC VOs
• Documentation and Support Provision– Links to VO documentation:
https://twiki.cern.ch/twiki/bin/view/EGEE/VOSupport – Liaison between operations and VO managers
• VO Tools Identification– Identified problems with VOMS functionality– Worked in collaboration with JSPG on changes to policies
related to VO management
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 9
Application Porting Support• Consultancy and Porting
– 15 applications ported; ~10 applications being ported• Training
– Group collects, prepares, reorganizes training materials and offers those as customized training packages for users
– Direct participation in NA3 training events• Provision of Infrastructure Services
– Group leader is VO manager for the NA4– Partly responsible for Application Database
• Public Relations– Writing of success stories of ported applications to increase
visibility and to help others with similar applications– “EGEE App. Porting Support Group” won Best Demo prize at
EGEE’08
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 10
http://www.ldps.sztaki.hu/gasuc/
Application Porting Support
10 applications being ported
15 applications ported
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 11
Direct User Support• Ticket Handling
– DUS support unit part of GGUS since mid-September– Have taken 2-person, 2-week shifts to treat tickets– The number of tickets assigned to DUS has been small
• Documentation and Use Cases– Reviewed and accessed existing documentation– Writing new documentation to fill identified gaps– Working with clusters and other teams to improve their
documentation
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 12
• Ganga/DIANE, AMGA, Dashboard– Used by 1000s in HEP, strong adoption by other communities– Ganga/DIANE tutorials: NSS IEEE, Helsinki, BalticGrid– Dashboard tutorial: UF4/OGF25– CERN Training for Trainers
• Grid validation for LHC data taking: CCRC’08, STEP’09– 4 expts.,
3 grid infra., 100s of sites, O(PB) data
– Sustained 4GB/s CERN T1, O(100K)jobs/day
High Energy Physics
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 13
Technology Transfer Collab.• HEP and Fusion clusters (since EGEE’08)
– Porting of specific fusion applications using Ganga/DIANE– Results of the collaboration shown during UF4/OGF25
• Lattice QCD (in production since 2007)– Running autonomously on a daily basis using Ganga/DIANE– Sustained rate of 1000 concurrent jobs, 750 CPUs and more
than 20TB of data transferred• GEANT4 simulation toolkit (since 2005)
– Widely used by astroparticle physics, medical applications, radiation studies, as well as HEP
– Validation of the new releases performed regularly on EGEE grid, again using Ganga/DIANE
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 14
Life Sciences• Major calculations:
– WISDOM (http://wisdom.eu-egee.fr/)– System biology on cancer data– Genetic linkage analysis for disease loci– Identification of causes for coronary artery diseases
• Tooling Support– AMGA– Medical
Data Manager
– MOTEUR– Taverna2
Plug-In
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 15
Nature Genetics Article• Genome-wide haplotype analyses of complex human
diseases– Study the impact of DNA mutations on human coronary diseases– Very CPU intensive analysis to study the impact of correlated
(double, triple) DNA mutations • EGEE grid deployment
– 1926 Coronary Artery Disease patients; 2938 healthy controls– 378,000 Single Nucleon Polymorphisms = local DNA mutations– 8.1 million combinations tested in less than 45 days (instead of
>10 years on a single Pentium 4)• Results in Nature Genetics Mar. 2009 (D. Tregouet et al)
– Major role of mutations on chromosome 6 was confirmed.
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 16
Earth Sciences• User and application support• Dissemination
– Session at European Geosciences Union (EGU) in 2008– Special issue of journal with 12 peer-reviewed papers– 2 PhDs and 5 papers based on Geocluster results from EGEE
• Specialized tools– Data
distribution, file explorer, storage access systems, workflow tools, …
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 17
• Pesticide risk assessment and management in Europe– FP6 EU project– BRGM France + 14 partners in 9 countries
• Creation of a large database including 4 million scenarios (climate, soil, pesticides, …).
• Successful results with the first 2 million scenarios obtained with EGEE running around 4800 jobs/day.
• Exploitation of database and results by all partners.– Creation of one SME in France for agriculture consultancy.
Footprint
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 18
Grid Observatory• Created the Grid Observatory Portal
– Store and publish monitoring information for analysis• Reaching out to CS community:
– EGEE’08: Grid Community Meeting– UF4/OGF25: Joint session “From Grid Monitoring to Analysis”– Grid Meeting Autonomic Computing (GMAC’09) at ICAC
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 19
Grid Observatory Portal
http://grid-observatory.org/
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 20
Computational Chemistry• Analysis of grid licensing models.• Expanding membership
– Training of young researchers– Availability of necessary software packages
• Tooling:– Chempo, Charon, ECCE, Wien2K– Parallel
version of GAMESS
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 21
Computational Campaigns• Chemical reactions
– N + N2, O + O2 and F + HD– Thermal rate coefficients
• Nanotube modeling• P-Grade port of ABC program
Executor: executed as many times in parallel as many parameters are generated by “Generator” Collector: collects all output
files into a single TAR file
Generator: generates input files with different parameters (currently 4
input)
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 22
Astronomy & Astrophysics• Development of active community
– Large number of applications ported to the grid– Focused training and dissemination
• Tooling:– Management of parameter sweep applications– Scheduling and bookkeeping systems– Visualization
• Interaction with EuroVO
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 23
Planck Satellite
Launched 14 May will be in L2 orbit in early July.• INAF: Ported full LFI mission
simulation to EGEE• IFCA: Ported several codes to
LFI Data Processing Center:– Mexican Hat Wavelet filters– Multi-frequency Matrix filters– Matched Multi-filter code
CMB Power Spectrum(cmbfast)
CMB mapsDataanalysis
CMB Map(synfast)
Foregrounds andBeam Patterns
Instrumental Noise Scanning Strategy
TOD
Missionsimulation
Workstation Grid Gain
short 330 m 25 m 13
long 15342 m 955 m 16
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 24
Fusion• Application porting
– 9 applications have been ported to give relevant scientific results• Tooling:
– Data mgt. tool development with goal of multi-machine analysis– GIF Portal for launching Generic Algorithm-based applications– Use of Kepler workflow engine (bridge to EUFORIA project)
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 25
Fusion Developments• ISDEP MC code follows ion trajectories inside plasma
– Self-consistent plasma profiles: intro. of non-linear effects– Divertor Studies: Map of 3D fluxes on wall of device– Tokamak geometry– Ion heating
• ASTRA-MaRaTra– First complex fusion workflow between
applications running on different platforms– ASTRA: SGI Application– MaRaTra: Grid Infrastructures
ASTRA MaRaTra
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 26
Activity Coordination• Activity Management
– All milestones and deliverables have been achieved– Maintain the RESPECT program– Encouraged community interaction via the User Forum– Encourage participation in meetings through “travel money”:
Financed 6 people to attend EGEE’08 and UF4 Sponsor of the GMAC’09 workshop
– Contributed to EGEE EGI migration via SSC Workshops– Collaboration with MathWorks for MATLAB on the grid
• Regional Coordination– Design, implementation, and filling of Application Database– First line support, document review, etc.– Liaison activities increasingly important as EGI approaches
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 27
RESPECT• Identify third-party software that works well with gLite.
– http://technical.eu-egee.org/index.php?id=290
• Simplified Access– P-GRADE, Ganga, Migrating Desktop,
g-Eclipse, i2glogin, Virtual Control Room• Workload Management
– GridWay Metascheduler, DIANE• New Resources
– GRelC, Instrument Element• Infrastructure Services
– StoRM
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 28
User Forum
UF1(CERN)
UF2-OGF20(Manchester)
UF3(Clermont-Ferrand)
UF4-OGF25
(Catania)
http://technical.eu-egee.org/index.php?id=148
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 29
Issues• Technical Issues
– Fragility of applications with upgrades– Ease of use (availability of Java APIs)– SAM Nagios transition for VO-specific tests– Firewall configurations and data transfers– MPI support
• Administrative Issues– Late recruiting– Unresponsive partners
• Systemic Problems– Visibility of the NA4 support services.– Underutilization of those support services.
Followed up
in project
Largely
resolved
Emphasis in
Year 2
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 30
Deviations from Work Plan
Task
ConsumedEffort (PM8)
PlannedEffort (PM8)
Deviation (%)
VO Support 13.7 15.1 -9.7%
App. Port. Support 39.4 60.3 -34.6%
Dir. User Support 36.6 49.6 -26.2%
High Energy Phys. 54.3 48.5 +11.8%
Life Science 40.7 32.9 +23.9%
Earth Science 20.1 20.4 -1.1%
Grid Observatory 21.8 11.0 +99.2%
Comp. Chemistry 26.8 20.4 31.7%
Astron. & Astro. 17.6 17.2 2.3%
Fusion 20.7 17.2 20.1%
Activity Mgt. 24.0 20.9 +14.9%
Reg. Coord. 49.4 57.4 -14.0%
Cross-Activity Tasks
102.9 102.9 0.0%
TOTAL 468.0 473.7 -1.2%
Higher spending than planned.Expect rate to continue.Becomes additional unfunded contributions.
Significant fraction of expended effort.
Lower spending than planned.Slow start up.Low visibility and under-utilization of services.
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 31
Plans for Year 2• Support Activities:
– Improve visibility and use of all support services– Publicize the seed resources for new users and new VOs– Work on transition to EGI support structures
• Strategic Discipline Clusters– Continue current scientific activities– Work on transition to the EGI SSC models
• Management– Continue coordination activities– Make more use of community building funds– Enhance cooperation with NA2 to increase dissemination
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 32
Specialized Support Centers• No major structural changes:
– NA4 Steering Committee User Forum Steering Committee– Strategic Discipline Clusters Specialized Support Centers
• Each SSC:– Must be much more autonomous than now– Must find and attract financial and political support– Must be the center of gravity for grid use within their
communities
• It will be a hard challenge to have fully functional SSCs in time for the start of EGI.
Enabling Grids for E-sciencE
EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 33
Summary• Three principal tasks of NA4 have worked well.• User Community
– 13000 users, 220 applications, 112 registered VOs– Majority of use from 23 core VOs– Overall CPU use increased by factor of 2
• Scientific impact– Shown results could only be achieved with the grid.– User Forum 4 program and Book of Abstracts– Detailed achievements provided in DNA4.4.1
• Future Work– Improve visibility and utilization of support services– Guide formation of SSCs for the EGI transition