NA4: User Community Support and Expansion

EGEE-III INFSO-RI-222667

Enabling Grids for E-sciencE

www.eu-egee.org

EGEE and gLite are registered trademarks

C. Loomis (CNRS/LAL)NA4 Activity Manager

EGEE-III First Review, 24-25 June, 2009

NA4: User CommunitySupport and Expansion


EGEE-III INFSO-RI-222667 NA4 - C. Loomis - EGEE-III First Review 24-25 June 2009 2

Activity OverviewCountry PM FTE

Austria 6 0.3

Belgium 12 0.5

CERN 162 6.8

Cyprus 12 0.5

Czech Republic

17 0.7

France 257 10.7

Germany 47 2.0

Greece 92 3.8

Hungary 126 5.3

Israel 12 0.5

Italy 209 8.7

Netherlands 29 1.2

Norway 30 1.3

Poland 42 1.8

Russia 66 2.8

Slovakia 24 1.0

Spain 211 8.8

Sweden 16 0.7

UK 41 1.7

TOTAL 1421 59.2

44 Partners

287 People

19 Countries

NA4: 19%



Tasks• TNA4.1: Support

– Virtual Organization Support (VOS)– Application Porting Support (APS)– Direct User Support (DUS)

• TNA4.2: Strategic Discipline Clusters– High Energy Physics (HEP)– Life Sciences (LS)– Earth Sciences (ES)– Grid Observatory (GO, CS)– Computational Chemistry (CC)– Astronomy & Astrophysics (AA)– Fusion (F)

• TNA4.3: Activity Coordination– Activity Management– Regional Coordination



User Community Organization

User

User

UserUser

User

User User

User

User

User

VO VO VO

Domain Domain

User C

omm

unityG

rid Auth.

Clusters



Community & Use

Domain

VOs

Users

AA 20 373

CC 4 347

CS 4 21

ES 7 142

F 2 68

HEP 36 8577

LS 9 379

MV 26 1658

OTH 28 1816

TOTAL 136 13381

Around 13000 Registered Users

Consistent doubling every 12-18 months.

EGEE-III = Y2EGEE-II = Y1

Accounting Portal: http://www3.egee.cesga.es/ CIC Portal: http://cic.gridops.org/

http://www3.egee.cesga.es/

http://cic.gridops.org/



CPU Utilization by Domain

Domain

VOs(>0%

)

VOs(>10%

)VO Names

(>10%)AA 11 3 astro.vo.eu-egee.org, auger,

virgo

CC 4 2 compchem, trgrida

CS 2 1 imath.cesga.es

ES 4 1 esr

F 1 1 fusion

HEP 32 4 alice, atlas, cms, lhcb

LS 7 1 biomed

MV 12 5 aegis, balticgrid, see, seegrid, vo.gear.cern.ch

OTH 19 2 geant4, theophys

UNK 79 3 bg, litgrid, vo.nanocmos.ac.uk

112 Registered VOs 171 “Visible” VOs23 “Core” VOs4167 “Core” Users

Domain Y1 Y2

Y2/Y1

AA 2520 15726 6.2

CC 17400 29206 1.7

CS 1 52 42.6

ES 286 3301 11.5

F 3681 1428 0.4

HEP 202794

444399

2.2

LS 14343 19698 1.4

MV 12797 24164 1.9

OTH 3013 14528 4.8

UNK 4135 5524 1.3

TOTAL 260970

558025

2.12x increase overallHEP largest users / contributorsAA/ES/OTH show strong increase

CPU Use: 1K-SI2K-Month



Applications

http://appdb.eu-egee.org

Alt. link: http://grid.ct.infn.it/egee_applications/

http://appdb.eu-egee.org/



Virtual Organization Support• VO Management Developments

– Improving the VO registration information– Integration of collaborative tools with VO information– Expansion of SAM testing framework for non-LHC VOs

• Documentation and Support Provision– Links to VO documentation:

https://twiki.cern.ch/twiki/bin/view/EGEE/VOSupport – Liaison between operations and VO managers

• VO Tools Identification– Identified problems with VOMS functionality– Worked in collaboration with JSPG on changes to policies

related to VO management

https://twiki.cern.ch/twiki/bin/view/EGEE/VOSupport



Application Porting Support• Consultancy and Porting

– 15 applications ported; ~10 applications being ported• Training

– Group collects, prepares, reorganizes training materials and offers those as customized training packages for users

– Direct participation in NA3 training events• Provision of Infrastructure Services

– Group leader is VO manager for the NA4– Partly responsible for Application Database

• Public Relations– Writing of success stories of ported applications to increase

visibility and to help others with similar applications– “EGEE App. Porting Support Group” won Best Demo prize at

EGEE’08



http://www.ldps.sztaki.hu/gasuc/

Application Porting Support

10 applications being ported

15 applications ported

http://www.ldps.sztaki.hu/gasuc/



Direct User Support• Ticket Handling

– DUS support unit part of GGUS since mid-September– Have taken 2-person, 2-week shifts to treat tickets– The number of tickets assigned to DUS has been small

• Documentation and Use Cases– Reviewed and accessed existing documentation– Writing new documentation to fill identified gaps– Working with clusters and other teams to improve their

documentation



• Ganga/DIANE, AMGA, Dashboard– Used by 1000s in HEP, strong adoption by other communities– Ganga/DIANE tutorials: NSS IEEE, Helsinki, BalticGrid– Dashboard tutorial: UF4/OGF25– CERN Training for Trainers

• Grid validation for LHC data taking: CCRC’08, STEP’09– 4 expts.,

3 grid infra., 100s of sites, O(PB) data

– Sustained 4GB/s CERN T1, O(100K)jobs/day

High Energy Physics



Technology Transfer Collab.• HEP and Fusion clusters (since EGEE’08)

– Porting of specific fusion applications using Ganga/DIANE– Results of the collaboration shown during UF4/OGF25

• Lattice QCD (in production since 2007)– Running autonomously on a daily basis using Ganga/DIANE– Sustained rate of 1000 concurrent jobs, 750 CPUs and more

than 20TB of data transferred• GEANT4 simulation toolkit (since 2005)

– Widely used by astroparticle physics, medical applications, radiation studies, as well as HEP

– Validation of the new releases performed regularly on EGEE grid, again using Ganga/DIANE



Life Sciences• Major calculations:

– WISDOM (http://wisdom.eu-egee.fr/)– System biology on cancer data– Genetic linkage analysis for disease loci– Identification of causes for coronary artery diseases

• Tooling Support– AMGA– Medical

Data Manager

– MOTEUR– Taverna2

Plug-In

http://wisdom.eu-egee.fr/



Nature Genetics Article• Genome-wide haplotype analyses of complex human

diseases– Study the impact of DNA mutations on human coronary diseases– Very CPU intensive analysis to study the impact of correlated

(double, triple) DNA mutations • EGEE grid deployment

– 1926 Coronary Artery Disease patients; 2938 healthy controls– 378,000 Single Nucleon Polymorphisms = local DNA mutations– 8.1 million combinations tested in less than 45 days (instead of

>10 years on a single Pentium 4)• Results in Nature Genetics Mar. 2009 (D. Tregouet et al)

– Major role of mutations on chromosome 6 was confirmed.



Earth Sciences• User and application support• Dissemination

– Session at European Geosciences Union (EGU) in 2008– Special issue of journal with 12 peer-reviewed papers– 2 PhDs and 5 papers based on Geocluster results from EGEE

• Specialized tools– Data

distribution, file explorer, storage access systems, workflow tools, …



• Pesticide risk assessment and management in Europe– FP6 EU project– BRGM France + 14 partners in 9 countries

• Creation of a large database including 4 million scenarios (climate, soil, pesticides, …).

• Successful results with the first 2 million scenarios obtained with EGEE running around 4800 jobs/day.

• Exploitation of database and results by all partners.– Creation of one SME in France for agriculture consultancy.

Footprint



Grid Observatory• Created the Grid Observatory Portal

– Store and publish monitoring information for analysis• Reaching out to CS community:

– EGEE’08: Grid Community Meeting– UF4/OGF25: Joint session “From Grid Monitoring to Analysis”– Grid Meeting Autonomic Computing (GMAC’09) at ICAC



Grid Observatory Portal

http://grid-observatory.org/



Computational Chemistry• Analysis of grid licensing models.• Expanding membership

– Training of young researchers– Availability of necessary software packages

• Tooling:– Chempo, Charon, ECCE, Wien2K– Parallel

version of GAMESS



Computational Campaigns• Chemical reactions

– N + N2, O + O2 and F + HD– Thermal rate coefficients

• Nanotube modeling• P-Grade port of ABC program

Executor: executed as many times in parallel as many parameters are generated by “Generator” Collector: collects all output

files into a single TAR file

Generator: generates input files with different parameters (currently 4

input)



Astronomy & Astrophysics• Development of active community

– Large number of applications ported to the grid– Focused training and dissemination

• Tooling:– Management of parameter sweep applications– Scheduling and bookkeeping systems– Visualization

• Interaction with EuroVO



Planck Satellite

Launched 14 May will be in L2 orbit in early July.• INAF: Ported full LFI mission

simulation to EGEE• IFCA: Ported several codes to

LFI Data Processing Center:– Mexican Hat Wavelet filters– Multi-frequency Matrix filters– Matched Multi-filter code

CMB Power Spectrum(cmbfast)

CMB mapsDataanalysis

CMB Map(synfast)

Foregrounds andBeam Patterns

Instrumental Noise Scanning Strategy

TOD

Missionsimulation

Workstation Grid Gain

short 330 m 25 m 13

long 15342 m 955 m 16



Fusion• Application porting

– 9 applications have been ported to give relevant scientific results• Tooling:

– Data mgt. tool development with goal of multi-machine analysis– GIF Portal for launching Generic Algorithm-based applications– Use of Kepler workflow engine (bridge to EUFORIA project)



Fusion Developments• ISDEP MC code follows ion trajectories inside plasma

– Self-consistent plasma profiles: intro. of non-linear effects– Divertor Studies: Map of 3D fluxes on wall of device– Tokamak geometry– Ion heating

• ASTRA-MaRaTra– First complex fusion workflow between

applications running on different platforms– ASTRA: SGI Application– MaRaTra: Grid Infrastructures

ASTRA MaRaTra



Activity Coordination• Activity Management

– All milestones and deliverables have been achieved– Maintain the RESPECT program– Encouraged community interaction via the User Forum– Encourage participation in meetings through “travel money”:

Financed 6 people to attend EGEE’08 and UF4 Sponsor of the GMAC’09 workshop

– Contributed to EGEE EGI migration via SSC Workshops– Collaboration with MathWorks for MATLAB on the grid

• Regional Coordination– Design, implementation, and filling of Application Database– First line support, document review, etc.– Liaison activities increasingly important as EGI approaches



RESPECT• Identify third-party software that works well with gLite.

– http://technical.eu-egee.org/index.php?id=290

• Simplified Access– P-GRADE, Ganga, Migrating Desktop,

g-Eclipse, i2glogin, Virtual Control Room• Workload Management

– GridWay Metascheduler, DIANE• New Resources

– GRelC, Instrument Element• Infrastructure Services

– StoRM

http://technical.eu-egee.org/index.php?id=290



User Forum

UF1(CERN)

UF2-OGF20(Manchester)

UF3(Clermont-Ferrand)

UF4-OGF25

(Catania)





Issues• Technical Issues

– Fragility of applications with upgrades– Ease of use (availability of Java APIs)– SAM Nagios transition for VO-specific tests– Firewall configurations and data transfers– MPI support

• Administrative Issues– Late recruiting– Unresponsive partners

• Systemic Problems– Visibility of the NA4 support services.– Underutilization of those support services.

Followed up

in project

Largely

resolved

Emphasis in

Year 2



Deviations from Work Plan

Task

ConsumedEffort (PM8)

PlannedEffort (PM8)

Deviation (%)

VO Support 13.7 15.1 -9.7%

App. Port. Support 39.4 60.3 -34.6%

Dir. User Support 36.6 49.6 -26.2%

High Energy Phys. 54.3 48.5 +11.8%

Life Science 40.7 32.9 +23.9%

Earth Science 20.1 20.4 -1.1%

Grid Observatory 21.8 11.0 +99.2%

Comp. Chemistry 26.8 20.4 31.7%

Astron. & Astro. 17.6 17.2 2.3%

Fusion 20.7 17.2 20.1%

Activity Mgt. 24.0 20.9 +14.9%

Reg. Coord. 49.4 57.4 -14.0%

Cross-Activity Tasks

102.9 102.9 0.0%

TOTAL 468.0 473.7 -1.2%

Higher spending than planned.Expect rate to continue.Becomes additional unfunded contributions.

Significant fraction of expended effort.

Lower spending than planned.Slow start up.Low visibility and under-utilization of services.



Plans for Year 2• Support Activities:

– Improve visibility and use of all support services– Publicize the seed resources for new users and new VOs– Work on transition to EGI support structures

• Strategic Discipline Clusters– Continue current scientific activities– Work on transition to the EGI SSC models

• Management– Continue coordination activities– Make more use of community building funds– Enhance cooperation with NA2 to increase dissemination



Specialized Support Centers• No major structural changes:

– NA4 Steering Committee User Forum Steering Committee– Strategic Discipline Clusters Specialized Support Centers

• Each SSC:– Must be much more autonomous than now– Must find and attract financial and political support– Must be the center of gravity for grid use within their

communities

• It will be a hard challenge to have fully functional SSCs in time for the start of EGI.



Summary• Three principal tasks of NA4 have worked well.• User Community

– 13000 users, 220 applications, 112 registered VOs– Majority of use from 23 core VOs– Overall CPU use increased by factor of 2

• Scientific impact– Shown results could only be achieved with the grid.– User Forum 4 program and Book of Abstracts– Detailed achievements provided in DNA4.4.1

• Future Work– Improve visibility and utilization of support services– Guide formation of SSCs for the EGI transition

NA4: User Community Support and Expansion

Documents

Transcript of NA4: User Community Support and Expansion