Www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE EGI-InSPIRE RI-261323 EGI-InSPIRE Review: Work...

www.egi.euEGI-InSPIRE RI-261323

EGI-InSPIRE


EGI-InSPIRE Review:Work Package 6 (SA3)

Jamie ShiersCERN

SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011


Activity Overview

• This slide will be provided by the PO• It will summarise the activity in tables by:

– The # partners, # people, # countries– The # PM and #FTE per country

• It will summarise the activity in graphics by:– The % effort of the activity within the project– The geographical spread across Europe



Partner Breakdown

Taken from the Description of Work

Participant number Participant Name Person-months per participant

10 KIT-6 27

12 CSIC 45

13 CSC 18

14 CNRS 83

19 TCD 21

21 INFN 126

28 CYFRONET 6

31 ARNES 3

32 UI SAV 18

35 CERN 341

37 EMBL 44

Total 732


Task Breakdown


Task ActivitiesTSA3.1 Activity ManagementTSA3.2 Shared services and toolsTSA3.3 Services for High Energy Physics (HEP)TSA3.4 Services for Life Sciences (LS)TSA3.5 Services for Astronomy and Astrophysics (A&A)TSA3.6 Services for Earth Sciences (ES)

Other HUCS: • Computational Chemistry and Materials Sciences and Technologies (CCMST),• Fusion (F)


Work Package Objectives

• Transition to sustainable support:

– Identify tools of benefit to multiple communities

– Migrate these as part of the core infrastructure

– Establish support models for those relevant to individual communities



Communities & Activities

Maria Girone, IEEE MSST

High Energy Physics The four LHC experiments use grid computing for data distribution,

processing and analysis. Strong focus on common tools and solutions. Areas supported include: Data Management, Data Analysis and Monitoring. Main VOs: ALICE, ATLAS, CMS, LHCb

Life Sciences

Covers a variety of projects including the European Extremely Large Telescope (E-ELT), the Square Kilometre Array (SKA) and Cerenkov Telescope Array (CTA). Activities focus on visualisation tools and database/catalog access from the grid.Covers seismology, atmospheric modelling, meteorological forecasting, flood forecasting and climate change. Provides access from the grid to resources within the Ground European Network for Earth Science Interoperations - Digital Repositories (GENESI-DR). Also assists scientists working on climate change via the Climate-G testbed.

Astronomy & Astrophysics

Earth Sciences

Life Sciences

Focuses on medical, biomedical and bioinformatics sectors to connect worldwide laboratories, share resources and ease access to data in a secure and confidential way. Supports 4 VOs (biomed, lsgri, vlemed and pneumogrid) across 6 NGIs via the Life Science Grid Community.


Communities & Activities


High Energy Physics The four LHC experiments use grid computing for data distribution,

processing and analysis. Strong focus on common tools and solutions. Areas supported include: Data Management, Data Analysis and Monitoring. Main VOs: ALICE, ATLAS, CMS, LHCb

Life Sciences

Covers a variety of projects including the European Extremely Large Telescope (E-ELT), the Square Kilometre Array (SKA) and Cerenkov Telescope Array (CTA). Activities focus on visualisation tools and database/catalog access from the grid.Covers seismology, atmospheric modelling, meteorological forecasting, flood forecasting and climate change. Provides access from the grid to resources within the Ground European Network for Earth Science Interoperations - Digital Repositories (GENESI-DR). Also assists scientists working on climate change via the Climate-G testbed.


Earth Sciences

Life Sciences

Focuses on medical, biomedical and bioinformatics sectors to connect worldwide laboratories, share resources and ease access to data in a secure and confidential way. Supports 4 VOs (biomed, lsgri, vlemed and pneumogrid) across 6 NGIs via the Life Science Grid Community.

These and other communities / projects supported by shared tools & services


Services for HUCs

1. Successfully supported major production computing at an unprecedented scale – both quantitatively and qualitatively

2. Successfully delivered common solutions in a variety of areas – with other activities in progress

3. Actively participated in EGI Technical & User Forum via presentations, tutorials and demos

4. Broadened the use of grid technology and HUC services to related projects within the HUC domain (such as unfunded – by EGI-InSPIRE – LS / ES projects)

5. Completed first round of Milestones & Deliverables together with associated technical work

6. Identified – across all HUC communities – areas of common technology investigation for the future

7. Developed a S.W.O.T. analysis of each main discipline and made significant steps on the road to sustainability


www.egi.euEGI-InSPIRE RI-261323SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011

10

Service Quantity & Quality

• The first year of EGI-InSPIRE saw the use of grid computing at an unprecedented scale!– More than 100 CPU-millennia delivered;– More than 50 PB of data stored;– Data transfer rates of 200TB/day

• At the same time, its power in turning scientific data into publications at record speed was publically acknowledged (Economist, July 2010)

• And service delivery was typically smooth with a small number of problems requiring in-depth investigation

• Quality plots in backup slides

http://www.economist.com/node/16690695?story_id=16690695


11

Common Solutions – ExamplesGanga Extensively used as a “gridification tool” by many projects /

disciplines. This includes not only communities within EGI-InSPIRE but also others in many fields

Numerous non-HEP and non-HUC projects

Mini-Dashboard

To be used with Ganga to monitor Ganga-based activity on the grid. Used by EnviroGRIDS and offered to NA3

EnviroGRIDS, NA3

Experiment Dashboards

Common schema and code base for all job monitoring applications: job summary, historical view & task monitoring – implemented from July 2010

ATLAS + CMS

GRelC Exploited by the Earth Science (ES) community (Climate-G testbed) and by other projects/disciplines related to Environment and Bioinformatics

LS, A&A, ES

MPI High impact on multiple user communities CCMST, A&A, F

Frameworks Use of LHCb’s DIRAC framework by LCD/ILC and Belle collaborations. Investigation of DIRAC by ES + others

HEP (beyond LHC), ES

Data Management

Data popularity (dynamic data placement / caching), consistency of catalogs / storage

LHC VOs

Site Stress Testing

HammerCloud service used at ATLAS, CMS and LHCb. Fully applicable to other VOs / communities

HEP


Common Solutions


Life Sciences


Earth Sciences

Life SciencesConnect various tools with

multiple communities:the main disciplines and also

Fusion, Computational Chemistry

Fusion

HEP

Comp Chem


Common Solutions


Life Sciences


Earth Sciences

Life Sciences

Ganga: job definition and management in HEP

(ATLAS and LHCb, Compass, Harp), Fusion, L&E Sciences.

Fusion

HEP

Comp Chem


Common Solutions


Life Sciences


Earth Sciences

Life SciencesHammercloud: a site stress

testing system to validate site usability. Used by ATLAS, CMS and

LHCb.

Fusion

HEP

Comp Chem


Common Solutions


Life Sciences


Earth Sciences

Life SciencesDashboards: single entry point to

monitoring of all 4 LHC experiment activities on the grid. (Mini)

Dashboard also used by E&L Sciences.

Fusion

HEP

Comp Chem


Common Solutions


Life Sciences


Earth Sciences

Life SciencesDIRAC: workload and data

management used by LHCb plusother HEP experiments withinterest from Earth Sciences.

Fusion

HEP

Comp Chem


Common Solutions


Life Sciences


Earth Sciences

Life Sciences

MPI: used by A&A, Fusion, Computational Chemistry to handle

parallel execution in grid environments

Fusion

HEP

Comp Chem


Common Solutions


Life Sciences


Earth Sciences

Life SciencesGRelC: set of advanced data grid

services to manage Databases on the Grid.

Used by L&E Sciences and A&A.

Fusion

HEP

Comp Chem


19

EGI TF & UF• Actively participated in both EGI Technical Forum (September

2010, Amsterdam) and User Forum (April 2011 Vilnius)

– TF: Two sessions covering overview of tasks and sub-tasks; One session dedicated to discussion of Common Requirements

– UF: Numerous presentations, tutorials, demonstrations covering all areas of activity

– All sessions well attended with good feedback

• EGI TF 2011: sessions planned to highlight main achievements of WP in meeting goals plus a mini-workshop on sustainability


20

Broadening Use of the Grid

Concrete examples:

– Use of HUC solutions and expertise for “3 generations” of HEP experiments – LEP to ILC

– Bi-directional sharing of solutions and techniques not only for those activities explicitly supported through EGI-InSPIRE but also other projects, such as PARTNER, ULICE and EnviroGRIDS

– Other possible Earth Science projects in the pipeline


EGI-InSPIRE


21

Domain-Specific Work

A Summary of the Main Sub-Tasks: Details in Backup Slides


22

Services for HEP

• Distributed Analysis Support– Support for the tools used by

the experiments; common error-logger for ATLAS/CMS/LHCb

• Data Management Support– Dynamic caching / data popularity:

common solutions deployed• Persistency Framework

– Event and detector data: common to three LHC experiments

• Monitoring– All aspects of production and analysis:

additional common solutions deployed

www.egi.euEGI-InSPIRE RI-261323NAx - <<name>> - EGI-InSPIRE EC Review 2011

23

Services for Life Sciences

• TSA3.4 - Virtual Research Community building– Life Sciences Grid Community (LSGC)

• Services development and provision– TSA3.2.1 - Dashboard: design phase– TSA3.2.3 - Data encryption service: prototype

deployed– TSA3.2.3 - Database interface: GRelC deployed– TSA3.2.4 - Workflows: work on Taverna to start in

year 2– TSA3.4 - CoreBio services: work to start in year 2


Services for A&A• Preparatory studies:

– Visualization tools: VisIVO integration (TSA3.5.2)– Parallel (MPI/OpenMP) and GPU computing using

CUDA (TSA3.5.3)– Database services and integration with Virtual

Observatory (TSA3.5.4)• Achieved results:

– ms608: Gridification of VisIVO and VisIVO Service (TSA3.5.2)

– Parallel programming: testing activities with cosmological simulations codes (FLY, Gadget,Flash) (TSA3.5.3)

– Database services analysis (TSA3.5.4)

SA3 – Jamie Shiers – EGI-InSPIRE EC Review 2011 24


Services for ES

• TSA 3.6 Support ES Activities in ES communities and projects, and carried out by researchers & students in Universities

• Main activity by proposal: access to GENESI-DEC– Status : Webservice available and validated with application– Extensions dependent on GENESI-DEC progress– Further developments in Task: Integration with available GEOSS services to access

Genesi and CLimate Data from ESG

• Since Jan 2011 common developments with the climate Earth System Grid (ESG)

– Access to ESG data from EGI-Infrastructure and vice versa– Main problem to solve: A & A due to different federations– Institute IPSL/CNRS, IPGP now unfunded partner in TSA3.6



26

Shared Tools & ServicesTool / Service Description Communities

Dashboards Experiment Dashboards used by LHC VOs – single schema and application. Mini-dashboard – to be used with Ganga – also by ES

HEP, ES

Ganga / DIANE Both distributed analysis frameworks (ATLAS, LHCb) and general gridification tools

Wide range of diverse communities and applications

Hydra / GRelC Encryption service for LS; Grid-access to DBs and catalogues

LS, A&A, ES

SOMA2 / Taverna / Kepler

MPI Handles parallel execution in grid environments

CCMST, A&A, F


27

S.W.O.T. Analyses – D6.2Discipline Strengths Weaknesses Opportunities Threats

HEP 24 x 7 petascale computing

Time to resolve some incidents

Increased commonality

Need to adapt to new technologies

Life Sciences Ramp-up of LSGC

Issues with use of multi-grids

Community growth

No long-term funding


Successful use of DCIs

Coordination of A&A activities

Wider use of DCIs

Funding

Earth Sciences Existing user community

Diversity of demands and technologies

Community growth

Funding

GRelC Community based approach, cross - discipline

Little feedback about the SA3 DB questionnaire

Community growth

Lack of use of registry

MPI Widely regarded as an important tool

External factors Use of large SMPs / GPGPUs

Need for standardisation

Notable overlap in these independent analyses


EGI-InSPIRE


28

Use of Resources

Breakdown by Task & Partner


29

Effort Breakdown (1/2)Task Partner Worked PM Funded Committed PM Achieved PM %

TSA3.1 35-CERN 8.8 6.0 147%

TSA3.2.1 14A-CNRS 0 0.7 0%

TSA3.2.1 14C-HealthGrid 0.3 3.3 8%

TSA3.2.1 35-CERN 22.4 20.0 112%

TSA3.2.2 31B-JSI 0.5 1.0 47%

TSA3.2.2 32-UI SAV 2.6 6.0 43%

TSA3.2.2 35-CERN 17.1 20.0 86%

TSA3.2.3 14A-CNRS 7.5 3.3 224%

TSA3.2.3 14B-CEA 0 2.7 0%

TSA3.2.3 21E-SPACI 4.0 9.0 44%

TSA3.2.4 12C-CIEMAT 6.9 6.0 115%

TSA3.2.4 13-CSC 5.5 6.0 91%

TSA3.2.4 28C-ICBP 0.7 2.0 33%

TSA3.2.4 37-EMBL 0 6.0 0%


30


TSA3.1 35-CERN 8.8 6.0 147%

TSA3.2.1 14A-CNRS 0 0.7 0%


TSA3.2.1 35-CERN 22.4 20.0 112%

TSA3.2.2 31B-JSI 0.5 1.0 47%

TSA3.2.2 32-UI SAV 2.6 6.0 43%

TSA3.2.2 35-CERN 17.1 20.0 86%

TSA3.2.3 14A-CNRS 7.5 3.3 224%

TSA3.2.3 14B-CEA 0 2.7 0%

TSA3.2.3 21E-SPACI 4.0 9.0 44%

TSA3.2.4 12C-CIEMAT 6.9 6.0 115%

TSA3.2.4 13-CSC 5.5 6.0 91%

TSA3.2.4 28C-ICBP 0.7 2.0 33%

TSA3.2.4 37-EMBL 0 6.0 0%


31


TSA3.1 35-CERN 8.8 6.0 147%

TSA3.2.1 14A-CNRS 0 0.7 0%


TSA3.2.1 35-CERN 22.4 20.0 112%

TSA3.2.2 31B-JSI 0.5 1.0 47%

TSA3.2.2 32-UI SAV 2.6 6.0 43%

TSA3.2.2 35-CERN 17.1 20.0 86%

TSA3.2.3 14A-CNRS 7.5 3.3 224%

TSA3.2.3 14B-CEA 0 2.7 0%

TSA3.2.3 21E-SPACI 4.0 9.0 44%

TSA3.2.4 12C-CIEMAT 6.9 6.0 115%

TSA3.2.4 13-CSC 5.5 6.0 91%

TSA3.2.4 28C-ICBP 0.7 2.0 33%

TSA3.2.4 37-EMBL 0 6.0 0%


32


TSA3.1 35-CERN 8.8 6.0 147%

TSA3.2.1 14A-CNRS 0 0.7 0%


TSA3.2.1 35-CERN 22.4 20.0 112%

TSA3.2.2 31B-JSI 0.5 1.0 47%

TSA3.2.2 32-UI SAV 2.6 6.0 43%

TSA3.2.2 35-CERN 17.1 20.0 86%

TSA3.2.3 14A-CNRS 7.5 3.3 224%

TSA3.2.3 14B-CEA 0 2.7 0%

TSA3.2.3 21E-SPACI 4.0 9.0 44%

TSA3.2.4 12C-CIEMAT 6.9 6.0 115%

TSA3.2.4 13-CSC 5.5 6.0 91%

TSA3.2.4 28C-ICBP 0.7 2.0 33%

TSA3.2.4 37-EMBL 0 6.0 0%


33


TSA3.2.5 12A-CSIC 5.6 9.0 62%

TSA3.2.5 19-TCD 7.0 7.0 100%

TSA3.2.5 21D-UNIPG 11.5 3.0 385%

TSA3.3 21A-INFN 0 20.0 0%

TSA3.3 35-CERN 65.6 67.7 97%

TSA3.4 14A-CNRS 2.6 3.3 77%

TSA3.4 14C-HealthGrid 2.9 6.3 46%

TSA3.4 37-EMBL 0 8.7 0%

TSA3.5 21C-INAF 8.3 10.0 83%

TSA3.6 10G-FRAUNHOFER 2.1 9.0 23%

TSA3.6 14A-CNRS 4.5 8.0 57%

Total: 186.3 244.0 76%


34


TSA3.2.5 12A-CSIC 5.6 9.0 62%

TSA3.2.5 19-TCD 7.0 7.0 100%

TSA3.2.5 21D-UNIPG 11.5 3.0 385%

TSA3.3 21A-INFN 0 20.0 0%

TSA3.3 35-CERN 65.6 67.7 97%

TSA3.4 14A-CNRS 2.6 3.3 77%

TSA3.4 14C-HealthGrid 2.9 6.3 46%

TSA3.4 37-EMBL 0 8.7 0%

TSA3.5 21C-INAF 8.3 10.0 83%

TSA3.6 10G-FRAUNHOFER 2.1 9.0 23%

TSA3.6 14A-CNRS 4.5 8.0 57%

Total: 186.3 244.0 76%

If the Grid Observatory effort is removed, the underspend in PY1 is > 25%. Hopefully, this effort will be made up early in PY2.


35

Plans for Next Year

• In PY2 we will continue to identify and deliver common solutions within and across existing communities, further the work on sustainability and address key areas of weakness / concern identified in PY1

• This includes the evaluation – and eventual deployment – of new solutions / techniques

• No modifications to the DoW are foreseen• Carry-over of PM to later in the project?


36

Review of ObjectivesObjective Status

Supporting the tools, services and capabilities required by different HUCs

Achieved – work continues in PY2 / PY3

Identifying the tools, services and capabilities currently used by the HUCs that can benefit all user communities and to promote their adoption

Several additional items identified and shared across other communities – work will expand and continue in PY2 / PY3

Migrating the tools, services and capabilities that could benefit all user communities into a sustainable support model as part of the core EGI infrastructure

Not started – needs further work and discussion

Establishing a sustainable support model for the tools, services and capabilities that will remain relevant to single HUCs

Collaborative support is the basic model for sustainability that does not depend on individual partners nor specific project funding. A workshop on sustainability at the EGI TF is proposed – work continues and expands in PY2 and PY3. See also D6.2.


Summary Successfully supported major production computing at an unprecedented scale

– both quantitatively and qualitatively Successfully delivered common solutions in a variety of areas – with other

activities in progress Actively participated in EGI Technical & User Forum via presentations, tutorials

and demos Broadened the use of grid technology and HUC services to related projects

within the HUC domain (such as unfunded – by EGI-InSPIRE – LS / ES projects) Completed first round of Milestones & Deliverables together with associated

technical work Identified – across all HUC communities – areas of common technology

investigation for the future Developed a S.W.O.T. analysis of each main discipline and made significant

steps on the road to sustainability



EGI-InSPIRE


38

BACKUP SLIDES


Milestones & Deliverables


Milestone / Deliverable

Due Date Lead Partner (#) Title

MS601 PM1 CSC (13) HUC Contact points and the support model

MS602 PM4 INFN (21) HUC Software Roadmap

MS603 PM4 CERN (35) Services for High Energy Physics

D6.1 PM4 CERN (35) Capabilities offered by the HUCs to other communities

MS604 PM4 CNRS (14) Services for the Life Science Community

MS605 PM8 TCD (19) Training and dissemination event

D6.2 PM9 CERN (35) Sustainability plans for the HUC activities

MS606 PM10 INFN (21) HUC Software Roadmap

D6.3 PM11 CERN (35) Annual Report on the Tools and Services of the HUCs

MS607 PM12 CNRS (14) Hydra service deployment

MS608 PM12 INFN (21) Integration of the VisIVO server with the production infrastructure

https://documents.egi.eu/public/ShowDocument?docid=91












EGI-InSPIRE


40

HEP


41

WLCG Service Incidents


42

SIRs – Time to Resolution


43

GGUS Tickets – LHC VOs

13-Ja

n23

-Feb5-A

pr

16-M

ay26

-Jun6-A

ug16

-Sep27

-Oct7-D

ec17

-Jan27

-Feb9-A

pr

20-M

ay30

-Jun

10-Aug

20-Se

p31

-Oct

11-Dec

21-Ja

n3-M

ar13

-Apr0

20

40

60

80

100

120

Total ALICETotal ATLASTotal CMSTotal LHCb


Data Management

• Crucial area for LHC and other HEP experiments– Data volumes: tens of PB/year, rates: up to 200TB/day between

sites, several hundred active analysis users / experiment, 1M analysis jobs / day

• Experience from first data taking has shown that some assumptions on data placement are no longer optimal– based on decade+ old model “MONARC” which assumed network was

scarce and expensive resource

• Working to adapt computing models to reflect such changes



Data Placement & Dynamic Caching

• Based on MONARC, the initial phase of LHC data distribution was based on static pre-placement – Significant fraction of such data never read!

• Computing models now driving towards dynamic data placement– Replication is based on usage (“popularity”) – this results in

better network and storage utilization

• Implemented first for ATLAS, now for CMS and LHCb



Catalogue Consistency

• With 50PB of data storage across a large number of sites worldwide, inconsistencies can easily arise!

– Data that resides on Storage Elements but not in various catalogs (grid, experiment) referred to as “Dark Data”

• One site recently reported 70TB dark data!

• Using a messaging-based system, various catalogs and SEs can talk to each other and implement lazy synchronization



Data Analysis Support

• Covers the final stage of data processing leading on to publication– Large number of users/month (~1000) and analysis jobs/day (~1M) running across

(~100) Tier2 and other sites – “chaotic” data access– All frameworks support heterogeneous back-ends

• Ganga (ATLAS, LHCb) used by 10 other communities and 500 – 600 users

• Common site stress testing system (Hammercloud) used by ATLAS, CMS and LHCb

• Areas of commonality and optimization– Move to “community support” model– Simplify data access and improve monitoring– Use common components and frameworks, such as for job submission and file

transfer (built on gLite /EMI FTS)• An area of potential future common work and simplification



EGI-InSPIRE


48

LS


Life Sciences Grid Community

• LSGC: http://wiki.healthgrid.org/LSVRC:Index – 4 VOs, 6 NGIs, HG association, 2 EU projects– Communication channels: wiki, mailing lists,

monthly phone conferences• Technical team (shifters)

– Infrastructure monitoring and troubleshooting (Nagios server with LS resource probes)

• User management tools – early design phase– User registration and management DB– To be integrated in LS Dashboard

SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011

http://wiki.healthgrid.org/LSVRC:Index

www.egi.euEGI-InSPIRE RI-261323NAx - <<name>> - EGI-InSPIRE EC Review 2011

50

LS Dashboard• Need to integrate

– Nagios monitoring dedicated interface (possibly based on GOC dashboard)

– VRC-level accounting information– User management tools (possibly based on

VOMRS)


Data encryption service• Hydra server

– Encryption keystore server based on Shamir’s secret sharing algorithm

– Software packages available for gLite 3.1– Installation procedure documented for gLite 3.0

• Current status (MS607)– Working on gLite 3.1, not yet on gLite 3.2– Client CLIs to be installed on all LS supporting sites

• Perspectives– Deployment of a three-heads hydra server on three

sitesSA3 – Johan Montagnat – EGI-

InSPIRE EC Review 2011


Database interface

• GRelC (Grid Relational Catalog) service provision to support LS use cases

• Discussion about new LS use cases and data resources analysis

• Identification of a couple of biological databases to be ported in grid (relational DBs)

• Contribution to the SA3 questionnaire related to “grid-databases”



LS issues, mitigation & perspectives

• VOMS and LFC servers are single point of failures– Replication procedures being set up

• Infrastructure monitoring is time consuming– Scheduled downtimes should be better reflected in BDII– Dedicated view of Nagios results in LS dashboard– Nagios probes improvements

• LSGC is a multi-VOs / multi-grids community– No tooling available to manage VRCs– Multi-grids hardly addressed in the context of EGI

• Few feedback on the SA3 questionnaire– More dissemination is needed in PY2



EGI-InSPIRE


54

A&A


A&A Grid Community

• 23 VOs and 7 NGIs– The most part of active A&A VOs relate to the

astroparticle physics community– Communication channels:

• wiki (under construction)• mailing lists (a general mailing list currently in place,

others more specialized could follow) • phone conferences (organized on a monthly basis)

– A&A VRC meetings and workshops• The first one of the EGI era is under organization in

Paris (7 November 2011, ADASS Conference)

SA3 – Johan Montagnat – EGI-InSPIRE EC Review 2011 55


VisIVO• Porting of VisIVO server in Grid.

– Preparatory activity: enabling the usage of VisIVO directly within a code during the production phase.

• A software layer has been developed using the internal arrays and without the need of producing intermediate files.

• A library of VisIVO was designed and implemented– The first issue of the gridified version of VisIVO

server has been released in April 2011 (milestone MS608).



A&A issues, mitigation & perspectives

• A&A is a complex community: its coordination is quite challenging.

• No EU A&A projects currently funded shortage of funds coordination activity tricky no easy deployment of tools and services

• Our willingness is to continue the coordination and the efforts to strengthen the community– We rely on EGI.eu and NGIs support– We exploit as much as possible collaborative

tools and services to establish robust and stable communications



EGI-InSPIRE


58

ES


ES HUC

• ES HUC VOs:– “ESR VO” „catch all ES people“ provides resources and support for ES

projects and anorganized ES users – EU project VERCE (seismology) will start with this VO and will

work with SA3.6• VOs associated to ES HUC

– See http://www.euearthsciencegrid.org/content/egee-virtual-organisations-earth-science

• ES HUC Activities: – Yearly session at European Geoscience Union, General Assembly

2011 about “ES and E-infrastructures


http://www.euearthsciencegrid.org/content/egee-virtual-organisations-earth-science

http://www.euearthsciencegrid.org/content/egee-virtual-organisations-earth-science


EGI-InSPIRE


60

Other Tools & Services


GRelC: Activity (I)

NAx - <<name>> - EGI-InSPIRE EC Review 2011 61

• User support through the implementation of a grid-database “registry” (EGI Database of Databases)– Easy search and discovery (cross-VO) of grid-DB

resources distributed across the EGI grid. – Community-based approach to attract new users and

address sustainability• SA3 Questionnaire – A census about database

resources, related needs and future plan– Distributed among the HUCs (end of Q3)– Few feedback; more dissemination during Y2


GRelC: Activity (II)


• The back-end modules of the “registry” finalized during Y1:– A MySQL catalog for the registry– Several Java classes to manage charts, grid-DB, VOs,

GRelC services, community-oriented aspects• Front-end modules related to the “registry”

finalized in Y1 (now tested, on line during Q5)– Registry view completed– Grid-Database view completed – Integration of Web2.0 (Mash-up, Google Maps,

permalinks) and community-oriented aspects (comments, scores, discussions groups, etc.)


GRelC: Registry Snapshots



GRelC: User support


• HUC support in terms of:– grid-metadata management for the Earth Science and

Environmental context (e.g. Climate-G, CMCC)– setup and hosting of a new GRelC service to implement

and run LS use cases (e.g. biological)– grid-DBs census and requirements collection for the HUC

through the SA3 Questionnaire (end of Q3)– Training and documentation resources

• Issues that arose:– Few feedback (from end of Q3) regarding the SA3

Questionnaire (further dissemination is needed during Y2, EGI-UF, EGI-TF, etc.)


GRelC: Plans for next year• The registry will be available online in PQ5• It will be the core part of the DashboardDB

(available during Y2)• The DashboardDB will provide specialized

views, charts and statistics about the GRelC instances deployed across the EGI grid

• The SA3 Questionnaire will be refined and distributed among the HUCs – The feedback will be reported into the registry

• No changes to the current plan are foreseen



66

Workflows / Schedulers– TSA3.2.4 – Kepler:

Kepler actors for gLite and Unicore fully operational. Tutorials provided, material available online. Use cases designed, created, and deployed. More use cases in year 2.

– TSA3.2.4 – GridWay: Available and used as standalone metascheduler. Integration GridWay - Kepler, to start in year 2.


Workflows / Schedulers

• Kepler actors for gLite and Unicore fully operational

• First workflows with different use cases have been developed

• Still work required for GridWay• Tutorials provided, online material• Interest of communities reached• SOMA2… (next)


SOMA2• SOMA2 in EGI-InSPIRE

– WP6: Services for the Heavy User Community (SA3)• TSA3.2 Shared Services and Tools• TSA3.2.4 Workflows and Schedulers

– 1st Project Year Goals• DCI integration

• Support for use of Grid middleware.• Users’ X509 certificate handling.

• Grid enabled services’ setup• Autodock 4 integration

• SOMA2 1.4 release• Includes grid support + more

– SOMA2 as a Service• Currently provided for Finnish academic researchers• In our roadmap we plan to offer the service to EGI as well (2nd year)

http://www.csc.fi/soma

http://www.csc.fi/soma


69

MPI/Parallel Computing (I)

• Cross disciplinary activity and support• PY1 core objective achieved

– Centralized documentation in the wiki:• Admin Manual: https://wiki.egi.eu/wiki/MAN03• User Guide: https://wiki.egi.eu/wiki/MPI_User_Guide

– Multiple application models and algorithms– Outreach, training and dissemination

• Training material centralized at user guide wiki: https://wiki.egi.eu/wiki/MPI_User_Guide#Application_Execution

• Modest increase in #production sites– Improvements in monitoring => better service


70

MPI/Parallel Computing (II)

• User defined processes per node allocation– implemented in EMI-1 WMS, CREAM and MPI-Start– However, not yet in production– Extensive exploitation/testing by CCMST/ INFN (See

Laganà et al talk @ EGI UF 2011)– Hybrid OpenMP/MPI testing by TheoMPI (see Alfieri

et al. talk @ EGI UF 2011)• GP-GPU integration (with KVM)

– Some open issues • batch support, Lack of standardisation, Accounting etc


CCMST (volunt. serv.)• Virtual Research Community building

– Computational Chemistry & Material Sciences and Technology (CCMST) members certification and user support

• Services development and provision– GriF - a user friendly tool for job distribution on the

grid providing QoU and QoS information– GCres - a credit award system based on GriF – Workflows - evolution of Kepler, Pgrade and others– Packages and programs - Gaussian, GEMS,

Chimere, DL_Poly, Gromacs, CMAST– Virtual laboratory - Insilico Lab


Www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE EGI-InSPIRE RI-261323 EGI-InSPIRE Review: Work...

Documents

Transcript of Www.egi.eu EGI-InSPIRE RI-261323 EGI-InSPIRE EGI-InSPIRE RI-261323 EGI-InSPIRE Review: Work...