SEE-GRID-SCI Branko Marovic JRA1 Leader University of Belgrade JRA1 overview and planning PSC01...

www.see-grid-sci.eu

SEE-GRID-SCI

Branko MarovicJRA1 Leader

University of [email protected]

The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no. 211338

JRA1 overview and planning

PSC01 Meeting, Athens, 22-23 May 2008

The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no. 211338

PSC01 meeting, Athens, 22-23/05/08 2/28

JRA1 Objectives (DoW)

Capture commonalities across scientific fields in terms of application requirements on Grid middlewareDefine development areas for middleware plug-ins and application-level services to cater to application demands and provide improvements to current infrastructure servicesImplement application-specific services and middleware extensionsProvide a baseline of operational tools from the perspective of application and end-user friendliness and define development areasDevelop and deploy operational tools easing the application-specific monitoring and control

Quite vague objectives – must be grounded ASAP!


Common JRA1 goals

“A seamless environment to a number of research communities” Meteorology, Environment, Seismology (MES)

Improve usability of the infrastructure and Grid services for the SEE-GRID-SCI user communities and their applicationsAddress issues not addressed by the existing middleware and infrastructure.Services/plug-ins should Have the capability to provide benefits to a wider range of ERA and Grid users Some should be discipline-neutral to a certain extent Built upon the existing middleware (gLite) Incorporate present security and operational infrastructure.

Improve manageability of the infrastructure Research the application-focused features of operational tools Develop new tools Extend existing tools towards the needs of the scientific communities, VOs, and

applicationsTools should Add functionality and improve both the flexibility and reliability of the tools Facilitate the deployment, monitoring and control of applications.

Development of tools and middleware plug-ins will be jointly supported by NA4 and JRA1 activities


JRA1 numbers

24 months 87.5PMs 395k€12.9% of the overall requested PMs, 12.3% of the total budgetRTD focused on the development of few services and tools for a target group of application communities Contributions to SA1 and NA4 Limited number of specialised partners

Strong commitments (RTD!) Pay per performance

Others can contribute through SA1 and NA4CERN IPP ICI TUBITAK SZTAKI UoBL UKIM UOB RBI TOTAL

5,50 15,00 9,00 12,00 4,00 5,00 8,00 27,00 2,00 87,50


Tasks

JRA1.1 – Application requirements analysis Leader: UoB, B.M.

JRA1.2 – Operational tools analysis Leader: IPP, E.A.

JRA1.3 – Implementation of application-specific services Leader: TUBITAK, C.O.

JRA1.4 – Design, implementation and assessment of the application-control tools Leaders: IPP, E.A. (d/i), ICI, A.S. (asmnt)

JRA1.5 – Application extensions deployment and assessment Leader: UKIM, B.J.

Metrics MTJRA1.1 - Application-level services developed

M12: 2, M24: 6 - 2 per VO MTJRA1.2 - Operational tools developed

M12: 2, M24: 4

Project administrative and technical management

NGI support and international collaboration

Dissemination and Training

JRA1

Application extensions deployment and assessment

Implementation of application-specific

services

Development of application-level services and operational tools

Application requirements analysis

Design, implementation and assessment

of the application-specific

monitoring and control tools

Operational tools analysis


Risks

R-JRA1.1: Dependency of application service development on integration with low-level gLite middleware components Opening date: M06 The project significantly relies on the developments of EGEE for the provision of middleware components of gLite. Although previous agreements have been established with EGEE regarding the integration, if needed, of SEE-GRID-SCI application-level plug-ins with gLite releases, there is a potential risk in delays by EGEE integration activity. Minimise software dependencies and produce truly pluggable application-level services to avoid the integration risk. If integration delays are inevitable, aim to run the application without specific application add-ons.

R-JRA1.2: EGEE middleware components and operational tools Opening date: M03 Developed operational tools that are extensions of and/or are based on the operational EGEE tools and middleware. In case of tools or middleware instabilities and/or major interface changes there is the risk that JRA1 developed tools will fail their purpose or be incompatible with the new tools and/or middleware. Functional Grid middleware is necessary for any Grid operation. The fall-back solution: using latest stable middleware and tools versions. Consider this risk from the very beginning and ensure the development of lightweight and generic tools that can easily adapt to middleware changes. Close follow-up of EGEE development plans will ensure the adequate development planning.


Deliverables

DJRA1.1 - Grid application requirements and commonalities assessment M06, 7 PM, UoB-RCUB, Editor: B.M.

DJRA1.2 - Application-specific monitoring and control tools design M08, 12 PM, IPP, Editor: E.A.

DJRA1.3 - Application services design report M12, 10 PM, UoB-RCUB, Editor: B.M.

DJRA1.4 - Application services implementation and deployment report M22, 30 PM, UoB-RCUB, Editor: B.M.

DJRA1.5 - Assessment of operational tools developed M23, JRA1 28,5 PM, ICI, Editor: A.S.


Milestones

MJRA1.1 - Baseline analysis of operational tools M03, IPP: wiki, informal report

MJRA1.2 - Application commonalities assessed and areas of development identified M06, UoB-RCUB: DJRA1.1

MJRA1.3 - Operational tools implemented and running M16, IPP: wiki, informal report

MJRA1.4 - Selected application services implemented and deployed M24, UoB: DJRA1.3, demonstration at review


JRA1 harmonisation

Clear separation in DoW high-level PERT, JRA1 tasks, deliverables and milestones - try to establish a synergy between application-level services and operational tools developments through Shared presentation of offered services and operational tools to user

communities Using operational tools to monitor some application-level services Feed services with data from operational tools (statistics,

configurations)Possible discrepancy between DoW JRA1 commitments and actual needs of applications Must harmonise with NA4 now when we have user communities

onboard Analyse the commonalities across disciplines and suggest common approaches Can we transcend application and community borders with plug-ins

and tools?Address above in NA4 (MES uses/apps) and JRA1 (services/tools) surveys


Expected impact of JRA1

Ease the involvement of new users, applications and resource providersAutomating and simplifying common and burdensome tasksWider influences Results available to other regional initiatives Provide its most successful tools and components to European

projects and initiatives that have the leverage to provide them to other players – EGEE, OMII

Take part in European and global standardization efforts and bodies, especially the Open Grid Forum

Contributions to Optimum development and use of eInfrastructures Establishing a future rich Grid environment for the needs of European

Research Aea


Development of app. services

1. Assessment of target applications’ requirements and available components and solutions.

2. Definition of goals and scope; specification of key features, functional requirements and draft interfaces.

3. Plan for reuse, adaptation and modification of existing solutions and development of necessary additions; detailed specification and assignment of related activities.

4. Building of solution components and development documentation; producing functional specifications and testing procedures.

5. Testing in a realistic environment with pilot applications; handling of detected problems; review of development and deployment documentation.

6. Deployment of production-level solutions through activity SA1, starting operational usage and support; employment with all interested communities and target applications; assessing user experiences and accomplished results.

JRA1JRA1.1JRA1.5

JRA1.3JRA1.2

6.Deployment

5. Testing

4. Building ofcomponents

3. Implementationplan

2. Definition ofscope andfeatures

1. Assessment ofrequirementsand solutions

JRA1.4


Application-level services

1. Assessment of requirements and solutions (JRA1.1, DJRA1.1, MJRA1.2, M06) Define development areas for middleware plug-ins and application-level services, initially

Data access and management Multidimensional visualization Interactivity

Inputs and expertise that go beyond local groups in all three communities Usage, processing and visualization of acquired/calculated spatial data Location and management of data used in collaboration and resource sharing

2. Selection and specification of commonalities (JRA1.1, DJRA1.1, MJRA1.2) Based on expertise of participating partners and user communities

3. Implementation plan (JRA1.1–JRA1.3) Selection of the designers, developers, testers and reviewers of services and plug-ins

4. Development (JRA1.3) Design (DJRA1.3) Implementation (DJRA1.4)

5. Testing (JRA1.3, DJRA1.4, MJRA1.4) Test and consolidation with one application Reuse

Inclusion into additional target applications if identified, or Offer for usage to the wider community

Fine-tuning of plug-in interfaces, implementation, documentation and deployment6. Production-level deployment & dissemination (JRA1.5, DJRA1.4, MJRA1.4)

Operational usage and support Wider dissemination and assessment

Feeding to WLCG/EGEE/gLite teams for potential assessment (EGEE RESPECT?) Integration with the gLite foundation services if necessary – although not envisaged


Meteorological user communityservices

Led by UOBOnly expectation that are not specific for a single local group will be addressedPotential services mentioned in DoW UOB (data access and management, interactivity - generic)

Extension of SEE-GRID File Management Java API with application-dependant metadata and dataset geo-referencing

Job-pooling? Application-specific events

RBI (multidimensional visualization - generic) Investigate multidimensional visualisation by defining a standardized inter-

application communication Integration with an meteorology application FGA: cooperate in scientific visualization technologies and their development and

implementation using grid with new scientific visualization methodologyThe above may bee too generic - more attention may be needed regarding Streamlining of forecast ensemble runs using different models Fusion of ensambles (spatial blending, identification of representative

outcomes from ensembles) Post processing/interpretation from the perspective of end consumers


Environmental user community services

Led by IPPIPP will analyze the possibility of ensuring (generic) QoS for job start/job execution Utilization of available performance data for improved throughput

ICI and its third parties (UTCN, NCIT, UVT) will identify Grid requirements for satellite image processing in environmental applications and other similar domains ESIP platform

Grid based satellite image processing software SOA providing the basic functionality for Grid and Web applications Develop a methodology for application development on this platform Experiment this approach on a pilot application developed using this

platform according to the NA4 activity workplan Investigate usability for other application areas


Seismological user communityservices

Led by TUBITAKDesign of plugins integrating applications and the middleware TUBITAK

Extension to the high-level programming interface tested with SDA, to cater for Seismic Risk Assessment

Massive Digital Seismological Signal Processing with the Wavelet Analysis UKIM (generic)

Middleware extensions which will enable easier and more reliable job execution.

Seismological data repositories 4 Tbyte of data is foreseen to be produced per year from the stations

of the participating seismology observatories. No foresight of computational requirements for applications

Storing and archiving of this huge amount of data will need deploying and testing of new grid services like dCache, DPM and FTS

Applications are not computationally intensive, thus SEE-GRID-SCI computing resources will be adequate


Analysis, extension and integration of services

Overall process management by UOBTUBITAK Contribute to the application requirements analysis which will assist to the application services

design As NA4 leader, oversee the applicability and feasibility of all designed tools and services for all core

VOs. NA4 survey of the 3 user communities & analysis of their requirements

Start immediately through detailed applications questionnaire The questionnaire should also capture JRA1 related requirements

In areas of seismology and meteorology SZTAKI will Support the analysis of existing solutions, their extensions and integration, focusing the application-

specific requirements by the help of its Grid Application Support Centre Analyse the possibility of the design and integration of two plug-ins in P-GRADE Portal (already

involved in the EGEE RESPECT program) in order to ensure the dissemination and exploitationCERN Alignment of SEE-GRID-SCI JRA1 developments with EGEE SA1 and JRA1 through

Avoiding of overlapping in implementation Prevention of implementing application services based on or oriented towards infrastructure elements and

services that will be phased out Check for existing analyses within other EGEE-related projects of user and application requirements

not already covered by the existing middleware R-JRA1.1 watchdog Alignment of SEE-GRID JRA1 assessment practices with EGEE SA3 If needed, be a liaison between the SEE-GRID-SCI JRA1 team and EGEE SA3 (Integration, Testing and

Certification) and JRA1 (Middleware Reengineering) M12: 2, M24: 6 - 2 per VO


Preliminary survey of application services and operational tools

Piloting/testing of surveys proved to be very usefulAfter the start of SEE-GRID-SCI we conducted a survey of those who are about to develop something within JRA1 13 propsals from 6 countries 4 tools 6 services 3 tool/services

7 with Concept status11 usable in < 1 year


Preliminary survey of application services and operational tools

Acronym or short nameSEE-GRID-SCI

partner acronym

Country Service Tool Status Usable after

Logwatch modules RBI Croatia X Concept 2 - 6 months

SELinux for grid RBI, CIC Croatia X Concept 6 months - 1 year

Advanced-workflow development tool and orchestration service

UKIM FYR of Macedonia X Concept 1 - 2 years

User-perspective monitoring tool UKIM FYR of Macedonia X Concept 6 months - 1 year

P-GRADE Portal SZTAKI Hungary X X Finished (grid) 0 / done

ESIP (Environment oriented Satellite Data Processing) Platform

UTCN Romania X In development 6 months - 1 year

ESIP/UVT (front-end or app?) UVT Romania X Concept 1 - 2 years

No-mercy ticketing ICI Romania X Concept 6 months - 1 year

Data Management Web Portal UOB Serbia X Finished (grid) Under a month

Event Logger UOB Serbia X X Finished (cluster) 2 - 6 months

Job Binder UOB Serbia X X In development 2 - 6 months

SEE-GRID File Management Java API UOB Serbia X Finished (grid) 0 / done

SDS (Seismic Data Server) ULAKBIM TUBITAK Turkey X Concept 2 - 6 months


Operational tools

Research the application-focused features of operational tools and develop new or extend existing tools Facilitate the deployment, monitoring and control of applications Complement the current monitoring tools Add functionality to existing ones Improve tools’ flexibility and reliability

The core of tools should, from the point of view of application developers and users Collect data relevant to the infrastructure operations and make it

available to the operators and users Optimize the performance and reliability Automate

Operational tasks and procedures related to application deployment and running

Follow-up of discovered problems by creating tickets


Tools extension and development

Analyze and evaluate existing operational tools (JRA1.2) against the SEE-GRID-SCI infrastructure and application requirements through an analysis of the existing operational tools (MJRA1.1, M03)Design and implement the operational tools easing the application-specific monitoring and control (JRA1.4, DJRA1.2, MJRA1.3)Finally, the activity will carry out the assessment of the operational tools developed (DJRA1.5)IPP will be responsible for the design and implementation of the application-specific monitoring and control toolsICI will contributeCERN Check for existing operational tools that in some other EGEE-related contexts

address scientific communities and applications (not only infrastructure) Support the analysis of existing EGEE operations and user support tools Investigate their usability and areas of improvement or extension Be a lead liaison between the SEE-GRID-SCI JRA1 team and the EGEE SA1 and,

if needed, EGEE SA3 (Integration, Testing and Certification)


Candidate tools from DoWCollect operational application data and make it available to the operators and users from a web-based and web-service based front-end IPP

Lead the design and implementation of a centralized operational tool providing web interface to relevant operational data focusing on application features

UoBL Web interface integrated with existing GRID service monitoring tools such as BBmSAM, to create one-stop-shop

for easy monitoring of applications and resources IPP and UKIM

Mechanisms for gathering statistics about use and performance of the main Grid services (CPU, storage, file transfers) by the target user applications and communities, integrated into a single point of access for users via the above web service interface.

Develop tools that automate some operational tasks and procedures which are particularly related to the deployment and running of the grid applications ICI

Study the interactions between existing operational tools Design and implement an interface between the monitoring and control tools and the trouble ticketing system Further developing the trouble ticketing system in order to provide a better application-specific support.

Other extensions UoB

Web-based tool for application-specific event logging and analysis Assess the operational aspects of usage of SEE-GRID File Management Java API and Web-based Data

Management Front End in administration of storage elements and management of files, file replicas and metadata.

SZTAKI Focus on a workflow oriented middleware portal extension, which enables all the research communities of SEE-

GRID-SCI to use a grid-aware workflow repositoryM12: 2, M24: 4


User operational tools (1/2)

ARDA dashboard - http://dashboard.cern.ch/ The dashboard collects information on job processing, data management, transfer tests, monitoring of

the reconstruction at Tier-0, or monitoring of site efficiency of LHC experiments (ATLAS, ALICE, CMS and LHCb). It works on different grid flavors (OSG, LCG, gLite).

CRAB - Cms Remote Analysis Builder - http://cmsdoc.cern.ch/cms/ccs/wm/www/Crab/ CRAB generates scripts and additional data files for each job which are submitted directly to the Grid

using BossLite to interface to the Grid scheduler, as well as for logging and bookkeeping. CRAB supports any CMSSW based executable, with any modules/libraries, including user provided ones,

deals with the output produced by the executable, provides an interface to CMS data discovery services (DBS and DLS), which are completely hidden to the final user. It also splits a task (such as analyzing a whole dataset) into smaller jobs, according to user requirements.

FCR FCR allows the VOs to define a preference on Grid resources, optionally taking the SAM test results in

account as well. They can select the set of Critical Tests for all services, set of Site Resources (CEs and SEs) to be used by the VO and set of Central Service Nodes ( Note : this will be used in the future)

UIPnP – User Interface Plug & Play This is the easiest way to access the grid and it will allow you to use the shared resources of nodes that

already belong to the Grid. UIPnP contains all files in a unique directory without any RPM. It is intended to be used by any user (no root privileges are needed!) of commonly used RedHat platforms, without changements of system configuration.

Parallel Pipe - http://gforge.cyfronet.pl/projects/parpipe/ Tool for executing piped commands in a parallel environment (Is this a service/commonality?)

http://dashboard.cern.ch/



http://cmsdoc.cern.ch/cms/ccs/wm/www/Crab/



http://gforge.cyfronet.pl/projects/parpipe/




User operational tools (2/2)

Migrating Desktop - http://desktop.psnc.pl/ Migrating Desktop is a framework that allows the user to access transparently the Grid resources, run

sequential or interactive, batch or MPI applications, monitoring and visualization, and manage data files. The graphical user interface integrates and makes use of number of middleware and integrates the individual tools into a single product providing a complete grid front-end.

OCM-G - Online Monitoring of Grid Applications using OCM-G - http://grid.cyfronet.pl/ocmg/

The purpose of the OCM-G is to provide on-line information about a running parallel / distributed application to application-development-support tools, specifically performance analysis tools, like the GPM tool. Using the information from the OCM-G, tools are enabled to obtain performance measurements of the monitored application, related to, for example, delay and volume of communication, CPU usage, etc.

PHPMySRB –PHPMy Storage Resource Broker - http://www.twgrid.org/Application/ndap/

PHPMySRB is friendly web user interface for defining and manipulating collections of files, as well as for creating personal data collections and sharing, searching, replicating, and more without installing any client software. SRB is a system in Academia Sinica, Taiwan used for the long-term preservation of the digital contents.

ELFI - EGEE Grid storage in a local filesystem interface - http://www.egrid.it/sw/elfi/ ELFI is a filesystem interface to the LFC catalog and EGEE SE's (both SRM-enabled and "classic SE").

With ELFI, you can see the entries in the LFC catalog as files in a locally-mounted filesystem, and directly operate on the replica contents: read/write operations on the local filesystem are acted as read/write operations on a remote SE via the SRMv2 and GSI-RFIO protocol.

GridAssist - http://dsgrid.xs4all.nl:8080/gridassist/index.html This tool provides access to Grids directly from the user desktops. It has an intuitive graphical user

interface which makes it very easy to use in order to create and run scientific data processing jobs and it takes care of the transfer of the data between the resources.

http://desktop.psnc.pl/



http://grid.cyfronet.pl/ocmg/



http://www.twgrid.org/Application/ndap/



http://www.egrid.it/sw/elfi/



http://dsgrid.xs4all.nl:8080/gridassist/index.html


Conducting collaboration

Collaboration in development and use of JRA1 results [email protected] Gridification guide Piggybacking NA4 and VO collaboration activities Workshops/trainings?


Intellectual property rights

Additions/modifications to readily available software Use Free / Libre / Open Source Software (FLOSS) whenever

possible The same user license as the original software, commit back to

the source tree Copyright of new source code: the consortium (+original authors)

Changes in the a non-FLOSS Require permission from the copyright Negotiate status of changes and copyrights

Development of completely new software FLOSS license Copyright: the consortium or authors Later external community additions - shared FLOSS copyright


Actual issues

NA4 applications surveyJRA1.1 application requirements and commonalities surveyMJRA1.1 – Baseline analysis of operational tools - M03DJRA1.1 – Grid application requirements and commonalities assessment - M06MJRA1.2 – Application commonalities assessed and areas of development identified – M06

Discussion Dependency of application service development on integration with low-level gLite middleware

components. Maintenance of existing gLite services, development in standardisation area Recommendation: Minimise software dependencies, do not hardwire to the middleware

Test-environment needed? JRA1 developments are initially deployed locally Test-environment for services – significant overhead Most likely compromise of the infrastructure by DoS-like behaviour Disperse load among VO services Are current monitoring and ticketing tools sufficient? Quick fix – can we use Training VO? Review this decision when more experience is gained

Can we transcend application and community borders with plug-ins and tools? Important element of commonalities analysis

Immediate actions Further articulation of service/tool proposals Articulation of the needs of user communities


Articulation of the needs of user communities (1/2)

Must be completed by the end of the summer!Finding of other analyses of user and application requirements and examples of integration of operational tools with applications (CERN) would this process a lot.

1. Preliminary offering of common services and monitoring tools – done DoW Preliminary survey (UOB) Draft list of candidate tools (CERN)

2. Outlining of community, applications, and infrastructure (SA1) involvement –NA4 survey + 2 weeks

Surveying of target applications (NA4 survey, TUBITAK, C.S.) Infrastructure-driven requirements for the tools (SA1, MJRA1.1, IPP, E.A.) Analysis of existing operational tools and their possible extensions towards applications and

communities (inputs from SA1, CERN, VOs and applications). This is also the input for MJRA1.1 artifacts that will be later incorporated into DJR1.2 (IPP, E.A.)

3. Presentation of offered services and operational tools – 20-30 June Service (UOB, B.M.) /tool (CERN, D.S.) descriptions (refine the current list, agree ob required details

with SA1 and VOs) Input on other analyses (CERN, D.S.) Survey among JRA1 developers (UOB, B.M.) All who are intending to develop services and tools should fill the questionnaire or/and prepare

introductory presentations of services and tools (All JRA1 developers) Mandatory service/tool presentations for (3)? If so, do we really need a questionnaire?…


Articulation of the needs of user communities (2/2)

4. Matchmaking with interested applications (second phase of NA4 survey?) (TUBITAK, C.S, UOB, B.M.) - July-August

Review outputs from survey (3) from the perspective of user communities, applications, and their interaction with the infrastructure

Identification of known/expected infrastructure related issues (directly from SA1 after NA1 applications survey)

Conducting the survey among applications– Offer already identified preliminary services and tools– Identification of possible outstanding commonalities (non-covered issues)– To be filled by developers of all applications (Representative developers of all MES application)

5. Analysis of commonalities (MJRA1.2, DJRA1.1, UOB, B.M.) – September-next PSC

Analysis of survey (4) Assignments for developments not covered by offered services and tools,

if any(4) and (5) will provide the necessary inputs for both DJRA1.1 "Grid application requirements commonalities assessment“ and DJR1.2 "Application-specific monitoring and control tools design“

SEE-GRID-SCI Branko Marovic JRA1 Leader University of Belgrade JRA1 overview and planning PSC01...

Documents

Transcript of SEE-GRID-SCI Branko Marovic JRA1 Leader University of Belgrade JRA1 overview and planning PSC01...