PRACE-1IP – Supporting the Implementation of the PRACE Infrastructure

Thomas Eickermann – Jülich Supercomputing Centre

• History, Status and Development

• Commitment

• Organization

• Peer Review and Access

• Implementation Phase Projects

• Computational Science and Engineering

Content

20.9.2011 2Thomas Eickermann- FZJ

• The EU has to increase its high standards in computational science and computational engineering

• The EU has to guarantee independent access to HPC-systems of the highest performance class for all computational scientists and European industry

• Scientific Excellence requires peer review on European scale to motivate and foster best ideas and groups

• User requirements as to a variety of architectures will profit from coordinated procurement

• EU and national governments are motivated through PRACE to establish robust and persistent funding schemes

Why do we need a “Partnership for Advanced Computing in Europe?

320.9.2011 Thomas Eickermann- FZJ

HISTORY, STATUS AND DEVELOPMENT

PRACE


PRACE-2IP

HPCEUR HET

PRACE History – An Ongoing Success Story

2004 2005 2006 2007 2008

Creation of the Scientific Case

HPC part of the ESFRI Roadmap;

creation of a vision involving 15

European countries

Signature of the MoU Creation of the PRACE Research Infrastructure

PRACE RI

2009 2010 2011 2012

PRACE Initiative

PRACE Preparatory Phase Project

PRACE-1IP PRACE-3IP


• European HPC-facilities at the top of an HPC provisioning pyramid– Tier-0: European Centres for Petaflop/s– Tier-1: National Centres– Tier-2: Regional/University Centres

• Creation of a European HPC ecosystem– HPC service providers on all tiers– Scientific and industrial user communities– The European HPC hard- and software industry– Other e-Infrastructures

Realizing the ESFRI Vision for a HPC RI

6

capa

bilit

y

# of systems


Tier-0European centres

Tier-1National centres

Tier-2Regional/University centres

21 PRACEMembers

April, 23rd 2010 creation of the legal entity (AISBL)

PRACEwith seat location in Brussels, Belgium

68+ Million € from EC FP7 for preparatory and

implementationGrants INFSO-RI-211528,

261557, and 283493Complemented by ~ 50 Million € from PRACE

members

Interest by Denmark,Israel, …

COMMITMENT AND OBJECTIVES


HPC HPC for Industryfor Industry

Guaranteeing in Guaranteeing in Europe the Europe the

independent independent access to HPC-access to HPC-competence for competence for

industryindustry

HPC HPC for Sciencefor Science

Implement the Implement the ESFRI vision of a ESFRI vision of a European HPC-European HPC-

service at the top service at the top of an HPC of an HPC

provisioning provisioning pyramidpyramid

HPC HPC by Vendorsby Vendors

Helping Helping European (hard- European (hard-

& software) & software) vendors to foster vendors to foster their technology their technology

and HPC-and HPC-competencecompetence

Three pillars of the PRACE Mission

9Thomas Eickermann- FZJ20.9.2011

• PRACE goes for a set of machines with successively increasing capability

– 1 PF (2010) + 1.5 PF (2011) + 1 PF (2011) + 3 PF (2012)

+ >3 PF (2013) and will add upgrade steps + 5 PF 2013 …

– Accumulated capability estimate >15 PF in 2013/2014

– PRACE includes 18 Tier-1 systems with accumulated capability

of > 2 PF (building on DEISA / DECI)

• PRACE provides capability support competence centres over several sites

Provision of Capability and Support


• Binding commitments: France, Germany, Italy, Spain

GENCI – GCS – CINECA – BSC

– Each: 100 Mio € over 5 years in terms of cycles

– Contribution accounted as TCO

• Access strictly by peer review at a European level

– Calls: 5/2010, 9/2010, 2/2011 ... incl. DECI call

– Start of provision: 8/2010

Provision of Capacity and Access


High-End HPC Resources grantedYear TFlop/s years granted / to

be granted Provider

2005 6 DEISA

2006 6 DEISA

2007 15 DEISA

2008 24 DEISA

2009 30 DEISA

2010 201 PRACE

2011 450 + 75 PRACE

2012 expect 1600 PRACE

2013 expect 3000 PRACE


PRACE is building the top of the pyramid... First production system available:

1 Petaflop/s IBM BlueGene/P (JUGENE) at GCS (Gauss Centre for Supercomputing) partner FZJ (Forschungszentrum Jülich)

Second production system available: Bull Bullx CURIE at GENCI partner CEA. Full capacity of 1.8 Petaflop/s reached by late 2011.

Third production system available by the end of 2011: 1 Petaflop/s Cray (HERMIT) at GCS partner HLRS (High Performance Computing Center Stuttgart). Upgrade to 4-5 Petaflop/s planned in 2013.

Fourth production system available by mid 2012: 3 Petaflop/s IBM (SuperMUC) at GCS partner LRZ (Leibniz-Rechenzentrum).

Italy and Spain expect to deploy their own Tier-0 systems from 2012.

Italy and Spain expect to deploy their own Tier-0 systems from 2012.

Tier-0

Tier-1

Tier-2

ORGANIZATIONPRACE


Organization

15

Director (CEO)

STRATOS

Scientific Steering

Committee

Council Executive

Committee

Financial Advisory

Committee

TechnicalSteering

Committee

OperationCommittee

UserForum

AccessCommittee

Council Chair:Achim Bachem


Scientific Communities in the Driver Seat

• Scientific Steering Committee Gives opinions on all matters of a

scientific and technical nature Proposes the members of the

Access Committee

• Access committee Gives advice on the scientific use of

the Tier-0 Infrastructure Provides recommendations on the

allocation of PRACE resources based on the Peer Review process

SSC Initial Peers

Richard Kenway (UK, Part. Phys., Chair)E.J. Baerends (Netherland, Chemistry)Kurt Binder (Germany, Cond. Matter)Miquel Coll (Spain, Biology)Filippo Giorgi (Italy, Climate)Olivier Pironneau (France, Math)

SSC Initial Peers

Richard Kenway (UK, Part. Phys., Chair)E.J. Baerends (Netherland, Chemistry)Kurt Binder (Germany, Cond. Matter)Miquel Coll (Spain, Biology)Filippo Giorgi (Italy, Climate)Olivier Pironneau (France, Math)

• Open to all scientific and industrial user communities

• Main communication channel between HPC users and PRACE AISBL

• Interaction with members of the PRACE AISBL• Discussion and issuing recommendations to

PRACE AISBL• Promoting HPC usage• Fostering collaborations between user

communities

PRACE User Forum


PEER REVIEW AND ACCESS PRACE


PRINCIPLES

• Transparency

• Expert assessment

• Confidentiality

• Right to reply

• Prioritisation

• Managing interests

• No parallel assessment with

national organizations

• Ensure fairness to the

science proposed1920.9.2011 Thomas Eickermann- FZJ

• Preparatory access

only technical peer review

prepare for project access, optionally with PRACE support

• Project access

both technical and scientific peer review

12 month allocations, 2 calls per year: watch www.prace-ri.eu

• Programme access (to come)

both technical and scientific peer review

Interested? – Respond to call for EoIs by October 1 !

Types of Access


IMPLEMENTATION PHASE PROJECTS

PRACE


The PRACE Projects

• The preparation and Implementation of the PRACE RI is supported by a series of FP7-funded projects:

• PRACE Preparatory Phase Project (Jan 2008 – June 2010)– EC-funded project to prepare the implementation of the Infrastructure– 20 Mio € Budget / 10 Mio € EC contribution– Partners from 14 Countries– Main Achievements

• Prepared the governance structure, statutes, funding agreements for the PRACE AISBL

• Surveyed the HPC Usage landscape; scaled and ported key applications• Assessed prototypes for near- and mid-term Petascale systems• Started an extensive HPC training programme

• PRACE 1st, 2nd, 3rd Implementation Phase Projects (2010 – …)


PRACE-1IP: Some Facts• Project life time: July 2010 – June 2012• Consortium: 20+1 partners from 20 countries• Budget: 28 Mio €• EC contribution: 20 Mio €• Planned Budget usage:

– Work-force: 2305 PM = 96 FTE for 2 years– Prototype systems: 5 Mio € (2.5 Mio € EC contribution)– Dissemination and Training: 455 k€ (455 k€ EC contribution)

This project receives funding from the European Community’s Seventh Framework Programme (FP7/2007-2013) under grant agreement no. RI-261557


PRACE-1IP Objectives & Activities (1/4)

• Evolution of the Research Infrastructure– Provide advice for the legal, administrative and financial processes of the

AISBL, e.g.– What are requirements and consequences of moving towards an ERIC ?– How will in-kind contributions be valued (application support, training, …) ?

• Dissemination and Training– Provide advanced HPC training and a training portal– Develop PRACE Advanced Training Centres (PATC)– Promote the PRACE brand in international events



• HPC Ecosystem Relations– Integrate PRACE in the European HPC ecosystem– Establish the User Forum– Update the Scientific Case for a European HPC Service– Develop a model for a Tier-1 exchange and access programme:

Distributed European Computing Initiative (DECI)

• Industrial User Relations– Address the specific requirements of European industry– Propose a PRACE Industrial Offer

• Free access for open research (the INCITE model)

• Provision of expertise and support

• Analyse and solve the related legal and financial issues



• Technical Operation and Evolution of the Distributed Infrastructure– Manage and support the distributed systems and diverse architectures

consistently– Define, deploy, and operate a comprehensive set of common services on all Tier-

0 systems to enhance user experience (leverages DEISA work):• AAA, Grid-access via UNICORE, Common user environment, Helpdesk, …

• Enabling Petascale Applications: Efficient Use of Tier-0 Systems– Port, petascale and optimize applications for the Tier-0 systems as a service for

the users of the PRACE RI• Support for PRACE Preparatory Access Projects• Support for important community codes• Provisioning of tools and libraries• Preparation of Best Practice Guides for Tier-0 Enabling and Optimization• Maintenance of the PRACE Benchmark Suite 2620.9.2011


• Support for the Procurement and Commissioning of HPC Service– Maintain the close relation with HPC vendors and advise the PRACE RI on

future procurements– Technology Watch (jointly with WP9)– Monitor developments and procurements throughout the world

• Future Technologies– Assess advanced architectures to foster European developments towards future

multi-Petaflop/s systems• Selection and assessment of a set of hardware prototypes with a focus on energy-

efficiency

• Assessment of new programming environments and tools

– Assume a driving role in the international exascale collaborations such as EESI


PRACE-1IP Achievements in the 1st year

• Provision of analysis and advice to the AISBL Evolution of the funding and usage model Options and business models for an industrial offer Procurement best practices, HPC market watch, centre operation

• Support for PRACE RI and its users Services for operation of the distributed RI defined >10 Applications and community codes scaled Best practice guides for Tier-0 programming available on the Web Extensive HPC training (seasonal schools and workshops)


PRACE-1IP Achievements in the 1st year (cont.)

• Promoted PRACE at major HPC events Booths and BoF-Sessions at SC and ISC Organized the 3rd DEISA PRACE Symposium in Helsinki Organized the 3rd industrial Seminar in Stockholm

• Prototypes for future energy-efficient technologies selected

• Preparation of a cross-national Tier-1 exchange programme Pilot call for DECI (Distributed European Computing Initiative) issued,

technical & scientific review managed (will be handed over to 2IP) Offers peer-reviewed Tier-1 access to 18 systems at 16 partner sites


PRACE-1IP Actions for the 2nd year

• Continuation and completion of the ongoing work !

• Provide decision proposals and technical advice to the AISBL Council Funding and Usage model, industrial offer, procurements, Tier-1 exchange

programme

• Assess the selected prototypes Select additional ones in close coordination with PRACE-2IP

• Continue and extend services to the RI Deployment and operation of services of the distributed RI Application enabling, documentation, PRACE advanced training centres


What’s next ?

• PRACE-2IP Proposal (September 2011 – August 2013)– 35 Mio € Budget / 18 Mio € EC contribution– Partners from 21 Countries

– Will integrate the national “DEISA-type” HPC resources and services– Will provide user community support, technology assessment, training, …

• PRACE-3IP Proposal (mid 2012 – mid 2014 + X)– Partners from >21 countries envisaged– 20 Mio € EC Contribution available in call– Will continue tasks of PRACE-1IP– Will pilot joint pre-commercial procurement and joint ownership– Will expand services to industrial users


COMPUTATIONAL SCIENCE AND ENGINEERING

PRACE


Granted PRACE Projects - Topics

33

Early access call and 1st regular call:


• 1st Regular Call – 58 proposals asked for 2900 Million Core hours– 9 projects granted with 362 Million Core hours

• 2nd Regular Call– 46 proposal asked for 1250 Million Core hours– 17 projects granted with 400 Million Core hours

Granted PRACE Projects – PI Countries

CURIE distribution per country

JUGENE distribution per country


35

Providing fundamental laws for weather and climate models

Objective: Determine the growth-rate law of turbulent atmospheric boundary layers under the influence of wind shear. Such a law plays a vital role in weather and climate models.Expected results: From simulations of developing boundary layers for different wind speeds and atmospheric stability conditions, a generic growth-rate law will be determined.

Resources granted: 35 million core hours on BG/P, Project duration: August 2010 – December 2010Principal Investigator: Prof Dr Harmen Jonker, [email protected], TU Delft, Netherlands


Scope of application: Aerodynamics, aerothermal coupling, combustion, propulsion, aeroacoustics for aero-engines, missiles, launchers, ramjet engines, etc. A software for industrial applications and research.Objective: Code porting to identify bottleneck’s pointsResults: Test of compiler options (35 executables generated), test of tools (HPM, mpi trace, scalasca, …),

routines optimization (6 routines for CPU, 3 for memory), test of environment ‘s variables (DCMF_EAGER, BGLMPIO_COMM, ….), great stability of the machine, good reproducibility of results of measure, good scalability of the code, but in the future:

- Above 8192 cores necessary to adapt the code (memory size : array function of number of core **2, Replace MPI_Allgather by MPI_AlltoAll). - Adapt the pre processor for treat 1 Billion’s cells : in multi-domains approach problem of memory and time for split one mesh in a great number of sub-meshs with a sequential pre–processor.

Resources granted: 1000000 core hours on BG/PProject duration: 3 monthsPrincipal Investigator:Prof. Refloch Alain, ONERA (the French Aerospace Lab )

([email protected], www.onera.fr, www.cedre.onera.fr)

CEDRE: Multi-Physics Platform for Energetics and Propulsion on general unstructured GRIDS


Objective: A better understanding and control of plasma turbulence – a key physics problem on the way to efficient fusion power plantsExpected results:

- Temperature and density profile predictions by coupling thepremier gyrokinetic plasma turbulence code GENE with atransport solver- Full-torus GENE simulations of “Internal Transport Barriers”

Resources granted: 50,000,000 core hours on BGPProject duration: 1st August – 30th NovemberPrincipal Investigator: Prof. Frank Jenko, Max-Planck-Institute for Plasma Physics at Garching, GermanyMore info: http://gene.rzg.mpg.de

Simulations of Turbulence in Fusion Plasmas

37

Full-torus gyrokinetic plasma turbulence

simulation with GENE (TCV tokamak, Lausanne)


• PRACE needs the capability and capacity of all its member countries to achieve its ambitious goals– Hosting member countries provide the financial capacity to

establish and maintain a competitive Tier-0 Infrastructure– Other members operate Tier-1 systems, supporting national

community-building and serving as a stepping stone to the Tier-0 Infrastructure

– All members contribute the intellectual capability and know-how in computational science and computer science needed to support the user communities in harnessing the PRACE Research Infrastructure

• PRACE is instrumental for building the European Research Area

PRACE – A Truly European Endeavour


Example: NCSA in PRACE

• National Center for Supercomputing Applications– Joined the PRACE Initiative (MoU) in September 8, 2009– Is a founding member of the PRACE AISBL, established on April

23, 2010 in Brussels

• NCSA is actively participating in PRACE-1IP:– Further development of the PRACE structure and processes– HPC promotion through user training, support and community

building for academic and industrial users– Enabling applications for petascale systemsNCSA is instrumental for building a strong PRACE user community

in Bulgaria !


Example: NCSA in PRACE cont.

• NCSA is playing an important role in the 2nd Implementation Phase project PRACE-2IP:– Continuation and extension of the tasks in PRACE-1IP– Participation in a Tier-1 exchange programme – to integrate the

national Supercomputer in the PRACE infrastructure and provide a seamless upgrade path for Bulgarian users from Tier-1 to the Top-level PRACE Tier-0 systems (especially on IBM BG/P)

– Prototyping of technologies for future multi-petascale Supercomputers and their programming

In PRACE-2IP NCSA is extending its commitment to and integration in PRACE !


PRACE-1IPThe First Implementation Phase Project

• PRACE-1IP complements the national investments of at least400 Mio € and accelerates the deployment of HPC services

• PRACE-1IP is the first of seriesof 3 IP projects

• The IP projects are instrumentalto continue the PRACE successstory

• PRACE-1IP builds on the competenceand commitment of its 21 partners

• Find out more at http://www.prace-ri.eu4120.9.2011 Thomas Eickermann- FZJ

30.8.2011 42Thomas Lippert - FZJ

30.8.2011 43Thomas Lippert - FZJ

PRACE-1IP – Supporting the Implementation of the PRACE Infrastructure

Documents

Transcript of PRACE-1IP – Supporting the Implementation of the PRACE Infrastructure