Computational Science at Edinburgh

From Excellence to Enterprise

Dr Arthur Trew

Director

on the shoulders of giants

for centuries science has relied on experiment and theory, but– theory is best suited to a reductionist approach– experiment is limited in its range of applicability

there is a need for a complementary approach: simulation study of emergent phenomena

Theory-Greece400 BC

Experiment -Italy

1,500 AD

For problems which are:- too large/small- too fast/slow- too complex- too expensive ...

Simulation -Edinburgh1,980 AD

becoming the best

in 1990 Edinburgh established EPCC as its focus for HPC simulation– following a decade of research in Physics

in 1992 EPCC set itself a mission “to be Europe’s top HPC centre within 5 years”– we succeeded– today, that success is benefiting Scottish research and industry

in 2001 Edinburgh & Glasgow created NeSC to extend that vision– over the past 18 months EPCC & NeSC have brought in £65M of

research grants/contracts • of which £2.3M from SHEFC for eDIKT

– Edinburgh & Glasgow backed this with £3M– … and foundation departments lead major UK Grid projects

arthur:

52.86 HPCx

1.0 Donofrio

.4 Eades

1.68 SRIF

2.3 eDIKT

1.0 SUR

2.3 NeSC/eSI

3 GCP

0.2 GSC

0.35 e-Strom

0.6 GridNet

arthur:

52.86 HPCx

1.0 Donofrio

.4 Eades

1.68 SRIF

2.3 eDIKT

1.0 SUR

2.3 NeSC/eSI

3 GCP

0.2 GSC

0.35 e-Strom

0.6 GridNet

arthur:

UoE = 3.19 - 1.68 UoG = 1.45

(AK’s figures)

arthur:

UoE = 3.19 - 1.68 UoG = 1.45

(AK’s figures)

what’s that to me? as computer performance improves, the range of applications

increases

0

10

20

30

40

50

60

70

80

90

100

0 1 2 3 4 5 6 7 8

year

Tfl

op

s c

ap

ab

ilit

y

whole earthclimate

organs

solar weather

materials design

cells

whole aircraft

drug designprotein structures

nanostructures

astroplasmaseddy resolution

oceans

HPCx

£53M: 3 machines

linking to data

… but we have the web, don’t we?

but simulation is only part of the story …

linking computer modelling to experiment is vital to the interpretation of many large experiments, eg LHC

… and with big experiments set to generate Pbyte/year data management is critical

EPCC overview

Academic:o National HPC facilities o Researcho Support

Training:o Academiao Industry

European leadership:o Visitor programmeso Technology Transfero Strategic Planning

Industry:o Projectso Consultancyo Standards

Technology Transfer

70 staff, £3M turnover

… possible through the “win-win-win” model

HPCx rationale

UK academic research is increasingly dependent upon high-end compute facilities

recent technological and Grid advances highlighted the need to upgrade UK resources

HPCx objectives are thus:– aim “to deliver optimum service resulting in world-

leading science”– address “the problems involved in scaling to the

capability levels required”

What is HPCx?

Consortium of leading UK organisations committed to creating and managing the new resource for the next 6 years– led by University of Edinburgh

multi-stage project to deliver a world-class academic computing resource, the largest in Europe

£54M/$100M budget Grid-enabled, a key component in the UK e-Science

programme

HPCx Consortium Members

Daresbury Laboratory, CCLRC

IBM

EPCC, University of Edinburgh

UoE HPCx Ltd – wholly-owned subsidiary of the University of Edinburgh and lead contractor

the University of Edinburgh is one of the top 5 research universities in the UK

EPCC is the leading computer centre in Europe, bridging the gap between academia and industry

… and provides both HPC and novel computing solutions to a wide range of problems and users

long experience of providing national HPC services including:– Meiko Computing Surfaces– Thinking Machines CM200– Cray T3D/T3E

Daresbury Laboratory

A multi disciplinary research lab with over 500 people

Provides large-scale research facilities both for UK academic and industrial research communities

Runs the UK’s Collaborative Computational Projects

Daresbury hosts the HPCx hardware

IBM provides the technology for HPCx Long standing involvement in HPC including the

development of a number of ASCI machines and 4 of the top dozen machines in the 21st TOP500 list:– ASCI White: Rmax = 7.3 TFlop/s– SP Power3 (6656 Processors): Rmax = 7.3 TFlop/s– xSeries (1920 Processors): Rmax = 6.6 TFlop/s– HPCx (1280 processors): Rmax = 3.2 TFlop/s

IBM has the long term technology road map essential to a 6 year project such as HPCx

HPCx in Place

HPCx: Phase 1

System will be commissioned in three main stages Phase 1 covering 2002-2004 consists of:– 40 Regatta-H SMP nodes, 1280 processors– Peak performance 6.6 TFlop/s, with 3.4 Tflop/s sustained

Linpack• currently 16th in the Top500

– 1.28 TB total memory capacity– Over 50 TB of storage capacity– Double plane Colony switch with total

peak bandwidth of 250 MB/s per processor

The e-Science Grid

CeSC (Cambridge)

HPCx Phases 2 & 3

Phase 2 (2004-2006)– aiming for 6 TFlop/s sustained on Linpack and 2.5 TFlop/s on

sPPM– O(48) Regatta-H+ SMP nodes– interconnect upgraded to Federation switch– doubling of I/O and storage

• already built a cluster with 8 Regatta-H+ frames and a pre-release Federation switch

• undertaking a phased upgrade during 1H2004 Phase 3 (2006-2008)

– target of 12 TFlop/s sustained on Linpack– may be additional nodes or alternative technology

HPCx Science Support

Outreach

Life sciences

New applications

Applications Support

Helpdesk Training

Liaising with users

Users

Technology

Software EngineeringUnderpinning technology Grid/e-Science

Systems & NetworkingFlexible and responsive capability computing service

Smooth transitions between phases

Terascaling Capability applications

Scalable algorithms Performance optimisation

18 staff in 5 dual-centre functional support teams

HPCx Status: Usage

Utilisation of Capability Region

0%

20%

40%

60%

80%

100%

Jan

-03

Fe

b-0

3

Ma

r-03

Ap

r-03

Ma

y-03

Jun

-03

Jul-0

3

Au

g-0

3

Se

p-0

3

Oct-0

3

No

v-03

De

c-03

CPU Usage by Job Size

0

500000

1000000

1500000

2000000

Jan

-03

Fe

b-0

3

Ma

r-03

Ap

r-03

Ma

y-03

Jun

-03

Jul-0

3

Au

g-0

3

Se

p-0

3

Oct-0

3

No

v-03

De

c-03

AU

s

>1024

1024

512

256

128

64

32

16

8

currently 28 active consortia, and over 450 users

Life Sciences outreach activity supported by IBM

HPCx Status: Application Areas

Usage by Application Area

Chemistry

Engineering

Environmental Science

Materials

Science Support

Atomic & Molecular Physics

Atomic and Molecular Physics The UK Multiphoton, Electron Collision and Bose Einstein

Condensates (MECBEC) HPC Consortium Two flagship projects model two-electron atoms (helium)

and molecules (hydrogen) exposed to intense, ultra-short laser pulses

Modelling involves the grid solution of multi-dimensional, time-dependent partial differential equations)

Visualisation techniques crucial in extracting information

Simulation of Double Ionization of laser-driven helium performed at Queen’s University Belfast

Requires large amounts of computing power

Environmental Science: POLCOMS

POLCOMS is a multi-disciplinary model developed at the Proudman Oceanographic Laboratory

… a 3-D hydrodynamic model integrating coasts and oceans using a wide range of associated models

A simulation of Chlorophyll density in UK

waters using the POLCOMS model

POLCOMS is a step towards real time modelling of coastal zones, enabling better analysis of impacts to, and sustainability of, the marine environment

Material Science: Crystal

CRYSTAL computes electronic structure and related properties of periodic systems

Developed jointly by Daresbury and the University of Turin A Fortran90 and MPI program that performs Hartree-Fock, density

functional and other approximation calculations On HPCx, CRYSTAL used to calculate the structure of the Crambin

molecule, the largest Hartree-Fock calculation ever converged (1284 atoms)– next, the Rusticyanin molecule (6284 atoms)

Engineering: UKTC UK Turbulence Consortium is developing world

leading turbulence simulation codes using HPCx essential that they can easily facilitate the

scientific usage of the resulting data the calculated data are transferred from HPCx to a

remote site for analysis so, the Grid is becoming increasingly important

Chemistry: RealityGrid RealityGrid is a UK collaboration

– which aims to grid-enable the realistic modelling and simulation of complex condensed matter structures at the meso and nanoscale levels

HPCx terascaling team has worked on parallelisation & optmisation of RealityGrid codes, such as LB3D

RealityGrid also uses the Grid for data transfer, computational steering, and remote visualisation– using Globus on HPCx– TeraGyroid won award at SC2003

The Vision

we want Edinburgh to lead the e-science revolution– to become the European equivalent of one of the big US centres, eg

San Diego Supercomputer Center