Computer Science and Mathematics

Division

Jack C. Wells

Computer Science and Mathematics Division &

Center for Nanophase Materials Science

Oak Ridge National Laboratory

RAMS Workshop

December, 2005

Oak Ridge

2

Fusion Simulation

Focused on grand challengescientific applications

Astrophysics Genomesto Life

Center for NanophaseMaterials

ClimateModeling

ComputationalChemistry

3

Complex

Systems

J. Barhen P. Boyd

R. Bennink3

Y. Y. Braiman R. J. Carter C. W. GloverW. P. GriceT. S. Humble5

N. ImamD. L. JungV. Kireev5

S. M. LenhartH. K. LiuY. LiuL. E. ParkerN. S. V. RaoD. B. Reister7

D. R. Tufano

Q. Wu5

M. Zhu

Network and

Cluster Computing

G. A. Geist

C. Sonewald

P. K. AgarwalN. E. Baldwin5

R. Baumann5

D. E. BernholdtM. L. ChenJ. J. DongarraW. R. ElwasifC. Engelmann

J. Gao5

G. H. Kora5a

J. A. KohlR. Krishnamurthy5a

A. Longo2

X. Ma2

J. L. Mugler5a

T. J. Naughton5a

H. H. Ong

B.-H. Park5

N. F. SamatovaS. L. ScottJ. SchwidderC. T. Symons5a

K. Uhlemann5

G. R. Vallee5

S. VazhkudaiW. R. WingM. Wolf2

S. Yoginath5a

B. Zhang5

Statistics and

Data Science

J. A. Nichols (acting)

L. E. Thurston

R. W. CountsJ. D. Evans1

E. L. FromeB. L. JacksonG. OstrouchovL. C. PouchardD. D. Schmoyer

D. A. Wolf1Student/1aFaculty2Joint Faculty3Wigner Fellow4Householder Fellow5Postdoc/5aPostmaster6Dual Assignment7Part-time

Climate Dynamics

J. B. Drake C. Sonewald

M. L. BranstetterD. J. EricksonX. Guo1a

M. W. HamF. HoffmanJ. L. HernandezG. MahinthakumarP. H. Worley

Computational

Mathematics

E. F. D’Azevedo L. E. Thurston

V. AlexiadesR. K. Archibald4

V. K. Chakravarthy G. I. FannL. J. GrayR. Hartman-Baker5

A. K. KhamaysehS. N. Fata5

S. Pannala

Computational

Materials Science

T. C. Schulthess L. C. Holbrook

N. D. Arnold5

J. A. Alford5

G. Alvarez3

G. A. AramayoR. M. Day5a

Y. Gao2

A. A. GorinI. Jouline2

T. KaplanP. R. Kent5

J. Lu5

T. A. Maier3

D. M. NicholsonP. K. NukalaY. OsetskiyL. Petit5

B. RadhakrishnanG. B. SarmaM. Shabrov5

S. SimunovicX. Tao5

J. C. WellsX-G. ZhangJ. Zhong C. Zhou5

Computational

Chemical

Sciences

R. J. Harrison L. E. Thurston

W. Alvaro2a

J. BernholcA. Beste5

B. Carlen1

S. Dag5

M. L. Drummond5

T. Fridman5

Z. Gan5

B. C. HathornD. Jiang5

A. Kalemos5

V. Meunier

M. B. Nardelli2

D. W. NoidW. A. SheltonS. Sugiki5

B. G. SumpterN. Vence1

Y. Xu5

J. Cobb, Lead,Teragrid

Future

Technologie

s

J. S. Vetter T. S.Darland

S. R. Alam5

R. F. BarrettN. Bhatia5a

J. A. KuehnC. B.

McCurdy5a

O. StoraasliK. J. RocheP. C. Roth

Operations Council

Finance

U. F. Henderson

Human Resources

M. J. Palermo

OrganizationalManagement

N. Y. Wright6

Recruiter

J. K. Johnson6

Technical Informationand Communications

B. A. Riley6

A. D. Harris

Facility 5600/5700

B. A. Riley

Computer Security

T. K. Jones6

ESH/Safety Officer

Ross Toedte6

5600 Labs Manager

N. S. Rao6

CSB Computing CenterManager

M. W. Dobbs6

Quality Assurance

R. W. Counts6

ADVISORY COMMITTEEDavid Keyes – Visiting Distinguished ScientistJack Dongarra – Distinguished ProfessorThom Dunning – Distinguished ProfessorJim Hack – NCAR ScientistThomas Sterling – Visiting DistinguishedScientist

Computer Science and MathematicsJ. A. Nichols, Director

L. M. Wolfe, Division SecretaryN. Fletcher1

B. Hagen1

11/15/05

4

Our Motivation:Opportunities for Breakthrough Science

Two recent examples:

High-TC superconducting materials:

First solution of 2D Hubbard Model

(T. Maier, PRL, accepted 10/2005)

Fusion plasma simulation:

Largest simulation of plasma behavior

in a tokamak

(F. Jaeger, APS-DPP invited

presentation, 10/2005)

5

Advancing scientific discoverythrough computing

• Practical nanotube devices demand airstable, controllable, large-scale doping

• Recent experiments suggest organicdoping is viable

• Simulations confirm experiment andenable detailed understanding & design

• Large-scale quantum conductancecalculations on Cray-X1 and SGI-Altix

• Meunier and Sumpter, Journal ofPhysical Chemistry (2005, accepted)

A New Second Variational BindingEnergy Model

Design of ceramic materials

• Interprets observed properties in termsof chemical bonding

• Control of the microstructureand mechanical properties

• Correctly predicts interface positionsof sintering additives

• Simulations led experiment!

• N. Shibata et al., Nature 428,730–733 (2004)

Design and Control of Nanoscale andMolecular Electronic Devices

Computational Nanoscience

6

Prediction and discovery of giant tunnelingmagnetoresistance (TMR) for spin valves

Magnetoresistanceapplications today:

Magnetic RandomAccess MemoryRecording head

Typical TMR foramorphousaluminumoxide barrier:

Computationalprediction: TMR of1000% is possiblefor crystalline MgObarrier, if interfacesare good enough

By 2004,MgO-basedheterostructureswith >300% TMRdiscoveredexperimentally

•Recording headin computerhard discs

•Magnetic randomaccess memory

• 1995: ~10%(discovery @ MIT)

• 2005: ~70%(after a bigexperimental effort)

•Butler, Zhang,Schulthess, andMacLaren (ORNL),Phys. Rev. B (2001)

• Parkin et al.,Nature Materials(2004)

• Yasa et al., NatureMaterials (2004)

7

2004 2005 2006 200

7

2008 2009

100 TF

1000 TF

250 TF

25 TF

5TF

Cray X1E

IBM BG

NLCF plan for the next 5 years:

Cray XT3

TBD

Cray X1E

IBM Blue Gene

Vector Arch

Global memory

Powerful CPU

Cluster Arch

Low latency

High bandwidth

Scalability

100K CPU

MB/CPU

Cray XT3

18 TF

8

Observations

• Memory-processor performancegap leading to low sustainedperformance on importantapplications

• ITRS predicts that Moore’s Lawwill be coming to an end in2012-2015

• Practical considerations (power,cooling, floor space) are forcingarchitects to change theirdesigns now

Power DensityPower Density

Will Get Even WorseWill Get Even Worse

Need to Keep the Junctions CoolNeed to Keep the Junctions Cool

•• Performance (Higher Frequency)Performance (Higher Frequency)

•• Lower leakage (Exponential) Lower leakage (Exponential)

•• Better reliability (Exponential)Better reliability (Exponential)

Hot PlateHot Plate

Nuclear ReactorNuclear Reactor

Rocket NozzleRocket Nozzle

Sun’s SurfaceSun’s Surface

4004400480088008

8080808080858085

80868086

286286386386

486486

PentiumPentium®®

processorsprocessors

11

1010

100100

1,0001,000

10,00010,000

’70’70 ’80’80 ’90’90 ’00’00 ’10’10

Power DensityPower Density

(W/cm2)(W/cm2)

© 2001 IEEE International Solid-State Circuits Conference © 2001 IEEE© 2001 IEEE International Solid-State Circuits Conference © 2001 IEEE

Source: Pat Gelsinger, CTO, Intel

October 15, 2004

9

How Big Is Big?

• Every 10X brings new challenges

64 processors was once considered large

It hasn’t been “large” for quite a while.

1024 processors is today’s “medium” size

2048-16192 processors is today’s “large”

We are struggling even here.

• 100K processor systems

are being designed/deployed

have fundamental challenges …

… and no integrated research programs

• Petascale data archives

the “personal petabyte” very near

• See recent PITAC report

www.nitrd.gov

0

5

10

15

20

25

30

128 256 512 1024 2048 4096 8192 16384 More

Count

Top 500 Size

Distribution

Preparing for Big:Math and CS challenges

• Theoretical Models (existing)

May not perform well on petascale computers

May not have needed fidelity

May be inadequate to describe new phenomena revealed by experiment or

simulation

• Scientific Modeling and Simulation Codes (existing)

Do not take advantage of new architectures (5%-10% of peak)

New computing capabilities lead to new simulation possibilities and, thus, new

applications codes

• Systems Software

Vendor operating systems do not provide needed functionality

Systems software for petascale applications non-existent

• Software to manage and visualize massive (petabyte) data sets

• Software to accelerate development and use of petascale scientific applications

• Techniques for porting software to the next generation inadequate

Few mathematical algorithms scale to thousand-million processors

11

People and partnerships

Thomas Maier Wigner Fellow

Jennifer RyanHouseholder Fellow

Gonzalo AlvarezWigner Fellow

We have methodically hired122 research staff in 5 yearsin support of computing

Elbio Dagotto Distinguished Scientist

Jeff VetterFuture Technologies

43464134Students

30564058Postdocs

23312115S&T

2005200420032002

• Strong partnerships with sister labs(ANL, LBNL, PNNL) and other centers

• Interagency partnerships with NSF, NSA, NNSA, NASA, DHS

– Key resource for Intergovernmental Panel on Climate Change (IPCC)

• Joint Institute for Computational Sciences/core universities

– Graduate program in computational Sciences

– Distinguished Scientists/ Joint faculty appointments

• Cray Center of Excellence

12

World leaderin scientific computing

Intellectual center incomputational science

Transform scientificdiscovery through

advanced computing

“User facility providingleadership-class

computing capability toscientists and engineersnationwide independent

of their institutionalaffiliation or source

of funding”

Create an interdisciplinaryenvironment where

science and technologyleaders converge

to offer solutions totomorrow’s challenges

“Deliver major researchbreakthroughs,

significant technologicalinnovations, medicaland health advances,enhanced economic

competitiveness, andimproved quality of life

for the American people”

Our Aspirations