The Cascade High Productivity Programming Language€¦ · Current parallel programming language,...

The Cascade High Productivity Programming Language

Hans P. Zima

University of Vienna, Austriaand

JPL, California Institute of Technology, Pasadena, CA

ECMWF Workshop on the Use of High Performance Computing in MeteorologyReading, UK, October 25, 2004

1 Introduction

2 Programming Models for High Productivity Computing Systems

3 Cascade and the Chapel Language Design

4 Programming Environments

5 Conclusion

11 IntroductionIntroduction

22 Programming Models for High Productivity Programming Models for High Productivity Computing SystemsComputing Systems

33 Cascade and the Chapel Language DesignCascade and the Chapel Language Design

44 Programming EnvironmentsProgramming Environments

55 ConclusionConclusion

Contents

Programming models and languages bridge the gap between

“reality” and hardware – at different levels of abstraction - e.g.,

�assembly languages

�general-purpose procedural languages

�functional languages

�very high-level domain-specific languages

Programming models and languages bridge the gap between Programming models and languages bridge the gap between

““realityreality”” and hardware and hardware –– at different levels of abstraction at different levels of abstraction -- e.g.,e.g.,

�assembly languages

�general-purpose procedural languages

�functional languages

�very high-level domain-specific languages

Abstraction in Programming

Abstraction implies loss of information –gain in simplicity, clarity, verifiability, portability

versus potential performance degradation

The designers of the very first high level programming language were aware that their success depended on acceptable performance of the generated target programs:

John Backus (1957): “… It was our belief that if FORTRAN … were to translate any reasonable scientific source program into an object program only half as fast as its hand-coded counterpart, then acceptance of our system would be in serious danger …”

The designers of the very first high level programming languThe designers of the very first high level programming language age were aware that their success depended on acceptable were aware that their success depended on acceptable performance of the generated target programs:performance of the generated target programs:

John Backus (1957): “… It was our belief that if FORTRAN … were to translate any reasonable scientific source program into an object program only half as fast as its hand-coded counterpart, then acceptance of our system would be in serious danger …”

The Emergence of High-Level Sequential Languages

High-level algorithmic languages became generally accepted standards for sequential programming since

their advantages outweighed any performance drawbacks

For parallel programmingno similar development took place

� Current HPC hardware: large clusters at reasonable cost�commodity clusters or custom MPPs

�off-the-shelf processors and memory components, built for mass market

�latency and bandwidth problems

� Current HPC software �application efficiency sometimes in single digits

�low-productivity “local view” programming models dominate

� explicit processors: local views of data

� program state associated with memory regions

� explicit communication intertwined with the algorithm

� wide gap between domain of scientist and programming language

�inadequate programming environments and tools

�higher level approaches (e.g., HPF) did not succeed, for a variety of reasons

�� Current HPC hardware: large clusters at reasonable costCurrent HPC hardware: large clusters at reasonable cost�commodity clusters or custom MPPs

�off-the-shelf processors and memory components, built for mass market

�latency and bandwidth problems

�� Current HPC software Current HPC software �application efficiency sometimes in single digits

�low-productivity “local view” programming models dominate

� explicit processors: local views of data

� program state associated with memory regions

� explicit communication intertwined with the algorithm

� wide gap between domain of scientist and programming language

�inadequate programming environments and tools

�higher level approaches (e.g., HPF) did not succeed, for a variety of reasons

The Crisis of High Performance Computing

Current parallel programming language, compiler, and tool technologies are unable to support high productivity computing

�

New programming models, languages, compiler, and tool technologies are necessary to address the productivity demands of future systems

Current parallel programming language, compiler, and tool Current parallel programming language, compiler, and tool technologies are unable to support high productivity computingtechnologies are unable to support high productivity computing

��

New programming models, languages, New programming models, languages, compiler, and tool technologies are compiler, and tool technologies are necessary to address the productivity necessary to address the productivity demands of future systemsdemands of future systems

State-of-the-Art

� Make Scientists and Engineers more productive:

provide a higher level of abstraction

� Support “Abstraction without Guilt” [Ken Kennedy]:

increase programming language usability without

sacrificing performance

�� Make Scientists and Engineers more productive:Make Scientists and Engineers more productive:

provide a higher level of abstractionprovide a higher level of abstraction

�� Support Support ““Abstraction without GuiltAbstraction without Guilt”” [Ken Kennedy]:[Ken Kennedy]:

increase programming language usability without increase programming language usability without

sacrificing performasacrificing performancence

Goals

1 Introduction




5 Conclusion






Contents

� Large scale architectural parallelism�hundreds of thousands of processors

�component failures may occur in relatively short intervals

� Extreme non uniformity in data access

� Applications are becoming larger and more complex�multi-disciplinary, multi-language, multi-paradigm

�dynamic, irregular, and adaptive

� Legacy codes pose a problem: long lived applications, surviving many generations of hardware�from F77 to F90, C/C++, MPI, Coarray Fortran etc.

�automatic re-write under constraint of performance portability is difficult

�� Large scale architectural parallelismLarge scale architectural parallelism�hundreds of thousands of processors

�component failures may occur in relatively short intervals

�� Extreme non uniformity in data access Extreme non uniformity in data access

�� Applications are becoming larger and more complexApplications are becoming larger and more complex�multi-disciplinary, multi-language, multi-paradigm

�dynamic, irregular, and adaptive

�� Legacy codes pose a problem: long lived applications, surviving Legacy codes pose a problem: long lived applications, surviving many generations of hardwaremany generations of hardware�from F77 to F90, C/C++, MPI, Coarray Fortran etc.

�automatic re-write under constraint of performance portability is difficult

Productivity Challenges of Peta-Scale Systems

Performance, user productivity, robustness, portability

Semantics

ProductivityModel

Programming Model

conceptual view of data and control

ProgrammingLanguage

ProgrammingLanguage

+ Directives

Library

Command-lineInterface

SP

SP

SP

SP

ExecutionModel

abstract machine

realizations

Programming Models for High Productivity Computing

� Programming models for high productivity computing and their realizations can be characterized along (at least) three dimensions:� semantics

� user productivity (time to solution)

� performance

� Semantics: a mapping from programs to functions specifying input/output behavior of the program:�S: P � F , where each f in F is a function f: I � O

� User productivity (programmability): a mapping from programs to a characterization of structural complexity:�U: P � N

� Performance: a mapping from programs to functions specifying the complexityof the program in terms of its execution on a real or abstract target machine: �C: P � G, where each g in G is a function g : I � N*

�� Programming models for high productivity computing and their reaProgramming models for high productivity computing and their realizations lizations can be characterized along (at least) three dimensions:can be characterized along (at least) three dimensions:� semantics

� user productivity (time to solution)

� performance

�� SemanticsSemantics: a mapping from programs to functions specifying input/output : a mapping from programs to functions specifying input/output behavior of the program:behavior of the program:�S: P � F , where each f in F is a function f: I � O

�� User productivity (programmability)User productivity (programmability): a mapping from programs to a : a mapping from programs to a characterization of structural complexity:characterization of structural complexity:�U: P � N

�� PerformancePerformance: a mapping from programs to functions specifying the complexity: a mapping from programs to functions specifying the complexityof the program in terms of its execution on a real or abstract tof the program in terms of its execution on a real or abstract target machine: arget machine: �C: P � G, where each g in G is a function g : I � N*

Programming Model Issues

1 Introduction




5 Conclusion






Contents

High ProductivityComputing Systems

Goals:Goals:

� Provide a new generation of economically viable high productivity computing systems for the national security and industrial user community (2007 – 2010)

Impact:� Performance (efficiency): critical national security

applications by a factor of 10X to 40X� Productivity (time-to-solution) � Portability (transparency): insulate research and

operational application software from system� Robustness (reliability): apply all known techniques

to protect against outside attacks, hardware faults, & programming errors

Fill the Critical Technology and Capability GapToday (late 80’s HPC technology)…..to…..Future (Quantum/Bio Computing)

Fill the Critical Technology and Capability GapToday (late 80’s HPC technology)…..to…..Future (Quantum/Bio Computing)

Applications:� Intelligence/surveillance, reconnaissance, cryptanalysis, airborne contaminant modeling and

biotechnology

HPCS Program Focus Areas

� One year Concept Study, July 2002 -- June 2003

� Three year Prototyping Phase, July 2003 -- June 2006

� Led by Cray Inc. (Burton Smith)

�Partners�Caltech/JPL

�University of Notre Dame

�Stanford University

�Collaborators in the Programming Environment Area

David Callahan, Brad Chamberlain, Mark James, John Plevyak

�� One year Concept Study, July 2002 One year Concept Study, July 2002 ---- June 2003June 2003

�� Three year Prototyping Phase, July 2003 Three year Prototyping Phase, July 2003 ---- June 2006 June 2006

�� Led by Cray Inc. (Burton Smith)Led by Cray Inc. (Burton Smith)

��PartnersPartners�Caltech/JPL

�University of Notre Dame

�Stanford University

��Collaborators in the Programming EnvironmentCollaborators in the Programming Environment AreaArea

David Callahan, Brad Chamberlain, Mark James, John David Callahan, Brad Chamberlain, Mark James, John PlevyakPlevyak

The Cascade Project

� High performance networks and multithreading contribute to tolerating memory latency and improving memory bandwidth

� Hardware support for locality aware programming and program-controlled selection of UMA/NUMA data access avoid serious performance problems present in current architectures

� Shared address space without global cache coherence eliminates a major source of bottlenecks

� Hierarchical two-level processing structure exploits temporal as well as spatial locality

� Lightweight processors in “smart memory” provide a computational fabric as well as an introspection infrastructure

�� High performance networksHigh performance networks and multithreading contribute to and multithreading contribute to tolerating memory latency and improving memory bandwidthtolerating memory latency and improving memory bandwidth

�� Hardware support for Hardware support for locality aware programminglocality aware programming and and programprogram--controlled selection of UMA/NUMA data access avoid controlled selection of UMA/NUMA data access avoid serious performance problems present in current architecturesserious performance problems present in current architectures

�� Shared address spaceShared address space without global cache coherence eliminates without global cache coherence eliminates a major source of bottlenecksa major source of bottlenecks

�� Hierarchical twoHierarchical two--level processing structurelevel processing structure exploits temporal as exploits temporal as well as spatial localitywell as spatial locality

�� Lightweight processorsLightweight processors in in ““smart memorysmart memory”” provide a provide a computational fabric as well as an introspection infrastructurecomputational fabric as well as an introspection infrastructure

Key Elements of the Cascade Architecture

A Simplified Global View of the Cascade Architecture

Smart Memory

HWP

SW-controlled Cache

Smart Memory

HWP

SW-controlled Cache…

Global Interconnection Network

Locale Locale

Heavyweight ProcessorVector

MultithreadedStreaming

Compiler-assisted cache

LocaleInterconnect

NetworkRouter

To Other Locales

DRAM

Lightweight Processors

Multithreaded

cache

Lightweight Processor Chip

Multithreaded

LWP LWP LWP LWP…

DRAM


Multithreaded

cache


Multithreaded

LWP LWP LWP LWP…

DRAM


Multithreaded

cache


Multithreaded

LWP LWP LWP LWP…

DRAM


Multithreaded

cache


Multithreaded

LWP LWP LWP LWP…

DRAM


Multithreaded

cache


Multithreaded

LWP LWP LWP LWP…

DRAM


Multithreaded

cache


Multithreaded

LWP LWP LWP LWP…

A Cascade Locale

Source: David Callahan, Cray Inc.

�Lightweight processors�co-located with memory

�focus on availability

�full exploitation is not a primary system goal

�Lightweight threads�minimal state – high rate context switch

�spawned by sending a parcel to memory

�Exploiting spatial locality�fine-grain: reductions, prefix operations, search

�coarse-grain: data distribution and alignment

�Saving bandwidth by migrating threads to data

��Lightweight processorsLightweight processors�co-located with memory

�focus on availability

�full exploitation is not a primary system goal

��Lightweight threadsLightweight threads�minimal state – high rate context switch

�spawned by sending a parcel to memory

��Exploiting spatial localityExploiting spatial locality�fine-grain: reductions, prefix operations, search

�coarse-grain: data distribution and alignment

��Saving bandwidth by migrating threads to dataSaving bandwidth by migrating threads to data

Lightweight Processors and Threads

Key Issues in High Productivity Languages

High-Level Features for Explicit Concurrency

High-Level Features for Locality Control

High-Level Support for Distributed Collections

global address space

Safety

object orientation

performance transparency

generic programming

Extensibility

Critical Functionality Orthogonal Language Issues

High-Level Support for Programming-In-the-Large

data work

data

align

workData/thread affinity

distribute

HPCS architecture

PM

PM

PM

PM

PM

PM

PM

PM

PM

PM

PM

Locality Awareness: Distribution, Alignment, Affinity

distribute

Issues

• Model partitioning of physical domains • Support dynamic reshaping • Express data/thread affinity• Support user-defined distributions

�Global name space�even in the context of a NUMA model

�avoid “local view” programming model of MPI, Coarray Fortran, UPC

�Multiple models of parallelism

�Provide support for:�explicit parallel programming

�locality-aware programming

�interoperability with legacy codes (MPI, Coarray Fortran, UPC, etc.)

�generic programming

��Global name spaceGlobal name space�even in the context of a NUMA model

�avoid “local view” programming model of MPI, Coarray Fortran, UPC

��Multiple models of parallelismMultiple models of parallelism

��Provide support forProvide support for::�explicit parallel programming

�locality-aware programming

�interoperability with legacy codes (MPI, Coarray Fortran, UPC, etc.)

�generic programming

Design Criteria of the “Chapel” Programming Language

� A modern base language�Strongly typed

�Fortran-like array features

�Objected-oriented

�Module structure for name space management

�Optional automatic storage management

� High performance features�Abstractions for parallelism

� data parallelism (domains, forall)

� task parallelism (cobegin)

�Locality management via data distributions and affinity

�� A modern base languageA modern base language�Strongly typed

�Fortran-like array features

�Objected-oriented

�Module structure for name space management

�Optional automatic storage management

�� High performance featuresHigh performance features�Abstractions for parallelism

� data parallelism (domains, forall)

� task parallelism (cobegin)

�Locality management via data distributions and affinity

Chapel Basics

Type Tree

primitive domain collection function

homogeneouscollection

heterogeneouscollection

sequence setarray class record

iterator other function

�The “Concrete Language” enhances HPF and ZPL�domains as first-class objects: index space, distribution, and associated set of arrays

�generalized arrays and HPF-type data distributions

� support for automatic partitioning of dynamic graph-based data structures

�high-level control for communication (halos,…)

�abstraction of iteration: generalization of the CLU iterator

�The “Abstract Language” supports generic programming�abstraction of types: type inference from context

�data structure inference: system-selected implementation for programmer-specified object categories

� specialization using cloning

��The The ““Concrete LanguageConcrete Language”” enhances HPF and ZPLenhances HPF and ZPL�domains as first-class objects: index space, distribution, and associated set of arrays

�generalized arrays and HPF-type data distributions

� support for automatic partitioning of dynamic graph-based data structures

�high-level control for communication (halos,…)

�abstraction of iteration: generalization of the CLU iterator

��TheThe ““Abstract LanguageAbstract Language”” supports generic programmingsupports generic programming�abstraction of types: type inference from context

�data structure inference: system-selected implementation for programmer-specified object categories

� specialization using cloning

Language Design Highlights

� locale view: a logical view for a set of locales

� distribution: a mapping of an index set to a locale view

� array: a map from an index set to a collection of variables

Domains

� index sets: Cartesian products, sparse, opaque

Source: Brad Chamberlain, Cray Inc.

2) Separation of Concerns: Definition

var Mat: domain(2) = [1..m, 1..n];var MatCol: domain(1) = Mat(2);var MatRow: domain(1) = Mat(1);

var A: array [Mat] of float;var v: array [MatCol] of float;var s: array [MatRow] of float;

s = sum(dim=2) [i,j:Mat] A(i,j)*v(j);

Example: Matrix Vector Multiplication V1

var L: array[1..p1,1..p2] of locale;

var Mat: domain(2) dist(block,block) to L = [1..m,1..n];var MatCol: domain(1) align(*,Mat(2)) = Mat(2);var MatRow: domain(1) align(Mat(1),*) = Mat(1);




0 53 0 0 0 0 0 0 0 0 19 0 0 0 0 0 0 0 0 0 0 0 0 17 0 0 0 0 0 930 0 0 0 0

0 0 0 0 21 00 0 1672 0 00 0 00 0 00 13 0

0 0 0 0 0 23 69 0 27 0 0 11

44 0 0 1937 0 0 00 0 64 0

D0____

53191793

C0____

2145

R0____

12233455

D0____

53191793

D0____

53191793

C0____

2145

R0____

12233455

C1____

2312

R1____

11234445

D1____

21167213

D0____

53191793

C2____

2314

R2____

1135

D2____

23692711

D3____

44193764

C3____

1413

R3____

1345

Sparse Matrix Distribution

var L: array[1..p1,1..p2] of locale;

var Mat: domain(sparse2) dist(myspdist) to Llayout(mysplay) = [1..m,1..n]where enumeratenonzeroes();

var MatCol: domain(1) align(*,Mat(2)) = Mat(2);var MatRow: domain(1) align(Mat(1),*) = Mat(1);




� Global name space

� High level control features supporting explicit parallelism

� High level locality management

� High level support for collections

� Static typing

� Support for generic programming

�� Global name spaceGlobal name space

�� High level control features supporting explicit parallelism High level control features supporting explicit parallelism

�� High level locality managementHigh level locality management

�� High level support for collectionsHigh level support for collections

�� Static typingStatic typing

�� Support for generic programmingSupport for generic programming

Language Summary

1 Introduction




5 Conclusion






Contents

�Reliability Challenges�massive parallelism poses new problems

�fault prognostics, detection, recovery

�data distribution may cause vital data to be spread across all nodes

� (Semi) Automatic Tuning�closed loop adaptive control: measurement, decision-making, actuation

�information exposure: users, compilers, runtime systems

�learning from experience: databases, data mining, reasoning systems

� Introspection�a technology for support of validation, fault detection, performance tuning

��Reliability ChallengesReliability Challenges�massive parallelism poses new problems

�fault prognostics, detection, recovery

�data distribution may cause vital data to be spread across all nodes

�� (Semi) Automatic Tuning(Semi) Automatic Tuning�closed loop adaptive control: measurement, decision-making, actuation

�information exposure: users, compilers, runtime systems

�learning from experience: databases, data mining, reasoning systems

�� IntrospectionIntrospection�a technology for support of validation, fault detection, performance tuning

Issues in Programming Environments and Tools

Parallelizingcompiler

sourceprogram

targetprogram

Expertadvisor

Transformation system

execution ontarget machine

end of tuning cycle

Program / ExecutionKnowledge Base

Example: Offline Performance Tuning

�Rewriting Legacy Codes�preservation of intellectual content

�opportunity for exploiting new hardware, including new algorithms

�code size may preclude practicality of rewrite

�Language, compiler, tool, and runtime support�(Semi) automatic tools for migrating code

�incremental porting

�transition of performance-critical sections requires highly-sophisticated software for automatic adaptation

� high-level analysis

� pattern matching and concept comprehension

� optimization and specialization

��Rewriting Legacy CodesRewriting Legacy Codes�preservation of intellectual content

�opportunity for exploiting new hardware, including new algorithms

�code size may preclude practicality of rewrite

��Language, compiler, tool, and runtime supportLanguage, compiler, tool, and runtime support�(Semi) automatic tools for migrating code

�incremental porting

�transition of performance-critical sections requires highly-sophisticated software for automatic adaptation

� high-level analysis

� pattern matching and concept comprehension

� optimization and specialization

Legacy Code Migration

� Fine grain application parallelism

� Implementation of a service layer

� Components of agent systems that asynchronously monitor the computation performing introspection, and dealing with:�dynamic program validation

� fault tolerance

� intrusion prevention and detection

�performance analysis and tuning

� support of feedback-oriented compilation

� Introspection can be defined as a system’s ability to:� explore its own properties

� reason about its internal state

�make decisions about appropriate state changes where necessary

�� Fine grain Fine grain application parallelismapplication parallelism

�� Implementation of aImplementation of a service layerservice layer

�� Components of agent systems that asynchronously Components of agent systems that asynchronously monitor the monitor the computationcomputation performing introspectionperforming introspection,, and dealing with:and dealing with:�dynamic program validation

� fault tolerance

� intrusion prevention and detection

�performance analysis and tuning

� support of feedback-oriented compilation

�� IntrospectionIntrospection can be defined as a systemcan be defined as a system’’s ability to:s ability to:� explore its own properties

� reason about its internal state

�make decisions about appropriate state changes where necessary

Potential Uses of Lightweight Threads

Example: A Society of Agents for Performance Analysis and Feedback-Oriented Tuning

Program/Performance

Data Baseexecution of instrumented

application program

compiler

instrumenter

data collection

analysisagent

invariantchecking

agent

simplificationagent

Performance Exception Handler

data reduction and filtering

� Today’s programming languages, models, and tools cannot deal with 2010 architectures and application requirements

� Peta scale architectures will pose new challenges but may provide enhanced support for high level languages and compilers

� The Cascade programming language “Chapel” targets the creation of a viable language system together with a programming environment for economically feasible and robust high productivity computing of the future

D.Callahan, B.Chamberlain, H.P.Zima: The Cascade High Productivity Language

Proceedings of the HIPS2004 Workshop, Santa Fe, New Mexico, April 2004

�� TodayToday’’s programming languages, models, and tools cannot deal s programming languages, models, and tools cannot deal with 2010 architectures and application requirementswith 2010 architectures and application requirements

�� PetaPeta scale architectures will pose new challenges but may scale architectures will pose new challenges but may provide enhanced support for high level languages and compilersprovide enhanced support for high level languages and compilers

�� The Cascade programming language The Cascade programming language ““ChapelChapel”” targets the targets the creation of a viable language system together with a creation of a viable language system together with a programming environment for economically feasible and robust programming environment for economically feasible and robust high productivity computing of the future high productivity computing of the future

D.Callahan, B.Chamberlain, H.P.Zima: The Cascade High ProductiviD.Callahan, B.Chamberlain, H.P.Zima: The Cascade High Productivity Languagety Language

Proceedings of the HIPS2004 Workshop, Santa Fe, New Mexico, AprProceedings of the HIPS2004 Workshop, Santa Fe, New Mexico, April 2004il 2004

Conclusion

The Cascade High Productivity Programming Language€¦ · Current parallel programming language,...

Documents

Transcript of The Cascade High Productivity Programming Language€¦ · Current parallel programming language,...