Denis Caromel 1

Denis CaromelInstitut Universitaire de France (IUF)

OASIS TeamINRIA -- CNRS - I3S -- Univ. of Nice Sophia-Antipolis

ECCOMAS, July 2004

ProActive:Components for Numerical GRIDs

1. The GridS

2. Asynchronous Distributed Objects

3. Hierarchical Components

4. Deployment and Environments

5. Application example: Electromagnetism

6. Conclusion, next step: Real Peer-to-Peer (P2P)

Denis Caromel 2

GRIDs

Denis Caromel 3

Enterprise Grids

Internet

EJBServletsApache Databases

Denis Caromel 4

Scientific Grids

Internet

Clusters

ParallelMachine

LargeEquipment

Denis Caromel 5

Internet Grids

Internet

Job management forembarrassingly parallel application (e.g. SETI)

Denis Caromel 6

The multiple GRIDs

• Scientific Grids

• Enterprise Grids

• Intranet and Internet Grids

Strong convergence in process!At least at the infrastructure level, i.e. WS

Define adequate,sound,programming modelsChallenges:

Composition and coupling of codes

Denis Caromel 7

Programming

ProActive: an integrated solution for the GRID

Composing

Deploying

W r a p p i n g

Denis Caromel 8

Distributed Objects

ProActiveProgramming

Denis Caromel 9

Model:• Remote Mobile Objects, Group Communications• Asynchronous Communications with synchro: automatic Futures • OO SPMD, Migration, Non Functional Exceptions (NFE)Environment:• XML Deployment, dynamic class-loading, PKI security• Various protocols: rsh,ssh,LSF,Globus,BPS, ...• Graphical Visualization and monitoring: IC2D

ProActive: A Java API + Tools for the GRIDParallel, Distributed, Mobile, Activities, across the world !

SMP ClustersLANDesktop

Denis Caromel 10

A

Creating AO and Groups

Typed Group Java or Active Object

A ag = newActiveGroup (“A”, […], VirtualNode)V v = ag.foo(param);...v.bar(); //Wait-by-necessity

V

Group, Type, and Asynchrony

are crucial for Cpt. and GRID

JVM

Denis Caromel 11

OO SPMD A ag = newSPMDGroup (“A”, […], VirtualNode)

// In each member myGroup.barrier (“2D”); // Global Barrier myGroup.barrier (“vertival”); // Any Barrier myGroup.waitFor (“north”,”south”,“east”,“west”);

A

Still,

not based on raw messages, but

Typed Method Calls

==> Components

Denis Caromel 12

Parallel, Distributed, Hierarchical

Componentsfor the Grid

Composing

Denis Caromel 13

A CORBA Component

MyBusiness

Component

Component interface

Facets

Eventsources

Eventsinks

Attributes

Receptacles

OF

FE

RE

DR

EQ

UIR

ED

Courtesy of Philippe Merle, Lille, OpenCCM platform

Denis Caromel 14

Building CCM Applications =Assembling CORBA Component Instances

Provide + Use, but flat assembly

Denis Caromel 15

Content

Controller

The Fractal model:Hierarchical Component

Defined by E. Bruneton, T. Coupaye, J.B. Stefani, INRIA & FT

Denis Caromel 16

Content

Controller

Interface = access point

Denis Caromel 17

Content

Controller

Hierarchical model : composites encapsulate primitives encapsulate Java code

Denis Caromel 18

Content

Controller

Component Binding

Denis Caromel 19

Binding = at Deployment or Dynamically

Content

Controller

Component = runtime entity

if needed

Denis Caromel 20

3. Parallel and composite component

1. Primitive component

2. Composite component

ProActive Components for the GRID An activity, a process, …potentially in its own JVM

C D

Composite: Hierarchical, and

Distributed over machines

Parallel: Composite

+ Broadcast (group)

Denis Caromel 21

Wrapping Legacy MPI Components

MPI CodeC/Fortran:

Messages on Tags

sent/converted to

Method Calls

ProActive Java:

Method Calls

sent as

Messages on Tags

Virtual Nodes for Deployments

Denis Caromel 22

EnvironmentDeploying

Denis Caromel 23

How to deploy on the Various Kind of Grids ?

Internet

EJBServletsApache Databases

Internet

ClustersParallelMachine

LargeEquipment

Internet

Job management forembarrassingly parallel application (e.g. SETI)

Denis Caromel 24

Abstract Deployment Model

Problem:• Difficulties and lack of flexibility in deployment

• Avoid scripting for: configuration, getting hosts, connecting, etc.

• Capture Parallel and Distributed nature of codes

A key principle: Virtual Node (VN) + XML deployment file • Abstract Away from source code:

• Machines

• Creation Protocols

• Lookup and Registry Protocols

Protocols and infrastructures:

• Globus, ssh, rsh, LSF, PBS, … Web Services, WSRF, ...

Denis Caromel 25

C A B

VNa VNb

C A B

VNc = VN(a,b)

XML Deployment (Not in source)

Separate or Co-allocation

Denis Caromel 26

Monitoring of RMI, Globus, Jini, LSF cluster Nice -- Baltimore

ProActive IC2D:

Width of links

proportional

to the number

of com-

munications

Denis Caromel 27

ProActiveApplications

Denis Caromel 28

Jem3D

Denis Caromel 29

JEM 3D : Java 3D Electromagnetism together with Said El Kasmi, Stéphane Lanteri (caiman)

Maxwell 3D equation solver, Finite Volume Method (FVM)Pre-existing Fortran MPI version: EM3D (CAIMAN team @ INRIA)

Up to 294 machines at the same time (Intranet and cluster)

Large data sets: 150x150x150 (100 million facets)

temps d'exécution de la boucle principale (sur cluster)

0

100

200

300

400

500

600

700

800

900

0 10 20 30 40 50 60 70nombre de processeurs

tem

ps

(sec

on

des

)

21*21*21

31*31*31

43*43*43

55*55*55

81*81*81

97*97*97

113*113*113

121*121*121

taille du maillage

Denis Caromel 30

Recent BenchmarksSeq. Java/Fortran: 2

Comparison:

Jem3D over

- ProActive/RMI Sun

- ProActive/RMI Ibis

Em3D in

- Fortran/MPI

On 16 machines:

Fortran MPI: 13.8

ProActive/Ibis: 12

ProActive/RMI: 8.8

Grid experiment on 5 clusters (DAS 2): Speed up of 100 on 150 machines

Denis Caromel 31

Perspective:

Real P2P

Denis Caromel 32

Architectures: Server to Peer-To-Peer (P2P)

Internet

EJBServletsApache Databases

Denis Caromel 33

Pure P2P: Definition

Only PEERs, no above everything, top level, server(s)

Every peer is, somehow, also a server

No master … No slave !

System get organized dynamically, without static configuration

Coherent, desired behavior, dynamically emerges

Denis Caromel 34

P2P Examples (1)

Denis Caromel 35

P2P can be difficult:need to be fault-tolerant, self healing

Denis Caromel 36

Not a P2P system

Denis Caromel 37

Neither a P2P system

Denis Caromel 38

P2P Examples (2)

Denis Caromel 39

P2P Examples (2bis)

Denis Caromel 40

A P2P system at work

Credit: from the movie Atlantis, Luc Besson

Denis Caromel 41

Conclusion

Available in Open Source ProActive http://ProActive.ObjectWeb.org in

Application: 3D Electromagnetism on 300 machines at once Goal: Towards a few 1000s !

Summary:•Programming: Distributed Objects, OO SPMD•Composing: Hierarchical Components•Deploying: ssh,Globus,LSF,PBS,…,Web Services

Denis Caromel 42

Next ProActive events

EuroPar 2004, Pisa, Italy, Monday 30th August, Full Day Tutorial• ProActive Tutorial, and

• Hands-on session

User Group & Grid Interoperability, Nice, ETSI, Oct. 18-20

• Detailed presentation of the platform

• User presentations and feedbacks

• Contest & Grid PLUGTESTS

Current Figures: Web Page Hits: ~ 2K-3K / months, Downoad: 150-300 / month, Users: us, mx, br, cl, ch, it, ...