A COOL Lecture

Mantis H.M. Cheng

Department of Computer ScienceUniversity of Victoria

Email: mcheng@cs.uvic.caHome page: http://www.csc.uvic.ca/~mcheng

2

An Overview

• I. Why Crisp?

• II. What is Crisp?

• III. COOL

• IV. Data Flow Computation

• V. Distributed Mutual Exclusion

• VI. Logical Clocks

• VII. Future of COOL

3

I. Why Crisp?

• The main goal:

Keep it as simple as possible!

Performance! Performance! Performance!

• The philosophy:

4

Concurrency Requirements

• Support of concurrency is a must indistributed real-time (reactive) controlsystems.

• How to express concurrency?

• Which support platform will meet ourperformance requirements?

• What about fault tolerence requirements?

5

Performance Bottleneck

• Today’s network bandwidth is approachingmemory bandwidth.

• Distributed real-time computing requiresfaster reaction time.

What to do when the network data arrivesfaster than context-switching and interrupt-latency on your typical RTOS?

6

Some Possibilities

• Build faster and smarter network interfacecard (it is now a H/W problem).

• Use symmetric multiprocessors.

• Fine-tune the kernel of RTOS for fastercontext-switching and interrupt handlingcapabilities.

• Call Bill Older at CTL (x32234)!

7

Performance Culprits

• UNIX-style processes are too heavy!

• Even with threads, context-switching timeis in the order of 10-50 microseconds.

• Interrupt-driven I/O is not suitable fordealing with high-speed network.

• Sophisticated runtime schedulingalgorithms add unnecessary overhead toreaction time.

8

Performance Culprits (Cont’d.)

• Using inappropriate/inefficient abstractionsfor concurrency introduces unduecomplexities in programming, thus reducesperformance and reliability.

• Impedance mismatch between S/W andH/W components creates unnecessary dataand protocol conversion.

9

Abstractions for Concurrency

• Monitors (e.g., Mesa, Concurrent Pascal)

• Processes (e.g., UNIX, MACH)

• Threads (e.g., Java, MACH)

• Objects (e.g., Java, Ada)

Today, objects seem to be the predominant

model of abstraction.

10

Objects

• An object is an encapsulated piece of data(its state) with a well-defined methodinterface (its behaviour).

• Objects are invoked via one of thepredefined methods.

• Most objects are passive; their states do notinclude a program counter or a stack.

11

Q. What to do when we need concurrent objects?

Objects and Concurrency

• The implementation of most objectsassumes a single thread of control executingtheir methods.

• Most object-oriented programminglanguages/systems do not support objects asunits of concurrency/scheduling.

12

Objects and Threads

• Threads are typically implemented ascoroutines within a shared address space.Context-switching is in the order of tens ofmicroseconds.

• Objects and threads are often distinctentities.

• Mapping concurrent objects into threads isthe most common approach.

13

Q. What to do when most passive objects are not thread-safe?

Threads and Concurrent Objects

• Each concurrent (active) object has acontext which includes the object’s stateand the thread’s state, its program counterand stack.

• Allocation/deallocation of active objects aredifferent from the passive ones.

14

Active and Passive Objects

• Proper synchronisation is needed if methodinvocations are to be interleaved non-deterministically.

• Method invocations must be handleddifferently; context-switching is requiredfor active objects.

• Simple locking/unlocking may introducedeadlocks.

15

Additional Problems

• Synchronisation constraints cannot be easilycombined under inheritance.

• Rollback and error recovery are painfuldue to the lack of understanding in mixingactive and passive objects.

• Typical (user-level or kernel-level) threadschedulers are not scalable to large numberof active objects.

16

Distributed Objects

Q. How can an object invoke another across anetwork transparently?

Q. What if the network is not reliable, that is,a method invocation may fail?

Q. What if a reference to an object is beingsent across a network?

Q. Does a particular thread library supportnetwork objects?

17

“Complexity in an engineeringdiscipline is a sign of immaturity.”

Niklaus Wirth

18

“Keep it as simple as possible,but not simpler.”

Albert Einstein

19

Crisp Approach

• Design an ultra fast and light platform forsupporting concurrent active objects.

• Use ATM H/W support for inter-objectcommunication (messages are cells).

• A simple run-to-completion executionmodel eliminates the need for scheduling.

• Message processing is atomic.

20

An Analysis of Active Objects

Active object = Object’s state + Thread’s state + Methods

Thread’s state = Registers + Program Counter + Stack

21

Simplifications

• Execution of every object always startsfrom the beginning (implies no programcounters).

• A “run-to-completion” model of methodexecution (implies no multiple stacks).

• Only persistent state variables are savedafter each method execution (implies noregisters).

22

Scheduling

• Method invocations are asynchronous(implies true message-based).

• Scheduling policy is determined by messageload (more messages means morefrequent).

Active object = Object’s state + Message Dispatcher

23

II.What is Crisp?

• Crisp is a highly efficient runtimeenvironment for supporting a large numberof distributed active objects.

• Local or remote active objectscommunicate via asynchronous messages.

• Messages are mapped onto ATM cells, andare delivered in order with a lowprobability of being lost.

24

Crisp Features

• Each active object is unique in a distributedCrisp environment.

• Timers are built-in and used frequently torecover from errors.

• Every message is processed atomically.Failures/errors occurring during messageprocessing are mapped into lost messages.

25

Applications and Performance

• Crisp’s intended applications are high-speeddistributed reactive control systems, such asATM switching (e.g., Phoenix/cyclone).

• On a 100MHz PowerPC, active objectswitching overhead is under 1 microsecond.

• Active objects in Crisp are extremely light-weight.

26

Crisp Architecture

O1

Crisp Engine Crisp Engine

ATM Network

Active Objects Active Objects

O2 O3 On Om

27

Crisp Internal Architecture

O1

Crisp Engine

ATM Network

Active Objects

O2

O3

Tx/R

x Bu

ffer

sRead/write

DMA

28

III. Crisp Object cOordination Language

• It does not support while, for, switch,pointers, arrays or structures.

• Data manipulation and machine dependentoperations are implemented in an externallanguage (e.g., C/C++, Java).

• COOL is a pure synchronisation andcoordination language for specifying activeobjects in Crisp.

29

Conceptual Runtime Architecture

• Active objects in COOL are supported byCrisp; passive objects are accessed viaexternal function interface.

Active Objects

Crisp Engine

PassiveObjects

30

Actors and Actor Classes

• Active objects in COOL are called actors.

• Every actor is an instance of some actorclass, which specifies the actor’s internalstate and message/method interface.

• Actors (local or remote) communicate via asingle asynchronous messaging primitive.

31

Two actor identities are equal if and

only if they denote the same actor.

Actor Equality

• Every actor has a globally unique identity.

• An actor identity includes its class name,instance number, and its localityinformation.

32

Primitive and User-defined Types

actor class P {

// instance variables declaration

int i = 0; bool c = false; string s = “joe”;

actor q = none; Timer t = none;

// message/method interface

message INIT ( actor r ) { // the very first message

q = r;

}

. . .

}

P p1 = new P; P p2 = new P;

INIT( p2 ) => p1; INIT( p1 ) => p2;

33

Actor Categories

• There are two main categories of actors:anonymous and regular. For example,

actor r; // an anonymous actorP p = new P; // a regular actor

• An anonymous actor is an alias of anotheractor; its runtime type changes from time totime. It is a polymorphic actor.

34

Default Attributes and Messages

• Every actor has a default read-only attributecalled owner, which stores the identity of itscreator, and is initialised by new.

• Every actor accepts two default messages:

message INIT() from owner { // does nothing }message STOP() from owner { terminate; }

35

Message Primitives

• A message has a functional syntax: f(x,y)denotes a message f with arguments x and y.

f(x,y) => p; // send f(x,y) to actor pf(x) => self; // send to itselfmessage h( int x ) {

f( x+x ) => *; // a reply to sender of h(x)}message g( int y ) {

=> r; // forward g(y) to r}

36

Legality of Messages

• Every message sent is checked against themessage interface of its receiver.

• It is a compile-time error if they are not typecompatible, or if the message is undefined.

• Actors may be sent as values of a message.

• Regular actors are assignable to anonymousactors.

37

Atomicity of Message Processing

There are three possible outcomes after eachmessage is being processed:

• commit (successful, commits all updates andmessages sent) (Default)

• abort (unsuccessful, undoes all changes andmaps the current message as a lost message)

• terminate (successful, commits and thenremoves this object)

38

State Labels and Message Blocks

actor class P {

message INIT ( int n ) from owner { . . . ; becomes IDLE; }

IDLE:

message f ( int x ) { . . .; becomes RUNNING; }

message g ( bool c ) { . . . }

RUNNING:

message f( int x ) { . . . }

message h( string s ) { . . . ; becomes READY; }

READY:

message f ( int x ) { . . . }

message k ( int x ) { . . .; becomes IDLE; }

}

39

A Predefined Timer Actor Class

actor class Timer {

int ticks = 0;

message INIT ( int n ) from owner {

if ( n > 0 ) { ticks = n; becomes RUNNING } else terminate;

}

RUNNING:

message TICK from none { // internally generated

- - ticks;

if ( ticks == 0 ) { TIMEOUT => (actor) owner; terminate; }

}

message STOP from owner { terminate; }

}

40

Crisp Engine as an Actor

extern actor new_actor( int );

actor class Crisp {

int my_id; actor mom;

message INIT ( int id ) { // cold boot

my_id = id; mom = *; becomes RUNNING; }

RUNNING:

message NEW ( int clsnum ) from mom { // create an actor

ACTOR( cls, new_actor( clsnum ) ) => (actor) mom;

}

message INIT from mom { // warm boot

// re-initialize all internal engine states and start all over }

}

41

Distributed COOL/Crisp

ActiveObjects

CrispEngine N

ActiveObjects

CrispEngine 0

ActiveObjects

CrispEngine 1M

onito

r (M

om)

N.B. Only Mom may communicate with each Crisp engine.

42

Automated Debugging

• In high speed communication/distributedsystems, debugging is tedious:– too much information

– happens too fast

– hard to correlate messages/events

– print statements affect timing, thus behaviour

• We need a “logic analyser” for distributedreal-time systems.

43

Monitor as a Logic Analyser

• From a behavioural model, we generate amessage/trace recogniser and incorporate itinto the Monitor.

• At runtime, the Monitor communicates withthe Crisp engine(s) for collecting relevantmessages and determines whether themessage sequence is acceptable.

44

Behavioural Modelling

• COOL has simple, precise and formalsemantics (based on a variant ofasynchronous pi-calculus with built-intimers).

The separation of coordination from data

manipulation has made possible behavioural

modelling of COOL applications.

45

Performance Estimation

• Every statement in COOL is boundedexcept the external functions.

• Execution time of each Crisp primitiveoperation can be estimated.

It is possible to deduce simple quantitative

results, such as message buffer size, worst case

message processing time per actor, etc.

46

IV. Data Flow Computation

• It is a concurrent model of computationbased on data flowing in a network ofprocessing elements.

f

x1

xn

y1

ym

yi1 = ... = yi

m = f(xi1,...,xi

n)

47

Property of a Data Flow Network

• Each processing element/node may have“n” input streams and “m” output streams.

• All output streams are identical.

• A node is enabled as long as there are dataavailable in all its input streams.

• A network is data-driven if every nodeproduces data to its output streams when itis enabled.

48

A Sum Node

+

x

y

z

zi= xi + yi

49

A Zip Node

zip

x

y

z

zi= (xi , yi )

50

A Simple Numerial Problem (1)

• Compute the following polynomial:

• Using Horner’s rule, we derive:

P(x) = 5x3 + 3x2 - 2x + 1

P(x) = x(x(5x + 3) - 2) + 1

51

A Static Data Flow Network (1)

G( x ) = ( 5x+3, x )H( u, x ) = (x*u - 2, x )

F( v, x ) = v*x + 1P(x) = F(H(G(x)))

HG Fx (u,x) (v,x) y

52

A COOL Solution (1)

HG Fx (u,x) (v,x) yA B C D

actor class G { actor in, out; message INIT ( actor l, r ) { in = l; out = r; } message A( int x ) from in { B( 5*x+3, x ) => out; }}

actor class H { actor in, out; message INIT ( actor l, r ) { in = l; out = r; } message B( int u, int x ) from in { C( u*x-2, x ) => out; }}

53

A COOL Solution (1) Cont’d

HG Fx (u,x) (v,x) yA B C D

actor class F { actor in, out; message INIT ( actor l, r ) { in = l; out = r; } message C( int v, int x ) from in { D( v*x+1, x ) => out; }}

message INIT { G g = new G; H h = new H; F f = new F; INIT( self, h ) => g; INIT( g, f ) => h; INIT( h, self ) => f; A( 1 ) => g;}message D( int y ) { // y = P(1); . . . }

54

Prime Numbers Problem (2)

• Generate an infinite sequence of all primenumbers.

• Use the Sieve method:

2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19

20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,...

55

A Dynamic Data Flow Network (2)

Ints - generate all integersFilter - eliminate all multiples of a prime

Filter2Ints

2,3,4,5,6,7,8,...

Filter3

3,5,7,9,11,...

56

A Dynamic Data Flow ... (2)

actor class Ints { Filter f; int v = 1; message INIT () { f = new Filter; NEXT => self; becomes GENERATOR; }GENERATOR: message NEXT { NUM( ++v ) => f; NEXT => self; }}

Filter2Ints

2,3,4,5,6,7,8,...

Filter3

3,5,7,9,11,...

NUM NUM NUM

57

A Dynamic Data Flow ... (2)

actor class Filter { Filter f; int prime; message INIT { becomes PRIME; }PRIME: FILTER: message NUM ( int p ) { message NUM ( int n ) { prime = p; if ( (n mod prime) != 0) { f = new Filter; NUM( n ) => f; becomes FILTER; } } }}

Filter2Ints

2,3,4,5,6,7,8,...

Filter3

3,5,7,9,11,...

NUM NUM NUM

58

Data Flow Computation in Crisp

• So far, these are examples of data-drivendata flow computation.

• They seem to work well in COOL/Crisp aslong as the network topology is a pipeline.

• For the following types of network, we havea problem:

fx

y

zzi = f ( xi, yi)

59

• If we represent x and y as message streams,we have no control on the order of arrival ofxi and yi.

Merge Anomaly in Crisp

fx

y

zzi = f ( xi, yi)

next message

x0 x1 x2 y0 x3 y1

60

Merge Anomaly in Crisp (cont’d)

• Multiple message streams are merged non-deterministrically by Crisp into one perobject.

• It is difficult to control the reception rates ofmultiple message streams without someform of buffering, i.e., one buffer permessage stream.

61

Unbounded Buffers in COOLextern int Qinit(), EnQ( int q, int x ), int DeQ( int q ), bool EmptyQ();actor class Buffer { actor out; int q;

message INIT { q = QInit(); becomes EMPTY; }EMPTY: STUCK: message PUT( int x ) message PUT ( int x ) { { EnQ( q, x ); becomes NONEMPTY; } GET( x ) => out; message NEXT becomes EMPTY; { out = *; becomes STUCK; } }

NONEMPTY: message NEXT { GET( DeQ(q)) => *; if (EmptyQ(q)) becomes EMPTY; } message PUT( int x ) { EnQ( q, x ); }}

62

V. Distributed Mutual Exclusion

• Consider the problem of coordinating twocontrol systems share a common switch,where they need access to switch memory:

CU1

SW

CU2

r/w

r/w

63

Distributed Atomic Lock

• A simple solution is to introduce an atomiclock on the switch, where an indivisibletest-and-set operation is provided.actor class AtomicLock { message INIT { becomes FREE; }FREE: LOCKED: message TEST-AND-SET { message TEST-AND-SET FREE => *; { LOCKED => *; } becomes LOCKED; message CLEAR } { becomes FREE; }}

64

A Use Case of Distributed Lock

DLCU1 CU2

FREE

LOCKED

TEST-AND-SET

TEST-AND-SET

FREE

LOCKEDCLEAR

FREE

65

Who has the lock?

DLCU1 CU2

FREE

LOCKED

TEST-AND-SET

TEST-AND-SET

FREE

LOCKEDTEST-AND-SET

LOCKED

66

An Improved Atomic Lock

• We need to remember who owns the lock!actor class AtomicLock { actor owner = none; message INIT { becomes FREE; }FREE: LOCKED: message TEST-AND-SET { message TEST-AND-SET { FREE => *; if (* == owner) { FREE => *; } owner = *; else { LOCKED => *; } becomes LOCKED; } } message CLEAR from owner {

becomes FREE; owner = none; }

}

67

Bounded Atomic Lock

• What if the CLEAR message is lost!actor class AtomicLock { Timer t; message INIT { becomes FREE; }FREE: LOCKED: message TEST-AND-SET( int n ) message TEST-AND-SET( int n ) t = new Timer( n ); { LOCKED => *; } FREE => *; message TIMEOUT becomes LOCKED; { becomes FREE; } } message CLEAR

{ becomes FREE; }}

68

VI. Logical Clocks

• An event a happened-before event b,denoted by a->b, is defined as follows:

• 1) a and b are on the same machine, and aindeeds happens before b;

• 2) a is the sending event and b is thereceiving event;

• 3) a->b and b->c, then a->c.

69

Logical Clocks in Crisp

• In a distributed Crisp system, we sometimesneed to know the ordering of events,possibly for debugging or for timestamping.

• If a->b, then L(a) < L(b), where L(e) is thelogical clock value of event e.

• Using the happened-before relation, we candefine logical clocks for individual actor orCrisp engine as follows.

70

Logical Clock per Actor

• Each actor p maintains a logical clock Lpwhich is updated as follows:

• 1) For every event a, {++Lp; L(a) = Lp;};

• 2) When p sends a message m, ittimestampes m with Lp;

• 3) When p receives a message m with atimestamp t, {Lp=max(Lp,t)+1; L(m)=Lp}.

71

Logical Clocks in COOL

extern int max( int, int );actor class P { int Lp; // logical clock message INIT { Lp = 0; } message f ( int ts, . . . ) { Lp = max( ts, Lp ) + 1; // L(f) is Lp . . . // “f” happended-before “g” g ( ++Lp, . . . ) => *; . . . // “g” happened-before “h” h ( ++Lp, . . . ) => self; }. . .}

72

Properties of Logical Clocks

• If event a causes event b, then a must havehappened before b.

• If a happened before b, then L(a) < L(b).

• Therefore, any causally related events areordered by logical clocks.

• But, L(a) < L(b) does not imply a causes b!

• So, if a is supposed to cause b, but L(a) >L(b), then we have a problem!

73

VII. Future of COOL

• COOL is a relatively simple language, andmay easily be embedded in C++ or Java.

• It is a pure coordination language.

• The structure of an actor follows thestructure of a finite-state machine.

• Everything in COOL is aboutcommunication and coordination amongfinite state machines.

74

A Generalized Send

• In COOL/Crisp, we have a basic “=>” forsending and forwarding messages.

• Consider the following extensionf(x, y) from p => q

where an optional sender may be specified.• This extension will add tremendous

flexibility! (may even introduce securityproblem as well)

75

Use Case of Generalized Send (1)

NameServerC1 C2

CONNECT(“c2”,x)

CONNECTED

CONNECT(x) from C1

76

Use Case of Generalized Send (2)

NamerC1 INET

CONNECT(“http”)

CONNECTED

CONNECT(“http”) from C1

HTTPD

CONNECT from C1

77

Use Case of Generalized Send (3)

GC1 G’

DATA(“c2”,x)

ACK

DATA(“c2”,x,C1)

C2

DATA(“c2”,x,C1)

DATA(x) from C1

Try to deliver DATAreliably

78

Actor Composition

• Renaming message names doesn’t changethe behaviour of an actor. Therefore,we can define a “new” actor class in termsof another existing actor class:

actor class P is Q where INS is PUT, DEL is GET;

• Taking two existing actor class Caller andCallee, we should be able to define

actor class Call is Caller + Callee;