A RESTful API for Controlling Dynamic Streaming TopologiesMasiar Babazadeh!University of Lugano Switzerland @masiarb

Cesare Pautasso!University of Lugano Switzerland @pautasso

1

2

F

CP

F

F

3

P

4

F

5

C

6

P C

7

8

Web Liquid Streams

8

Web Liquid Streams

8

Web Liquid Streams

9

9

9

9

9

9

9

10

x

11

F

CP

F

F

WebSocket

WebRTC

WebSocket

ZeroMQ

ZeroMQ

x

11

F

CP

F

F

WebSocket

WebRTC

WebSocket

ZeroMQ

ZeroMQ

x

11

F

CP

F

F

WebSocket

WebRTC

WebSocket

ZeroMQ

ZeroMQ

12

F

Workers

13

F

Operator Elasticity

13

F

Operator Elasticity

13

F

Operator Elasticity

13

F

Operator Elasticity

13

F

Operator Elasticity

13

F

Operator Elasticity

13

F

Operator Elasticity

14

F

Fault Tolerance

14

F

Fault Tolerance

14

F

Fault Tolerance

14

F

Fault Tolerance

15

F

CP

F

FX

15

F

CP

F

FX

15

F

CP

F

15

F

CP

F

F

16

Operator MigrationF

CP

F

16

Operator MigrationF C

P

F

17

Peer OverloadingF

CP

F

17

Peer OverloadingF

CP

F

17

Peer OverloadingF

CP

F C

17

Peer OverloadingF

CP

F C

17

Peer OverloadingF

CP

F C

17

Peer OverloadingF

CP

F C

18

API

WLS Runtime

P C

19

RPC

20

REST

21

Operator

Tight Coupling

22

Operator

22

Operator

22

Operator

22

Operator

22

Operator

23

Independent Evolution

Operator

23

Independent Evolution

Operator

23

Independent Evolution

Operator

24

Operator

Error Detection & Recovery

24

Operator

Error Detection & Recovery

operators[0].getWorkers();

24

Operator

Error Detection & Recovery

Exception

24

Operator

Error Detection & Recovery

GET …/workers

24

Operator

Error Detection & Recovery

“request timeout”

24

Operator

Error Detection & Recovery

{!! “error” : …!}

24

Operator

Error Detection & Recovery

GET …/workers

24

Operator

Error Detection & Recovery

25

Representations

{ "topology": { "id": "example", "operators": […], "bindings": […] } }

<topology> <id>example</id> <operators> <operator>…</operator> … </operators> <bindings> <binding>…</binding> … </bindings> </topology>

2626

F

CP

F

F

2626

F

CP

F

F

Operators

2626

F

CP

F

F

Bindings

2626

F

CP

F

F

Workers

2626

F

CP

F

F

Peers

2626

F

CP

F

F

Topologies

27

Hypermedia Data Model

28

Peer Discovery

C

F GET /!GET /peers!POST /peers!GET /peers/:pid!DELETE /peers/:pid

29

Peer Representation

{! "peer": ! {!!!!!!"id": "example",! !!!!!"operators": […],! !!!!!"topologies": […],! !!!!!"CPU": [0, 50, 0] ! !!!}! }

GET /peers/example

30

Topologies

GET /topologies!POST /topologies!GET /topologies/:tid!PUT /topologies/:tid!DELETE /topologies/:tid

F

CP

F

31

Topology Representation

{ "topology": { "id": "example", "operators": […], "bindings": […] } }

GET /topologies/example

32

F

Operators

GET /operators!GET /topologies/:tid/operators!GET /topologies/:tid/operators/:oid!PUT /topologies/:tid/operators/:oid!PATCH /topologies/:tid/operators/:oid/script!PATCH /topologies/:tid/operators/:oid/bindings!DELETE /topologies/:tid/operators/:oid

33

{! "operators" : [! {! "topology" : "example",! "id" : "prod",! "workers" : […],! "href" : “…/topologies/example/operators/prod”,! "peer" : "http://IP:port/",! "replicas" : [! ! "http://IP2:port2/topologies/example/operators/prod"!! ! ] !

! },!

]!

}

Operator RepresentationGET /topologies/example/operators

34

“topology” : { “operators: […], "bindings": [ { "from": “…/topologies/example/operators/a”, "to": “…/topologies/example/operators/b”, "type": "round_robin" }, { "from": “…/topologies/example/operators/b”, "to": “…/topologies/example/operators/c”, "type": "broadcast" } ] }

Binding Representation

35

Workers

GET !/topologies/:tid/operators/:oid/workers!GET !/topologies/:tid/operators/:oid/workers/:wid!POST !/topologies/:tid/operators/:oid/workers!DELETE !/topologies/:tid/operators/:oid/workers/:wid

36

“operators“ : [!

!{!

! "topology" : "example",!

"id" : "prod", ! "workers" : [!! ! {!! ! ! "id" : "0",!! ! ! "href" : “…/topologies/example/operators/prod/workers/0”,!! ! ! "operator" : “…/topologies/example/operators/prod”,!! ! ! "uptime" : 3600,!! ! ! "messages" : 42,!! ! ! "req-res-ratio" : 1.5!! ! }, …!

! ],!

! …!

! }!

]

Worker Representation

37

3838

F

Load Balancing

3838

F

Load Balancing

3838

F

GET /topologies/:tid/operators/:oid/workers

Load Balancing

3838

F

“workers”: [ ! {!! ! ! "id" : "0",!! ! ! "href" : …,,!! ! ! "operator" : …,!! ! ! "uptime" : 3600,!! ! ! "messages" : 42,!! ! ! "req-res-ratio" : 1.5”,!! ! },! !…! ]Load Balancing

3838

F

POST /topologies/:tid/operators/:oid/workers

Load Balancing

3838

F

Load Balancing

3838

F

Load Balancing

3939

F

Fault Tolerance

3939

F

Fault Tolerance

3939

F

GET /topologies/:tid/operators/:oid/workers

Fault Tolerance

3939

F

“workers”: [ ! {!

! ! ! "id" : "0",!! ! ! "href" : “…/prod/workers/0”,!! ! ! …!! ! },!

{!

! ! ! “id” : “1”,!! ! ! “href” : …,!! ! ! “error”: “request timeout”!

! ! }!

]Fault Tolerance

3939

F

POST /topologies/:tid/operators/:oid/workers

Fault Tolerance

3939

F

Fault Tolerance

3939

F

DELETE /topologies/:tid/operators/:oid/workers/:wid

Fault Tolerance

3939

F

Fault Tolerance

40

F

CP

F

Fault Tolerance

40

F

CP

F

Fault Tolerance

40

F

CP

F

GET /topologies/:tid/operators

Fault Tolerance

40

F

CP

F

“operators”: [ ! {!! ! ! …!! ! },! {!! ! “id” : “C”,!! ! “href” : …,!! ! “error” : !! ! “request timeout”!! ! }! ]Fault Tolerance

40

F

CP

F

PUT /topologies/:tid/operators/C

Fault Tolerance

PATCH /topologies/:tid/operators/bindings

40

F

CP

F

C

Fault Tolerance

40

F

CP

F

C

DELETE /topologies/:tid!/operators/C

Fault Tolerance

40

F

P

F

C

Fault Tolerance

41

F

CP

F

Fault Tolerance

41

F

CP

F

Fault Tolerance

41

F

CP

F

GET /peers/:pid

Fault Tolerance

connection timeout

41

F

CP

F

Fault Tolerance

PUT /topologies/:tid/operators/…

41

F

CP

F

Fault Tolerance

41

F

CP

F

CF

Fault Tolerance

PATCH /topologies/:tid/operators/bindings

41

F

CP

F

CF

Fault Tolerance

41

F

CP

F

CF

Fault Tolerance

41

F

CP

F

CF

Fault Tolerance

DELETE /topologies/:tid!/operators/…

41

F

CP

F

CF

Fault Tolerance

41

F

P

CF

Fault Tolerance

42

F

CP

F

Peer Overloading

42

F

CP

F

GET /peers/:pid

Peer Overloading

42

F

CP

F

{ !

! “peer”: ! {!

! ! ! …!

! ! ! “CPU” : !

! ! ! [100,100,100]! !

! }!

}Peer Overloading

PUT /topologies/:tid/operators/C

42

F

CP

F

Peer Overloading

PATCH /topologies/:tid/operators/bindings

42

F

CP

F

C

Peer Overloading

42

F

CP

F

C

Peer Overloading

42

F

CP

F

C

Peer Overloading

43

F

CP

F

Leaving Peer

43

F

CP

F

DELETE /peers/IP:port

Leaving Peer

43

F

CP

F

Leaving Peer

43

F

CP

F

PUT /topologies/:tid/operators/…

Leaving Peer

43

F

CP

F

CF

PATCH /topologies/:tid/operators/bindings

Leaving Peer

43

F

CP

F

CF

DELETE /topologies/:tid!/operators/…

Leaving Peer

43

F

P

CF

Leaving Peer

43

F

P

CF

Leaving Peer

44

44

44

44

44

44

A RESTful API for Controlling Dynamic Streaming TopologiesMasiar Babazadeh!University of Lugano Switzerland @masiarb

Cesare Pautasso!University of Lugano Switzerland @pautasso

45

http://design.inf.usi.ch/