A RESTful API for Controlling Dynamic Streaming Topologies

A RESTful API for Controlling Dynamic Streaming TopologiesMasiar Babazadeh!University of Lugano Switzerland @masiarb

Cesare Pautasso!University of Lugano Switzerland @pautasso

1

2

F

CP

F

F

6

P C

8

Web Liquid Streams

x

11

F

CP

F

F

WebSocket

WebRTC

WebSocket

ZeroMQ

ZeroMQ

12

F

Workers

13

F

Operator Elasticity

14

F

Fault Tolerance

15

F

CP

F

FX

15

F

CP

F

15

F

CP

F

F

16

Operator MigrationF

CP

F

16

Operator MigrationF C

P

F

17

Peer OverloadingF

CP

F

17

Peer OverloadingF

CP

F C

18

API

WLS Runtime

P C

19

RPC

20

REST

21

Operator

Tight Coupling

22

Operator

23

Independent Evolution

Operator

24

Operator

Error Detection & Recovery

24

Operator


operators[0].getWorkers();

24

Operator


Exception

24

Operator


GET …/workers

24

Operator


“request timeout”

24

Operator


{!! “error” : …!}

24

Operator


GET …/workers

24

Operator


25

Representations

{ "topology": { "id": "example", "operators": […], "bindings": […] } }

<topology> <id>example</id> <operators> <operator>…</operator> … </operators> <bindings> <binding>…</binding> … </bindings> </topology>

2626

F

CP

F

F

2626

F

CP

F

F

Operators

2626

F

CP

F

F

Bindings

2626

F

CP

F

F

Workers

2626

F

CP

F

F

Peers

2626

F

CP

F

F

Topologies

27

Hypermedia Data Model

28

Peer Discovery

C

F GET /!GET /peers!POST /peers!GET /peers/:pid!DELETE /peers/:pid

29

Peer Representation

{! "peer": ! {!!!!!!"id": "example",! !!!!!"operators": […],! !!!!!"topologies": […],! !!!!!"CPU": [0, 50, 0] ! !!!}! }

GET /peers/example

30

Topologies

GET /topologies!POST /topologies!GET /topologies/:tid!PUT /topologies/:tid!DELETE /topologies/:tid

F

CP

F

31

Topology Representation

{ "topology": { "id": "example", "operators": […], "bindings": […] } }

GET /topologies/example

32

F

Operators

GET /operators!GET /topologies/:tid/operators!GET /topologies/:tid/operators/:oid!PUT /topologies/:tid/operators/:oid!PATCH /topologies/:tid/operators/:oid/script!PATCH /topologies/:tid/operators/:oid/bindings!DELETE /topologies/:tid/operators/:oid

33

{! "operators" : [! {! "topology" : "example",! "id" : "prod",! "workers" : […],! "href" : “…/topologies/example/operators/prod”,! "peer" : "http://IP:port/",! "replicas" : [! ! "http://IP2:port2/topologies/example/operators/prod"!! ! ] !

! },!

]!

}

Operator RepresentationGET /topologies/example/operators

34

“topology” : { “operators: […], "bindings": [ { "from": “…/topologies/example/operators/a”, "to": “…/topologies/example/operators/b”, "type": "round_robin" }, { "from": “…/topologies/example/operators/b”, "to": “…/topologies/example/operators/c”, "type": "broadcast" } ] }

Binding Representation

35

Workers

GET !/topologies/:tid/operators/:oid/workers!GET !/topologies/:tid/operators/:oid/workers/:wid!POST !/topologies/:tid/operators/:oid/workers!DELETE !/topologies/:tid/operators/:oid/workers/:wid

36

“operators“ : [!

!{!

! "topology" : "example",!

"id" : "prod", ! "workers" : [!! ! {!! ! ! "id" : "0",!! ! ! "href" : “…/topologies/example/operators/prod/workers/0”,!! ! ! "operator" : “…/topologies/example/operators/prod”,!! ! ! "uptime" : 3600,!! ! ! "messages" : 42,!! ! ! "req-res-ratio" : 1.5!! ! }, …!

! ],!

! …!

! }!

]

Worker Representation

3838

F

Load Balancing

3838

F

GET /topologies/:tid/operators/:oid/workers

Load Balancing

3838

F

“workers”: [ ! {!! ! ! "id" : "0",!! ! ! "href" : …,,!! ! ! "operator" : …,!! ! ! "uptime" : 3600,!! ! ! "messages" : 42,!! ! ! "req-res-ratio" : 1.5”,!! ! },! !…! ]Load Balancing

3838

F

POST /topologies/:tid/operators/:oid/workers

Load Balancing

3838

F

Load Balancing

3939

F

Fault Tolerance

3939

F

GET /topologies/:tid/operators/:oid/workers

Fault Tolerance

3939

F

“workers”: [ ! {!

! ! ! "id" : "0",!! ! ! "href" : “…/prod/workers/0”,!! ! ! …!! ! },!

{!

! ! ! “id” : “1”,!! ! ! “href” : …,!! ! ! “error”: “request timeout”!

! ! }!

]Fault Tolerance

3939

F

POST /topologies/:tid/operators/:oid/workers

Fault Tolerance

3939

F

Fault Tolerance

3939

F

DELETE /topologies/:tid/operators/:oid/workers/:wid

Fault Tolerance

3939

F

Fault Tolerance

40

F

CP

F

Fault Tolerance

40

F

CP

F

GET /topologies/:tid/operators

Fault Tolerance

40

F

CP

F

“operators”: [ ! {!! ! ! …!! ! },! {!! ! “id” : “C”,!! ! “href” : …,!! ! “error” : !! ! “request timeout”!! ! }! ]Fault Tolerance

40

F

CP

F

PUT /topologies/:tid/operators/C

Fault Tolerance

PATCH /topologies/:tid/operators/bindings

40

F

CP

F

C

Fault Tolerance

40

F

CP

F

C

DELETE /topologies/:tid!/operators/C

Fault Tolerance

40

F

P

F

C

Fault Tolerance

41

F

CP

F

Fault Tolerance

41

F

CP

F

GET /peers/:pid

Fault Tolerance

connection timeout

41

F

CP

F

Fault Tolerance

PUT /topologies/:tid/operators/…

41

F

CP

F

Fault Tolerance

41

F

CP

F

CF

Fault Tolerance

DELETE /topologies/:tid!/operators/…

41

F

CP

F

CF

Fault Tolerance

41

F

P

CF

Fault Tolerance

42

F

CP

F

Peer Overloading

42

F

CP

F

GET /peers/:pid

Peer Overloading

42

F

CP

F

{ !

! “peer”: ! {!

! ! ! …!

! ! ! “CPU” : !

! ! ! [100,100,100]! !

! }!

}Peer Overloading

PUT /topologies/:tid/operators/C

42

F

CP

F

Peer Overloading


42

F

CP

F

C

Peer Overloading

43

F

CP

F

Leaving Peer

43

F

CP

F

DELETE /peers/IP:port

Leaving Peer

43

F

CP

F

Leaving Peer

43

F

CP

F

PUT /topologies/:tid/operators/…

Leaving Peer

43

F

CP

F

CF


Leaving Peer

43

F

CP

F

CF

DELETE /topologies/:tid!/operators/…

Leaving Peer

43

F

P

CF

Leaving Peer

A RESTful API for Controlling Dynamic Streaming TopologiesMasiar Babazadeh!University of Lugano Switzerland @masiarb

Cesare Pautasso!University of Lugano Switzerland @pautasso

45

http://design.inf.usi.ch/

A RESTful API for Controlling Dynamic Streaming Topologies

Software

Transcript of A RESTful API for Controlling Dynamic Streaming Topologies