AARON MORTON THE LAST PICKLE

SCALABLE IOT WITH APACHE CASSANDRA

QUICK LOOK AT A DEV CYCLE

REQUIREMENTS IN IOT

HIGH THROUGHPUTREQUIREMENT #1

5

6

7

TRAFFIC SPIKESREQUIREMENT #2

9

IT’S 5PM, SEND THE

DATA!

GEOGRAPHYREQUIREMENT #3

11

12

160ms260ms

GROWTHREQUIREMENT #4

14

WHY CASSANDRA?

LINEAR SCALABILITY

17

the original ring 8 nodes

18

12 nodes 50% improvement

19

CREATE KEYSPACE my_first_keyspace WITH replication = { 'class': 'SimpleStrategy', 'replication_factor': 3 };

20

HIGH AVAILABILITY

22

23

CREATE KEYSPACE iot WITH replication = { 'class':'NetworkTopologyStrategy', 'los_angeles': 3, 'london': 3 };

24Los Angeles London

25

160ms260ms

DATA MODELING

LOOKS LIKE RELATIONAL

SIMPLE TABLES

create table sensor ( id text primary key, location text, date_installed timestamp );

PARTITION KEY

30

hash(partition)

hash(id=“mysensor”)

31

hash(partition)

hash(id=“mysensor”)

32

hash(id=“mysensor”)

33

write

read ???

CLUSTERING KEYSCOMPLEX TABLES

35

CREATE TABLE clustered ( partition int, clustering_key int, primary key (partition, clustering_key) );

36

partition=“my partition”

CLUSTERING_KEY=1 CLUSTERING_KEY=82

CLUSTERING_KEY=104

CASSANDRA STORES ROWS IN SORTED ORDER

38

INSERT INTO clustered (partition, clustering_key) VALUES (1, 1); INSERT INTO clustered (partition, clustering_key) VALUES (1, 2); INSERT INTO clustered (partition, clustering_key) VALUES (1, 3);

39

cqlsh:iot> SELECT * FROM clustered WHERE partition = 1;

partition | clustering_key -----------+---------------- 1 | 1 1 | 2 1 | 3

(3 rows)

40

cqlsh:iot> INSERT INTO clustered (partition, clustering_key) ... VALUES (1, -1); cqlsh:iot> SELECT * FROM clustered WHERE partition = 1;

partition | clustering_key -----------+---------------- 1 | -1 1 | 1 1 | 2 1 | 3

(4 rows)

41

CREATE TABLE sensor_reading ( sensor text, created_at timestamp, reading int, primary key (sensor, created_at) ) WITH CLUSTERING ORDER BY (created_at DESC);

42

INSERT INTO sensor_reading (sensor, created_at, reading) VALUES ('test', toTimestamp(now()), 1);

INSERT INTO sensor_reading (sensor, created_at, reading) VALUES ('test', toTimestamp(now()), 2);

INSERT INTO sensor_reading (sensor, created_at, reading) VALUES ('test', toTimestamp(now()), 3);

43

cqlsh:iot> SELECT * from sensor_reading WHERE sensor = 'test';

sensor | created_at | reading --------+---------------------------------+--------- test | 2017-06-30 01:32:40.533000+0000 | 3 test | 2017-06-30 01:32:39.741000+0000 | 2 test | 2017-06-30 01:32:39.738000+0000 | 1

(3 rows)

SCALE IT FURTHER

45

partition: mysensor 100MM readings?

46

CREATE TABLE sensor_reading_by_day ( sensor text, day date, created_at timestamp, reading int, primary key ((sensor, day), created_at) ) WITH CLUSTERING ORDER BY (created_at DESC);

47

48

CREATE TABLE daily_metrics ( sensor text, day date, readings counter, primary key (sensor, day) );

49

UPDATE daily_metrics SET readings = readings + 1 WHERE sensor = 'mysensor' AND day = '2017-07-07';

50

CREATE TABLE sensor_reading_by_day ( sensor text, day date, created_at timestamp, bucket int, reading int, primary key ((sensor, day, bucket), created_at) ) WITH CLUSTERING ORDER BY (created_at DESC);

51

CREATE TABLE air_sensor_readings ( sensor text, day date, created_at timestamp, bucket int, temp float, humidity int, polution_ppm int, primary key ((sensor, day, bucket), created_at) ) WITH CLUSTERING ORDER BY (created_at DESC);

52

SELECT temp, humidity, polution_ppm FROM air_sensor_readings WHERE sensor = ‘yow_conf' AND day = '2017-07-07' AND bucket = 0 LIMIT 5;

53

temp | humidity | polution_ppm ------+----------+-------------- 27.1 | 50 | 62 27.1 | 50 | 52 27.1 | 50 | 51 27.1 | 50 | 51 27.1 | 50 | 51

(5 rows)

TECHNOLOGY LANDSCAPE

55

EVENTS

LONG TERM HISTORY

FAST ACCESS CURRENT STATE

BULK OPERATIONS

55

56

DISASTER RECOVERY

WEB API STREAMING

INSIGHT PRODUCT DEV

EVENTS

57

BLOB STORE LOW $ LOW PERF

DB HIGH $ HIGH PERF

BLOCKSTORE MID $ MID PERF

EVENTS

58

DISASTER RECOVERY

WEB API STREAMING

INSIGHT PRODUCT DEV

REPUTATION RISK INCREASES

59

ALL TIME BREAK INS

BREAKING IN NOW !

BREAK INS THIS MONTH

REPUTATION RISK INCREASES

60

QUESTIONS?