Mission impossible

1/24/13 Cloud-Native: Flexible, Redundant, & Infinitely Scalable by Nature.

Mission: imPossible

Scaling in the Cloud

1Saturday, May 11, 13


The Core: Message Bus100% Cloud Native



What Cloud Translates To

3

Small to mid size instances

Lack of Control Over Physical Resources

No delay in expanding

Limited Storage Options

Redundancy is Easy Redundancy is Necessary

Saturday, May 11, 13


What Does Your Data Look Like?• Log Data Classic OLTP• By Nature, it’s Distributed • Long & Mid Term Storage• Eventual v. Immediate Consistency • Speed of Retrieval -Classing OLAP Warehouse



Requirements

5

• No Data Loss• Ability to Pause, Promote, and Update Seamlessly• Do Not be the Bottleneck • Allow for Out of Sequence Events• Restate and Correct Analytics



Tools• Chef• Liquibase• HornetQ• Rep Mgr• Postgres



Distributed OLTP

7

Disparate event types can be sent to separate instances.

Within the same event, parallel writes don’t require synchronous multi-masters.

Batch Process Writes with Copy Into



Vertical Sharding- Inheritance is your friend!

8

One parent per event, sharded by time, allows for efficient querying, individual backups, and general sanity.




9

Constraints: Timestamp not DateEvent ID (pkey)Explain Analyze!




10

WHILE startday <= endday endday LOOP tomorrow := startday + interval '1 day'; index_name := 'idx_name' || replace(startday::text, '-', ''); tname := 'event_' || to_char(startday, 'YYYY_MM_DD'); EXECUTE 'create table ' || quote_ident(tname) ||'() inherits(event)'; EXECUTE 'ALTER TABLE ' || quote_ident(tname) ||' ADD CONSTRAINT createdck CHECK (created >= ' || quote_literal(startday) ||' AND created < '|| quote_literal(tomorrow) ||')'; EXECUTE 'CREATE INDEX '|| index_name ||' ON '|| quote_ident(tname)||'(act_id) ; startday := tomorrow;END LOOP;



Close The Shard

11

FOR tbl IN SELECT t.table_name FROM information_schema.tables t WHERE LEFT(t.table_name, 6 ) = 'event_' AND t.table_name NOT IN (SELECT table_name FROM information_schema.constraint_column_usage WHERE constraint_name = 'idcheck') AND REPLACE(REPLACE(t.table_name, 'event_', ''), '_', '-')::DATE < NOW()::DATELOOP EXECUTE 'SELECT coalesce(min(event_id), 0), coalesce(max(event_id), 0) FROM ' || quote_ident(tbl) ||';' INTO smin, smax; EXECUTE 'ALTER TABLE '|| quote_ident(tbl) || ' ADD CONSTRAINT idcheck CHECK (event_id BETWEEN '|| smin || ' AND ' || smax ||' )';END LOOP;



Star & Snowflake OLAP

12

Because API calls can’t time out.

Partition-able on fact, account - what works for you.



Build Your Snowflake

13

• Pull Aggregates• Add/Update base fact• Cascade Updates Through Dimensions

Quartz Triggered Calls Three Phase Process:



Build Your Snowflake

14

Driven by Sequential IDPosition Lookup, Pull, & Update in a Single

TransactionSave State for Performance Analysis



Aggregate on Pull

15

CREATE TABLE source_epoch AS SELECT * FROM dblink('dbname=src host=127.0.0.2 port=5432 user=rickastley password=never_gonna_give_you_up', 'SELECT some, thing, COUNT(id), DATE_TRUNC(''hour'', TO_TIMESTAMP(date)), MAX(id) AS id FROM events WHERE id > 50823461 GROUP BY 1, 2 ORDER BY id) AS t(some TEXT, thing, INT event_count INT, hour TIMESTAMP, id INT)



The Defense

16

Bugs for the entire system will find their way through to your warehouse. Do not always trust your data.



What It All Looks Like

17

Writer

Writer




18

Writer

Writer

OLTPTypeA

OLTPTypeB

OLTPTypeB




19

Writer

Writer

OLTPTypeA

OLTPTypeB

OLTPTypeB

OLAPWH

Type1




20

Writer

Writer

OLAPWH

Type1Query Head

OLTPTypeA

OLTPTypeB

OLTPTypeB




21

Writer

Writer

OLAPWH

Type1Query Head

OLTPTypeA

OLTPTypeB

OLTPTypeC

OLTPTypeAOLTPTypeA

OLTPTypeC

OLTPTypeC




22

Writer

Writer

OLAPWH

Type1Query Head

OLTPTypeA

OLTPTypeB

OLTPTypeC

OLTPTypeAOLTPTypeA

OLTPTypeC

OLTPTypeC

OLAPWH

Type1Query Head



http://news.messagebus.com/2013/01/03/mission-impossible/

Questions?





Thank You!24


Mission impossible

Spiritual

Transcript of Mission impossible