The Elastically Scalable Database™

1  

2  

20th Century Database

Powerful Query Language

Industry Standards

Data Guarantees

Employee Skills

Tools

Existing Data

44%

21%

19%

4%3%

9%

ORACLEIBMMicrosoftSybaseTeradataOthers

3  

21st Century Problem Commodity Datacenters ✗

Big Data ✗ Modern Workloads ✗ 24x7 Operation ✗

Geo-distribution ✗ Developer Empowerment ✗

Powerful Query Language

Industry Standards

Data Guarantees

Employee Skills

Tools

Existing Data

4  

Database Crisis

Source: Marc Bojoly

Wikipedia Flickr

Facebook

Amazon

Google

5  

Jim Starkey

‣  DEC RDB/ELN ‣  InterBase ‣  Firebird ‣  Falcon ‣  BLOBS ‣  MVCC

“Elastically Scalable Transactions represent the biggest breakthrough in database technology in 25 years”

6  

"   “An emergent behavior can appear when a number of simple entities operate in an environment, forming more complex behaviors as a collective.”

"   - Wikipedia

Emergent Database Architecture

7  

Second  Machine    Instant  Performance  Increase  

"  Second machine typically doubles TPS

"  Second machine is added to live database while it is running at 1,000’s of TPS

"  Performance increase is immediate

"  BTW - you can take either machine away and the database keeps running without data loss

NuoDB Plus One

8  

Second  &  Third  Machine    Instant  Performance  Increase  

"  Third machine typically triples single machine TPS

"  Third machine is added to live database while it is running at 1,000’s of TPS

"  Performance increase is immediate

"  BTW - you can take any machine away and the database keeps running without data loss

Adding a Third Machine

9  

Number  of  Nodes  

TPS  "   Technical Details: ‣  2-9 Tx engines ‣  1 storage

manager ‣  Best sustained TPS

and # clients combination

‣  50% updates

Nodes TPS

MySQL 1 3,000 NuoDB 1 4,500

NuoDB 9 27,000

NuoDB running on 9 nodes was approx. 9x faster than MySQL running on 1 node.

More Machines? Bring ‘em On

10  

Number  of  EC2  Nodes  

TPS  

‣  Nuodb scales linearly on EC2

‣  Per-node performance on m1.large nodes approx 50% of our commodity servers

‣  Just started on optimizing

‣  RDS runs on 1 node, and gets overloaded with 10+ connections

Or Scale-out on IAAS

Trek Palmer

Building An Elastically Scalable Database The Easy Way

13  

Tonight’s Agenda "   A Bit About Me "   Introduction to NuoDB "   Architecture Overview "   Some Unnatural Acts

"   Quick Demo "   Beer

14  

A Bit About Me

"   A Refugee from Academia "   -Researched Programming Languages "   -Transactional Memory impl. and

semantics "   Worked on distributed metadata

database for HDS HCP "   -Clustered appliance

15  

What is NuoDB?

"   Elastically scalable "   Multi-tenant "   Transactionally Consistent "   Easy to Manage

16  

娜 graceful, like a cloud

Architecture "   Three tiers

"   Each is independent "   Single model for all

environments "   Extensible at

various points

Management  

TransacEon  Handling  

Storage  

18  

Agents

"   Management tier "   Provision hosts for use "   Expose XML messaging for

management "   Make scripting and automation easy

19  

Brokers

"   Agent with additional special knowledge

"   At least one per domain "   Redirects clients to TE

"   -Clients need no knowledge of topology

"   -Brokers are responsible for any load-balancing

20  

Transaction Engines "   Peer-to-peer "   In-memory "   Multi-Version Concurrency Control "   Asynchronous messaging (replication) "   Atoms

21  

Storage Managers

"   Persistence points for atoms "   Key-value backing stores "   -Local FS, S3, HDFS

"   Independent archives

22  

And Now, Some Unnatural Acts

23  

Sharding, an Unnatural Act "   The ideal DB application

DB  

client  

client  

Scales  up  to  the  capacity    of  a  single  node  

What  if  you  need  more  read    and/or  write  throughput?  

24  

Sharding

"   Shard the DB among several nodes

DB0   DB1  

Client1   Client2  

New  Client  Layer   Now  you  need  to  implement  consistency  in  your  applicaEon  

TransacEonal  consistency  is  very  very  hard  to  get  right  

25  

Other Sharding Bugbears

"   Global operations (searches, scans) "   -Doing joins in the application "   -Implementing Cursors "   -Chunking and memory

management "   And, of course, adding or removing

shards

26  

Scaling Shards

"   A recipe for changing the number of shards

"   1) Ask boss for permission "   2) Provision hardware "   3) Rewrite the app over 6 months "   4) Hope / Pray

27  

NuoDB Solution

Host A

  Broker  

Host B

  Agent  

Host C

  Agent  

Domain

 

Management  Client  

28  

NuoDB on a single node

Host A

  Broker  

Txn  Engine  

Host B

  Agent  

Storage  Manager  

Host C

  Agent  

Domain

 

Management  Client  

Database A

Client  

29  

NuoDB Scaling out

Host A

  Broker  

Txn  Engine  

Host B

  Agent  

Storage  Manager  

Host C

  Agent  

Domain

 

Management  Client  

Database A

Client   Client   Client  

Txn  Engine  

Storage  Manager  

30  

NuoDB ‘Adding a Shard’

Host A

  Broker  

Txn  Engine  

Host B

  Agent  

Storage  Manager  

Host C

  Agent  

Domain

 

Management  Client  

Database A

Client   Client   Client  

Txn  Engine  

Storage  Manager  

Txn  Engine  

Storage  Manager  

31  

NuoDB ‘Sharding’

"   Literally as simple as just adding nodes

"   No client code had to be harmed in the making of this distributed database

"   -Brokers hide topology changes "   -NuoDB is transactionally consistent

32  

Eventual Consistency

"   Eventual consistency is latent inconsistency

"   -Not transactionally consistent "   -Application porting is non-trivial "   -Performance/Correctness tradeoff

icky

33  

NuoDB Consistency

"   NuoDB is transactionally consistent "   All the time, everywhere "   When a transaction is committed,

it’s guaranteed consistent "   Tradeoff is between Performance and

Availability

34  

Multi-Tenancy

"   Traditional databases monopolize a node

"   NuoDB supports many databases in a single pool of machines (a domain)

"   Each DB can be scaled as needed, independantly

35  

Multi-Tenancy Example

Host A

  Broker  

Txn  Engine  

Host B

  Agent  

Storage  Manager  

Host C

  Agent  

Domain

 

Management  Client  

Database A

JDBC  Client  

36  

Multi-Tenancy Example

Host A

  Broker  

Txn  Engine  

Host B

  Agent  

Storage  Manager  

Host C

  Agent  

Domain

 

Management  Client  

Database A

JDBC  Client  

Txn  Engine  

JDBC  Client  

37  

Multi-Tenancy Example

Host A

  Broker  

Txn  Engine  

Host B

  Agent  

Storage  Manager  

Host C

  Agent  

Domain

 

Management  Client  

Database A

JDBC  Client  

Txn  Engine  

JDBC  Client  

Storage  Manager  

Txn  Engine  

Database 1

38  

Multi-Tenancy Example

Host A

  Broker  

Txn  Engine  

Host B

  Agent  

Storage  Manager  

Host C

  Agent  

Domain

 

Management  Client  

Database A

JDBC  Client  

Txn  Engine  

JDBC  Client  

Storage  Manager  

Txn  Engine  

Database 1

SQL  Client   ...  

And now, a demo…

The Elastically Scalable Database™

40