Post on 10-May-2015
description
The Elastically Scalable Database™
1
2
20th Century Database
Powerful Query Language
Industry Standards
Data Guarantees
Employee Skills
Tools
Existing Data
44%
21%
19%
4%3%
9%
ORACLEIBMMicrosoftSybaseTeradataOthers
3
21st Century Problem Commodity Datacenters ✗
Big Data ✗ Modern Workloads ✗ 24x7 Operation ✗
Geo-distribution ✗ Developer Empowerment ✗
Powerful Query Language
Industry Standards
Data Guarantees
Employee Skills
Tools
Existing Data
4
Database Crisis
Source: Marc Bojoly
Wikipedia Flickr
Amazon
5
Jim Starkey
‣ DEC RDB/ELN ‣ InterBase ‣ Firebird ‣ Falcon ‣ BLOBS ‣ MVCC
“Elastically Scalable Transactions represent the biggest breakthrough in database technology in 25 years”
6
" “An emergent behavior can appear when a number of simple entities operate in an environment, forming more complex behaviors as a collective.”
" - Wikipedia
Emergent Database Architecture
7
Second Machine Instant Performance Increase
" Second machine typically doubles TPS
" Second machine is added to live database while it is running at 1,000’s of TPS
" Performance increase is immediate
" BTW - you can take either machine away and the database keeps running without data loss
NuoDB Plus One
8
Second & Third Machine Instant Performance Increase
" Third machine typically triples single machine TPS
" Third machine is added to live database while it is running at 1,000’s of TPS
" Performance increase is immediate
" BTW - you can take any machine away and the database keeps running without data loss
Adding a Third Machine
9
Number of Nodes
TPS " Technical Details: ‣ 2-9 Tx engines ‣ 1 storage
manager ‣ Best sustained TPS
and # clients combination
‣ 50% updates
Nodes TPS
MySQL 1 3,000 NuoDB 1 4,500
NuoDB 9 27,000
NuoDB running on 9 nodes was approx. 9x faster than MySQL running on 1 node.
More Machines? Bring ‘em On
10
Number of EC2 Nodes
TPS
‣ Nuodb scales linearly on EC2
‣ Per-node performance on m1.large nodes approx 50% of our commodity servers
‣ Just started on optimizing
‣ RDS runs on 1 node, and gets overloaded with 10+ connections
Or Scale-out on IAAS
Trek Palmer
Building An Elastically Scalable Database The Easy Way
13
Tonight’s Agenda " A Bit About Me " Introduction to NuoDB " Architecture Overview " Some Unnatural Acts
" Quick Demo " Beer
14
A Bit About Me
" A Refugee from Academia " -Researched Programming Languages " -Transactional Memory impl. and
semantics " Worked on distributed metadata
database for HDS HCP " -Clustered appliance
15
What is NuoDB?
" Elastically scalable " Multi-tenant " Transactionally Consistent " Easy to Manage
16
娜 graceful, like a cloud
Architecture " Three tiers
" Each is independent " Single model for all
environments " Extensible at
various points
Management
TransacEon Handling
Storage
18
Agents
" Management tier " Provision hosts for use " Expose XML messaging for
management " Make scripting and automation easy
19
Brokers
" Agent with additional special knowledge
" At least one per domain " Redirects clients to TE
" -Clients need no knowledge of topology
" -Brokers are responsible for any load-balancing
20
Transaction Engines " Peer-to-peer " In-memory " Multi-Version Concurrency Control " Asynchronous messaging (replication) " Atoms
21
Storage Managers
" Persistence points for atoms " Key-value backing stores " -Local FS, S3, HDFS
" Independent archives
22
And Now, Some Unnatural Acts
23
Sharding, an Unnatural Act " The ideal DB application
DB
client
client
Scales up to the capacity of a single node
What if you need more read and/or write throughput?
24
Sharding
" Shard the DB among several nodes
DB0 DB1
Client1 Client2
New Client Layer Now you need to implement consistency in your applicaEon
TransacEonal consistency is very very hard to get right
25
Other Sharding Bugbears
" Global operations (searches, scans) " -Doing joins in the application " -Implementing Cursors " -Chunking and memory
management " And, of course, adding or removing
shards
26
Scaling Shards
" A recipe for changing the number of shards
" 1) Ask boss for permission " 2) Provision hardware " 3) Rewrite the app over 6 months " 4) Hope / Pray
27
NuoDB Solution
Host A
Broker
Host B
Agent
Host C
Agent
Domain
Management Client
28
NuoDB on a single node
Host A
Broker
Txn Engine
Host B
Agent
Storage Manager
Host C
Agent
Domain
Management Client
Database A
Client
29
NuoDB Scaling out
Host A
Broker
Txn Engine
Host B
Agent
Storage Manager
Host C
Agent
Domain
Management Client
Database A
Client Client Client
Txn Engine
Storage Manager
30
NuoDB ‘Adding a Shard’
Host A
Broker
Txn Engine
Host B
Agent
Storage Manager
Host C
Agent
Domain
Management Client
Database A
Client Client Client
Txn Engine
Storage Manager
Txn Engine
Storage Manager
31
NuoDB ‘Sharding’
" Literally as simple as just adding nodes
" No client code had to be harmed in the making of this distributed database
" -Brokers hide topology changes " -NuoDB is transactionally consistent
32
Eventual Consistency
" Eventual consistency is latent inconsistency
" -Not transactionally consistent " -Application porting is non-trivial " -Performance/Correctness tradeoff
icky
33
NuoDB Consistency
" NuoDB is transactionally consistent " All the time, everywhere " When a transaction is committed,
it’s guaranteed consistent " Tradeoff is between Performance and
Availability
34
Multi-Tenancy
" Traditional databases monopolize a node
" NuoDB supports many databases in a single pool of machines (a domain)
" Each DB can be scaled as needed, independantly
35
Multi-Tenancy Example
Host A
Broker
Txn Engine
Host B
Agent
Storage Manager
Host C
Agent
Domain
Management Client
Database A
JDBC Client
36
Multi-Tenancy Example
Host A
Broker
Txn Engine
Host B
Agent
Storage Manager
Host C
Agent
Domain
Management Client
Database A
JDBC Client
Txn Engine
JDBC Client
37
Multi-Tenancy Example
Host A
Broker
Txn Engine
Host B
Agent
Storage Manager
Host C
Agent
Domain
Management Client
Database A
JDBC Client
Txn Engine
JDBC Client
Storage Manager
Txn Engine
Database 1
38
Multi-Tenancy Example
Host A
Broker
Txn Engine
Host B
Agent
Storage Manager
Host C
Agent
Domain
Management Client
Database A
JDBC Client
Txn Engine
JDBC Client
Storage Manager
Txn Engine
Database 1
SQL Client ...
And now, a demo…
The Elastically Scalable Database™
40