From Startup to Enterprise

A Story of MySQL Evolution

Vidur Apparao, CTOStephen O’Sullivan, Manager of Data and Grid TechnologiesApril 2009

Data Size Growth

2009 LiveOps, Inc. 2

The data volume for a majority of companies increases 50-100% every year. (IDC)

Common Solutions: Rely on Moore’s Law Spend more money

But there are other ways…

About LiveOps

Technology Platform for Contact Centers On-Demand, Multi-tenanted Contact Center Platform Virtual Call Center of 20,000 independent home agents

Eight years of continuous growth Founded in 2000 Profitable since 2006 300 employees

2009 LiveOps, Inc. 3

LiveOps’ Data

Main data classes: Configuration data – low GBs, slow

change Logging data – low TBs, ever increasing System state – MBs, rapid change Customer-specific data – high GBs,

versioned

Largest table has 1.4 billion rows

Tenant key as a column on all tables

Multi-site deployment for high availability

2009 LiveOps, Inc. 4

Configuration Data Transaction data Session data

Configuration Tools Reporting Tools Monitoring Tools

Web Applications Telephony Applications

Phase 1: The Basic Model

Application servers connecting to a single DB

Replication to a slave for backup & load balancing

2009 LiveOps, Inc. 5

Web & Telephony Applications

R/W Master

MySQLReplication

Slave/Backup

Primary Drivers for Change

AvailabilityPerformanceScale

Options for Improving (Write) Scale

Sharding Partition data into distinct databases based on a sharding key

Functional Segmentation Separate functional data classes into distinct databases

MySQL Partitioning

LiveOps choice: Sharding, Functional Segmentation

2009 LiveOps, Inc. 7

Options for Improving (Query) Performance

Replication & Load balancing Distribute query load across multiple replicants

Separation of DB roles Separate fast from slow, OLTP from OLAP

Caching Reduce dependency on the database for queries

Consistent query tuning/optimization

LiveOps choice: Load balancing, separation of roles

2009 LiveOps, Inc. 8

Options for Improving Availability

Application resilience Remove requirement of direct write access and/or degrade

gracefully

MySQL Cluster

Multi-master replication Split tables or databases between ring replicating masters

DRBD or SAN HA

LiveOps choice: Application resilience, multi-master

2009 LiveOps, Inc. 9

2. Functional segmentation of data between multiple masters.

6. Separation of DB roles based on type and cost of query.

4. Replication to a farm of read-only replicants within and across data centers.

5. Load balancing using DB monitoring and pushed configuration files.

3. Multi-master replication and quick recovery processes on master failure.

1. Data writers use store-and-forward pattern for fault tolerance.

Phase 2: A Pure MySQL Solution

2009 LiveOps, Inc. 10

Reporting and Analytics

We

b &

Te

lep

ho

ny

Ap

plic

atio

ns

R/W Masters

Read-only Replicantsw/ Roles

Queries

Queries

Writes

Writes

DB Monitor/Load Balancer

Monitoring

Config. Push

Logging

Config + Session

All of these techniques still don’t get us to horizontal data scalability

Horizontal Scalability Options

Distributed Storage Systems DFS for unstructured file storage BigTable/HBase for structured data storage Various vendors with distributed RDBMSs

Grid Processing MapReduce and Hadoop

2009 LiveOps, Inc. 11

Our Approach

Take logging data out of the transactional databases

Reduce replication load

Store logging data as text files in a DFS

Use MapReduce for ETL into OLAP databases

Leverage open source tools like ActiveMQ and Hadoop

2009 LiveOps, Inc. 12

6. MySQL as a data mart.

Phase 3: MySQL w/ Horizontal Scalability

2009 LiveOps, Inc. 13

BrokerActiveMQBrokers

RepositoryProcess

MapReduce

DFS

Hadoop

Backup Storage Array

Reporting and Analytics

R/W MastersRead-only Replicants

We

b &

Te

lep

ho

ny

Ap

plic

atio

ns

Data Marts

AuditProcess

1. MySQL continues as OLTP solution.

2. Logging data now written to log files on local disk.

3. Log files moved via ActiveMQ to a log repository and DFS.

5. Hadoop as MapReduce system for ETL.

Horizontal scalability is now in reach!

4. Audit process to reconcile data between log files and DFS.

Learned Best Practices

Know your data

Build and enforce a data access layer

Put your data only where you need it

Experiment early and often

2009 LiveOps, Inc. 14

Conclusions

MySQL, other open source technology, and commodity hardware can be used to build a horizontally scalable data solution

Companies today are left to chart their own evolutionary paths

Collaboration and communication between companies in this area can help everyone

2009 LiveOps, Inc. 15

Thank You.Thank You.

Vidur Apparao,

vidur@liveops.com

Stephen O’Sullivansosullivan@liveops.com