Managing Big Data with Percona Server, XtraBackup, and Tungsten
-
Upload
oleksiy-kovyrin -
Category
Documents
-
view
625 -
download
4
description
Transcript of Managing Big Data with Percona Server, XtraBackup, and Tungsten
Managing Big Data withManaging Big Data withPercona Percona Server,Server,XtraBackupXtraBackup, and Tungsten, and Tungsten
Robert Hodges CEO, Continuent,Vadim Tkachenko, Founder & CTO, Percona
© Copyright 2011 Percona and Continuent
AgendaAgenda
/ Introductions/ What Is The Big Data Problem?/ Percona Server/ XtraBackup/ Tungsten/ Commercial Services and Support
© Copyright 2011 Percona and Continuent
About PerconaAbout Percona
/ Percona is the oldest and largest independent MySQLsupport and consulting company
// Our products:Our products:•• Percona ServerPercona Server•• Percona XtraBackupPercona XtraBackup
// We provide consulting, support, training andWe provide consulting, support, training anddevelopment for MySQL and LAMPdevelopment for MySQL and LAMP
© Copyright 2011 Percona and Continuent
About ContinuentAbout Continuent
/ Continuent is the leading provider of data replicationand clustering for open source relational databases
/ Our Products:• Tungsten Replicator - High-performance, MySQL replication• Tungsten Enterprise - Commercial replication and data
management solution for MySQL and PostgreSQL
/ Our Services:• Consulting on Tungsten plus replication and clustering in general• Subscriptions for commercial products
© Copyright 2011 Percona and Continuent
What Is the Big DataProblem?
© Copyright 2011 Percona and Continuent
The Big Data Problem
/ Large quantities of data• Application performance• Slow server starts
/ High transaction loads• Resource contention• SQL processing overhead
/ Distributed data• Datasets replicated across servers and sites• Master failover and slave promotion• Slave lag
/ Continuous, 24x7 Operation• Hot backup and restore• System upgrades
© Copyright 2011 Percona and Continuent
Percona Server
© Copyright 2011 Percona and Continuent
Percona Server
/ Drop-in replacement for MySQL/ High performance/ Diagnostics/ Operations / data management/ NOSQL
© Copyright 2011 Percona and Continuent
High Performance
/ Tuned for multi-core multi-disks systems/ SSD / Flash optimizations
© Copyright 2011 Percona and Continuent
Diagnostic
/ Advanced statistic on query execution/ # User@Host: mailboxer[mailboxer] @ [192.168.10.165]
/ # Thread_id: 11167745 Schema: board
/ # QC_Hit: No Full_scan: No Full_join: No Tmp_table: Yes Disk_tmp_table: No
/ # Filesort: Yes Disk_filesort: No Merge_passes: 0
/ # Query_time: 0.000659 Lock_time: 0.000070 Rows_sent: 0 Rows_examined: 30 Rows_affected: 0 Rows_read: 30
/ # InnoDB_IO_r_ops: 1 InnoDB_IO_r_bytes: 16384 InnoDB_IO_r_wait: 0.028487
/ # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000
/ # InnoDB_pages_distinct: 5 select count(distinct author_id) from art87.article87 force index (forum_id) whereforum_id = 240215 and thread_id = '710575'
/ Additional INFORMATION_SCHEMA tablesCLIENT_STATISTICSINDEX_STATISTICSINNODB_BUFFER_POOL_PAGESINNODB_BUFFER_POOL_PAGES_BLOBINNODB_BUFFER_POOL_PAGES_INDEXINNODB_BUFFER_POOL_CONTENTINNODB_INDEX_STATSINNODB_IO_PATTERNINNODB_RSEGINNODB_SYS_INDEXESINNODB_SYS_TABLESINNODB_TABLE_STATSPROCESSLISTTABLE_STATISTICSUSER_STATISTICS
© Copyright 2011 Percona and Continuent
Operations / management
/ Export / Import InnoDB tables• Move shard from one server to another• Try copy 100GB user1.table1 from Server1 to Server2
/ Fast warmup• Periodic dump and load at start content of buffer pool• Very important for big (memory, data) slaves
/ Integration with FlashCache• Hints for queries to ignore caching• FlashCache is Flash/SSD cache layer over regular HDD/RAID
© Copyright 2011 Percona and Continuent
NOSQL
/ HandlerSocket plugin• Access to data without SQL queries / overhead• Wide range of operations on PK/ index
• Batch, range ( < , >, <= X <= ) access
/ Outstanding performance• Reported 750,000 request/sec
© Copyright 2011 Percona and Continuent
Percona Server plans
/ Cloud support• Scalability + performance
/ Flash/SSD• Continue research and improve performance
/ Diagnostic• You never can have enough statistics
© Copyright 2011 Percona and Continuent
Introducing PerconaXtraBackup
© Copyright 2011 Percona and Continuent
Percona XtraBackup
/ OpenSource hot-backup for MySQL/InnoDB/ Free
© Copyright 2011 Percona and Continuent
Designed with big data in mind
/ Non-blocking for InnoDB tables/ Streaming directly to backup server / host/ Incremental ( differential ) backups/ Partial backups/ Parallel copy/ Minimal actions to setup slave
© Copyright 2011 Percona and Continuent
Non-Blocking
/ Copy InnoDB files in background/ Server continues to process workload
© Copyright 2011 Percona and Continuent
Streaming
/ Tar-compatible file• innobackupex-1.5.1 --stream=tar ./ > backup.tar
/ Compress• innobackupex-1.5.1 --stream=tar ./ | gzip - > backup.tar.gz
/ Copy to remote host• innobackupex-1.5.1 --stream=tar ./ | ssh vadim@desthost "cat - > /data/vol1/mysqluc/backup.tar“
© Copyright 2011 Percona and Continuent
Incremental/Differential
/ Differential• Copy ONLY changes since last full backup
/ Incremental• Copy ONLY changes since last incremental / differential
© Copyright 2011 Percona and Continuent
Partial Backup
/ Copy only selected databases / tables• Databases:
user1user2…user1000
• You want to copy only user567 database from server1 to server2• Create new database or import into existing
© Copyright 2011 Percona and Continuent
Setup slave
/ innobackupex-1.5.1 --stream=tar /tmp/ --slave-info | sshuser@DESTSERVER "tar xfi - -C /DESTDIR“
/ innobackupex-1.5.1 --apply-log --use-memory=2G/DESTDIR
© Copyright 2011 Percona and Continuent
Percona XtraBackup plans
/ Cloud/S3• Copy / restore data from cloud storage
/ Compression on table level/ Improve incremental and partial backup operations
© Copyright 2011 Percona and Continuent© Continuent 2011© Continuent 2010
Introducing Tungsten
© Copyright 2011 Percona and Continuent
What Is Tungsten?
/ Tungsten Replicator is a fast, open-sourcereplication engine for MySQL
/ Tungsten Enterprise is a complete datamanagement solution based on replication1. Ensure reliable, fast access to data2. Simplify common management tasks3. Increase performance and resource utilization…Without changing applications
© Copyright 2011 Percona and Continuent
Tungsten Replicator ArchitectureTungsten Replicator Architecture
MySQLMySQLTungstenReplicator(replicator)
MasterMaster
SlaveSlave
Binlogs TransactionHistory Log
(Transactions + Metadata)
replicator.properties
(Replicator configuration)
MySQLMySQL
TungstenReplicator(replicator)
TransactionHistory Log
replicator.properties
Tail binlog orlogin as client
Transport viaTCP/IPconnection
Apply using JDBC
© Copyright 2011 Percona and Continuent
Tungsten Replicator: So Many Features
/ Global transaction IDs/ Transaction filters/ Replicate from MySQL to PostgreSQL & Oracle/ Time-delay replication/ Backup and restore integration/ Cross-version replication: 5.5 -> 5.1 -> 5.0 -> 4.1/ Row and statement replication/ Automatic consistency checks/ Auto-provisioning/ Multi-master replication/ Parallel replication
© Copyright 2011 Percona and Continuent
Tungsten Replicator Pipelines (Slave)
THLTHLSlaveSlaveDBMSDBMS
ApplierApplierExtractorExtractor FiltersFilters ApplierApplierExtractorExtractor FiltersFilters
StageStage StageStagePipelinePipeline
Tungsten Replicator ProcessTungsten Replicator Process
Transaction History LogTransaction History Log
Master THLMaster THL(via network)(via network)
© Copyright 2011 Percona and Continuent
Apache/Mod_PHPlibmysqlclient.alibmysqlclient.a
Tungsten Enterprise ArchitectureTungsten Enterprise Architecture
Apache/Mod_PHP
DBMSDBMS
Replicator
DBMSDBMS
Replicator
DBMSDBMS
Replicator
libmysqlclient.alibmysqlclient.a
MasterMasterSlaveSlave SlaveSlave
ConnectorConnector
Manager ManagerManager
© Copyright 2011 Percona and Continuent
Tungsten Enterprise Management
Broadcast commands Broadcast commands and monitoring dataand monitoring data
BusinessBusinessRulesRules
Local DBMS ResourcesLocal DBMS Resources
Local DBMS Local DBMS ResourcesResources
Manager(Coordinator)
Manager
ManagerCctrl Client
Cctrl Client
Group Group CommunicationsCommunications
Cctrl Client
Local DBMS Local DBMS ResourcesResources
© Copyright 2011 Percona and Continuent
Tungsten Replicator vs. Enterprise
Feature Replicator EnterpriseSimple admin with trepctl X X
(MySQL) Row/Statement 5.0/5.1/5.5 X XMulti-master replication X XParallel apply (2.0) X XReplication to other DBMS types X XDistributed management XAutomated failover and recovery XZero-downtime upgrade XSQL routing and load-balancing XFloating IP support X
© Copyright 2011 Percona and Continuent© Continuent 2011© Continuent 2010
Moving Data:Tungsten Replicator
© Copyright 2011 Percona and Continuent
Big Data and Multi-Master Replication
/ Big data is spread across multiple servers and sites/ Many-to-one: consolidate updates from multiple
servers into one slave• Roll-up reporting from remote servers
/ Bi-directional: Updates from two masters propagateto each other
• High-volume merchant apps and distributed directories
/ Cross-site/Cross-Host: Processing distributed overmultiple hosts and sites
• Sharded, multi-tenant datasets with centralized data
© Copyright 2011 Percona and Continuent
Multi-Master Data Consolidation
MySQLMySQLTungstenReplicator
Master PipelineMaster Pipeline
MySQLMySQL
TungstenReplicator
Slave PipelineSlave Pipeline
MySQLMySQLTungstenReplicator
TungstenReplicator
MySQLMySQL TungstenReplicator
Direct Pipeline, Direct Pipeline, Multiple ServicesMultiple Services
MySQLMySQL
MySQLMySQL
master1
master2
© Copyright 2011 Percona and Continuent
Bi-Directional ReplicationBi-Directional Replication
nyc1nyc1
Site NYC Site NYC
nyc2nyc2
sjc1sjc1
sjc2sjc2
Replicator
njc-localnyc-remote
Replicatorsjc-local
Replicator
nyc-localsjc-remote
Replicatornyc-local
Site SJC Site SJC
Cool feature: Tungsten slaves rememberpositions on remote and local masters
© Copyright 2011 Percona and Continuent
Big Data and Failover
/ Slave promotion is a severe problem for high-volumeapplications with multiple slaves
/ Tungsten global transaction IDs eliminate binloghacks to reconfigure slaves
/ Tungsten epoch numbers prevent re-connectionwhen logs diverge after failover
/ Say goodbye to binlog hacks for failover!
© Copyright 2011 Percona and Continuent
MasterMaster
Replicator
Seqno
Epoch Transaction
1 1 UPDATE CUST..2 1 INSERT INTO
CALL_TAB…3 3 DELETE FROM
CALL_TAB…4 3 UPDATE CUST…
SlaveSlave
Replicator
Replicator Log
Transfer log from master
BackupBackupseqno=3seqno=3
Global ID and Epoch Number Concepts
Seqno
Epoch Transaction
1 1 UPDATE CUST..2 1 INSERT INTO
CALL_TAB…3 3 DELETE FROM
CALL_TAB…4 3 UPDATE CUST…5 3 UPDATE CALL_TAB…
Replicator LogBackup reloadable Backup reloadable
on any serveron any server
Epoch numbers preventEpoch numbers preventinconsistent slaves frominconsistent slaves from
connectingconnecting
© Copyright 2011 Percona and Continuent
MasterMaster
ReplicatorManager
SlaveSlave(Relay)(Relay)
ReplicatorManager
SlaveSlave
ReplicatorManager
SlaveSlave
ReplicatorManager
Floa
ting
IP
Main Cluster
BackupBackupBackupBackupBackupsBackups
BackupBackupBackupBackupBackupsBackups
DR Cluster
DR Site Enabling with Global IDs
Seqno 25Seqno 25Epoch 1Epoch 1
Seqno 21Seqno 21Epoch 1Epoch 1Seqno 28Seqno 28
Epoch 1Epoch 1
Seqno 35Seqno 35Epoch 1Epoch 1
© Copyright 2011 Percona and Continuent
Old Old MasterMaster
ReplicatorManager
SlaveSlave(Relay)(Relay)
ReplicatorManager
New New MasterMaster
ReplicatorManager
SlaveSlave
ReplicatorManager
Floa
ting
IP
Main Cluster
BackupBackupBackupBackupBackupsBackups
BackupBackupBackupBackupBackupsBackups
DR Cluster
Seqno 25Seqno 25Epoch 1Epoch 1
Seqno 21Seqno 21Epoch 1Epoch 1
Seqno 29Seqno 29Epoch 29Epoch 29
Seqno 35Seqno 35Epoch 1Epoch 1(epoch number(epoch numberpreventspreventsreconnect asreconnect asslave)slave)
Reconfiguration After Failover
© Copyright 2011 Percona and Continuent
/ Single-threaded MySQL replication suffers fromimpedance mismatch between CPU and I/O for high-volume data replication
/ Tungsten optimizes resource utilization for large-scale data transfer
/ Pipeline parallelism - Spread work across stages/ Service parallelism -- Spread work across different
replicators and replication services within replicators/ Parallel apply -- Spread apply operations across
multiple threads
Big Data and Resource Utilization
© Copyright 2011 Percona and Continuent
Parallel Slave Apply -- Internals View
THLTHL SlaveSlaveDBMSDBMS
ApplierApplierExtractorExtractor ApplierApplierExtractorExtractor
PipelinePipeline
Tungsten Replicator ProcessTungsten Replicator Process
Transaction Transaction History LogHistory Log
Master THLMaster THL(via network)(via network)
StageStageApplierApplierExtractorExtractor
StageStage StageStage
ApplierApplierExtractorExtractorApplierApplierExtractorExtractor
ParallelQueue
shard.listshard.listfilefile
© Copyright 2011 Percona and Continuent
db1 db2 db3db1 db2 db3db1 db2 db3db1 db2 db3
Replicator
Demo: Parallel Apply with TPC-B Tests
TPC-BTPC-BTestTest
TPC-BTPC-BTestTest
TPC-BTPC-BTestTest
Replicator
MasterSlave
© Copyright 2011 Percona and Continuent
Future Plans
Tungsten Replicator will go fullyGPL V2 in 2011!
Initial Replicator-Only Buildsavailable in February
Keep Watchingwww.continuent.com
© Copyright 2011 Percona and Continuent
Commercial Services andSupport
© Copyright 2011 Percona and Continuent
Continuent Product Subscriptions
/ Per-server subscriptions for Tungsten Enterprise;pricing based on number of DBMS servers
/ Software-only -- Software license; use consulting forsupport; bug-fixes are free
/ Software + Support -- Software license + 24x7support (SLAs negotiable)
© Copyright 2011 Percona and Continuent
Continuent ConsultingContinuent Consulting
/ Practical consulting by industry experts on anyaspect of Tungsten or replication/clustering for opensource databases
/ Hourly support contracts• Advice, troubleshooting, bug fixes
/ Projects• Cluster design, roll-outs, upgrades, special replication problems
/ Sponsored development• Implementation of features large and small
/ Offered for both open source and commercialproducts
© Copyright 2011 Percona and Continuent
Percona Software and Services
/ Consulting• Our specialties include MySQL architecture and design,
diagnosing and solving performance problems, recoveringcorrupted data, setting up high availability, clustering, and muchmore.
/ Support & Maintenance• Expert technical support for MySQL• Insurance against bugs in MySQL• Customized MySQL server binaries
/ Training• Public and custom onsite
/ Software• Percona Server with XtraDB 5.1 and 5.5 coming• Percona XtraBackup
© Copyright 2011 Percona and Continuent
Conclusion
/ Big data has a unique set of challenges for MySQLdeployments
• Large volumes• High transaction rates• Distribution across servers and sites• Constant availability
/ Percona Server, XtraBackup, and Tungsten addressbig data problems
/ Combine Percona and Continuent for your nextproject!
© Copyright 2011 Percona and Continuent
Continuent, Inc.560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1-866-998-3642 Fax +1-408-668-1009e-mail: [email protected]
Percona, Inc.+1-888-316-9775 or+1-208-473-2904 (USA)+44-208-133-0309 (UK)http://www.percona.com/contact/sales/
Contact InformationContact Information
Continuent Web Site:http://www.continuent.com
Percona Web Site:http://www.percona.com