MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch...
Transcript of MariaDB for Advanced DBAs MariaDB Introduction MariaDB Replication GTID Complex Scenarios Semi-Synch...
MariaDB for Advanced DBAsMariaDB Training
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Introduction
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Introducing MariaDB AbFounders from MySQL — the
Company and the Community Funded by Founders, Employees,
and Venture Capital Several former MySQL Employees
and Community Members Located in over 14 Countries
3
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Personal IntroductionsInstructor
Name and Background
Participants Name and Company MariaDB Experience How You Use MariaDB Needs Related to Course Topics
4
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Class Schedule & Personal ConcernsStarting and Ending Times Planned Breaks On- Site
Location of Rest Rooms Smoking Areas Snacks and Drinks
5
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Course OutlineHigh Availability
MariaDB Replication & GTID
Complex Scenarios & Semi-Sync
MariaDB Replication Manager
MariaDB Enterprise Cluster
Client Proxies
Optimizing Optimizing Overview
Schema Tuning
InnoDB In-Depth
Collecting Information
Benchmarking
Query Tuning The Query Process
Query Tuning
Advanced Index Usage
Diagnosing Systems Common Bottlenecks
Troubleshooting
Hacking with GNU Debugger
6
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
High Availability Overview
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
MariaDB Replication
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Purpose of MariaDB Replication
9
Load Balancing — High-Traffic Reads on Slaves High Availability
Fail Over — Promote a Slave to Master Back-Ups — Take a Slave Off-Line
Minimize Downtime for Upgrades or Schema Changes Apply Changes to a Slave Promote Slave to Master
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Replication Terrain
10
mysqld Data Storage
Slave 1
Relay Log
Data Storage
Slave 2/Master 2
Relay Log
Binary Log
mysqldSlave 2A
Slave 2B
Slave 2C
INSERT UPDATE DELETE
CREATE ALTER DROP
mysqld Data Storage
Master
Binary Log Dump Thread
Client Threads
SQL Thread
IO Thread
IO Thread
IO Thread
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
MariaDB Replication Factors
11
One Master, Multiple Slaves No True Multi-Master Solution, but Circular
Replication
Close to Real Time (Asynchronous) Semi-Synchronous Replication Mode Crash-Safe Slaves with InnoDB
Slaves may also be Masters (log_slave_updates) Replication Filtering Rules Storage Engine Changes Optional on Intermediate
Slaves
Documentation on log_slave_updates: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/#log_slave_updates
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Replication Threads
12
Master binlog Dump Thread Pushes Binary Log Events to Slave
Slave IO Thread
Gets binlog Events from Master Writes to Local Relay Log
Slave SQL Thread
Reads Relay Log & Executes Checks Result Codes Match Master
Slave Multiple Execution Threads
Separates Events Based on Database Updates Applied in Parallel
SHOW SLAVE STATUS \G
... Slave_IO_Running: Yes Slave_SQL_Running: Yes ...
SHOW PROCESSLIST \G ***** 1. row ***** Id: 615 User: mariadb_replicator
Host: 12.34.56.01:42422 db: NULL Command: Binlog Dump Time: 798022
State: Master has sent all binlog to slave; waiting for binlog to be updated
Executed on Master
Executed on Slave
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Parallel Replication
13
Documentation on Parallel Replication: https://mariadb.com/kb/en/mariadb/documentation/replication/standard-replication/parallel-replication/
Replication Process on Slaves Events from Master through IO Thread and Queued in
Relay Log Each Relay Log Entry is Retrieved by SQL Thread Each Transaction is Applied to Slave
Non-Parallel Systems Apply Transactions Sequentially through SQL Thread
Parallel Systems Apply Transactions in Pool of Separate Replication Worker Threads
Smaller Transactions are Required
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Troubleshooting Problems with Replication
14
Check Slave Error Log Look for Disconnects from Network Binary or Relay Log Event Corruption will cause Slave
SQL Thread to Stop Different Query Error Codes on Slave indicates Not
Synchronized
Percona Toolkit can Help with Troubleshooting May Need to Rebuild Slave with New Snapshot (Back-up
of Master or Another Slave)
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Slave Filtering Rules — Database Level
15
Documentation on Slave Options: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/
Exclude Specific Databases (e.g., mysql)
Include Specific Databases
Excluding can Cause Problems with Joins
SET GLOBAL replicate_ignore_db = ''; SET GLOBAL replicate_do_db = 'sales,inventory';
SET GLOBAL replicate_ignore_db = 'mysql';
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Slave Filtering Rules — Table Level
16
Documentation on Slave Options: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/
Ignore Specific Tables
SET GLOBAL replicate_ignore_db = (employees), replicate_do_table = (employees.names, employees.contacts);
SET GLOBAL replicate_ignore_table = (employees.salary);
SET GLOBAL
replicate_ignore_db = (sales), replicate_do_table = (sales.europe_%), replicate_ignore_table = (sales.europe_uk_%);
Include Specific Tables
Wildcards for Multiple Tables
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Global Transaction Identification
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
GTID (Global Transaction Identification)
Unique Identification for Binary Log Transactions
Used to Eliminate Confusion amongst Replication Servers
18
Documentation on GTID: https://mariadb.com/kb/en/mariadb/gtid/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Confusion Without GTIDReplication Process
Changes on Master are Recorded in Binary Log as Events Slaves get Events from Master and Execute Locally Slaves Use Master's Binary Log File Name and Position
Numbers for Clarity
Potential Source of Confusion Slaves may be Masters to Other Slaves (a.k.a., Intermediate
Masters) Intermediate Masters have Different Log Positions Promoting an Intermediate Master to Master is
Cumbersome
19
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Clarity from GTIDUnique Identifier for Each binlog Event Group
Transactions are Collections of Events to Apply Together Can Include Non-Transactional Statements
Promoting a Slave to Master is Easy State of Slave is Recorded in Crash-Safe Method
20
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
GTID FormatThree Numbers Separated by Dashes (e.g., 0-1-8000)
Domain ID (32-bit Unsigned Integer) Server ID (32-bit Unsigned Integer) Sequence Number (64-bit Unsigned Integer)
Domain ID Indicates Individual Write Stream Server ID is Set with server_id Variable — Must be Unique
(1 to 2 32) Sequence is Incremented with Each Transaction
Stored in gtid_seq_no
21
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Effect of the GTID DomainBinlog Consists of Different Streams —
Not a Single Ordered Stream Each Identified by its Domain ID GTIDs always in Same Order
Different Streams can be Interleaved Slaves use Last GTID Applied for
Domain Slaves Start from a Different Point in
binlog for Each Domain when there's a New Master
22
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Enabling GTID on MasterSet gtid_domain_id on Master & Restart
Execute SHOW MASTER STATUS to get Current binlog File Name and Position
Use BINLOG_GTID_POS() to Get the binlog GTID Position Equivalent
23
Article on Enabling GTID for Replication: https://mariadb.com/resources/blog/enabling-gtids-server-replication-mariadb-100
[mysqld] gtid-domain-id=1 SHOW MASTER STATUS\G
****** 1. row ****** File: mariadb-bin.000001 Position: 510 ...
SELECT BINLOG_GTID_POS('mariadb-bin.000001', 510) AS binlog_gtid_pos;
+-----------------+ | binlog_gtid_pos | +-----------------+ | 1-101-1 |
+-----------------+
Tell the slave this value.
Excerpt from master's configuration file.
Executed on master.
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Informing Slave of GTIDSet gtid_slave_pos on Slave to
binlog GTID Position from Master
Use CHANGE MASTER to set master_use_gtid to slave_pos
Execute SHOW SLAVE STATUS to check if GTID is in Use
24
STOP SLAVE;
SET GLOBAL gtid_slave_pos = '1-101-1';
CHANGE MASTER
TO master_use_gtid=slave_pos;
START SLAVE;
SHOW SLAVE STATUS \G ... Using_Gtid: Slave_Pos
Gtid_IO_Pos: 1-101-1 ...
Executed on slave.
Executed on slave.
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
GTID VariablesSeveral Positions are Stored Use SHOW VARIABLES to Find Values
25
SHOW VARIABLES
WHERE Variable_Name LIKE 'gtid%' OR Variable_Name LIKE 'last_gtid';
+------------------------+-----------+ | Variable_name | Value |
+------------------------+-----------+ | gtid_binlog_pos | 0-1-96217 | | gtid_binlog_state | 0-1-96217 | | gtid_current_pos | 0-1-96217 |
| gtid_domain_id | 0 | | gtid_ignore_duplicates | OFF | | gtid_seq_no | 0 | | gtid_slave_pos | | | gtid_strict_mode | OFF |
| last_gtid | | +------------------------+-----------+Documentation on GTID Variables: https://mariadb.com/kb/en/mariadb/gtid/#system-variables-for-global-transaction-id
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Showing EventsUse the SHOW BINLOG EVENTS
Statement to see a list of Binary Log Events
Several Event Types Format_desc Gtid_list Binlog_checkpoint Gtid Query
26
SHOW BINLOG EVENTS IN 'ip-12-34-56-01-bin.000056' LIMIT 1 \G
*********** 1. row *********** Log_name: ip-12-34-56-01-bin.000005
Pos: 4 Event_type: Format_desc Server_id: 1 End_log_pos: 249 Info: Server ver: 10.1.23-MariaDB,
Binlog ver: 4
Documentation on SHOW BINLOG EVENTS: https://mariadb.com/kb/en/mariadb/show-binlog-events/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
GTID Strict ModeHelps keep binlogs Identical across
Multiple Servers Slave Stops when Encountering a Problem
Won't Allow Slave to use Lower Sequence for a Domain — SQL Thread Stops
Rejects Attempts to Set SESSION.gtid_seq_no to Lower Sequence
Stops Slave from Connecting with a GTID missing in Master's binlog
Controlled by Setting Strict Mode on Slave
27
Documentation on gtid_strict_mode: https://mariadb.com/kb/en/mariadb/gtid/#gtid_strict_mode
SET gtid_strict_mode = 'ON';
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
GTID Ignores DuplicatesMulti-Source Masters may Process
Event Groups with same GTID Only One Applied — Others are Ignored
Sequence Number of Domain Used to Determine if Duplicate
Need to Ensure GTID Sequence are Increased Strictly
28
Documentation on gtid_ignore_duplicates: https://mariadb.com/kb/en/mariadb/gtid/#gtid_ignore_duplicates
SET gtid_ignore_duplicates = ON;
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Switch Slave to GTIDSlaves can Use GTID or Binary Log File
Name with Position ID Use CHANGE MASTER Statement to Switch
to GTID — Give Value of NO to Disable
29
CHANGE MASTER TO master_use_gtid=slave_pos
Documentation on CHANGE MASTER Statement: https://mariadb.com/kb/en/mariadb/change-master-to/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Switching PlacesInitial Scenario
Server 1 is Master; Server 2 is Slave Server 1 goes Down Promote Server 2 to Master
New Scenario Server 1 is Revived and Made a Slave
Doesn't have Prior GTIDs since it was Never a Slave The gtid_slave_pos is Empty
Use Instead current_pos with CHANGE MASTER
30
mysqldData
Storage
Server 1
Binary Log Dump Thread
Data Storage
Server 2
Relay Log
Binary Log
mysqld
CHANGE MASTER
TO master_use_gtid=current_pos
Executed on Server 1
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Complex Scenarios
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Non-Typical Replication MethodsMariaDB Replication can use Non-
Traditional Topology Circular Replication Multi-Source Replication
32
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Circular ReplicationClients can Write to any Server — All are
Masters and Accept Writes
Servers can Replicate Concurrent Changes to Each Other (e.g., Two Changes to Same Row)
No Conflict Resolution
33
server-id100
server-id200
server-id300
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Multi-Source ReplicationSlaves are Used Typically for Read Load
Balancing Writes can also be Load Balanced with
Multi-Source Replication Don't Confuse with Multi-Master
Replication (i.e., Circular or Point-to-Point Replication)
34
Documentation on Multi-Source Replication: https://mariadb.com/kb/en/mariadb/multi-source-replication/
server-id101
server-id103
server-id102Two Masters
One Slave
t1 t2
The default_master_connection can equal t1 or t2.
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Multi-Source PreparationUse an Alias with the CHANGE MASTER
Statement to Distinguish between Sources
Different Replication Filters may be Used for Differen Master Sources
35
CHANGE MASTER 'master1' TO
MASTER_HOST='35.161.145.71', MASTER_PORT=3306, MASTER_USER='maria_replicator', MASTER_PASSWORD='rover123';
START SLAVE 'master1';
Executed on slave.
Documentation on CHANGE MASTER with Alias: https://mariadb.com/kb/en/mariadb/change-master-to/#connection_name
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Multi-Source MonitoringUse SHOW SLAVE STATUS with Alias to
Monitor Slave in Relation to a Specific Master
Use ALL SLAVES to Show Slave Status for All Sources
36
SHOW SLAVE 'master1' STATUS;
SHOW ALL SLAVES STATUS;
Documentation on SHOW MASTER STATUS: https://mariadb.com/kb/en/show-master-status/ Documentation on SHOW SLAVE STATUS: https://mariadb.com/kb/en/mariadb/show-slave-status/
Executed on slave.
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Stopping Multi-Source ReplicationUse STOP SLAVE with Alias to Stop
Replicating a given Master Use ALL SLAVES Option to Stop
Replication Completely
37
STOP SLAVE 'master1';
STOP ALL SLAVES;
Executed on slave.
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
MariaDB Replication — AsynchronousMaster Doesn’t Wait for Slaves IO Thread may be Slow to Receive
binlog Packets Network Congestion or Disconnects
SQL Thread may be Slow in Processing Relay Log Events
Load on Slave or Network Problems
38
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Semi-Sync Plugin
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Semi-Synchronous
40
Documentation on Semi-Synchronous Replication: https://mariadb.com/kb/en/mariadb/semisynchronous-replication/
Master waits for a Slave to Acknowledge Receipt of Transaction
Waits for Slave to Write to Relay Log, Not to Execute — Slave SQL Thread may still Lag
One Slave Response needed for Master to Continue (i.e., Semi-Synchronous, Not Synchronous)
Can Significantly Affect Performance of Master
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Enable Semi-SynchronousMaster Switches to Asynchronous
if Not Acknowledged Semi-Synchronous is Resumed
When One Slave Synchronizes
41
SET GLOBAL rpl_semi_sync_master_enabled = ON;
SET GLOBAL rpl_semi_sync_slave_enabled = ON;
SET GLOBAL
rpl_semi_sync_master_timeout = 10000;
INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so';
INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';
Documentation on Semi-Sync Status Variables: https://mariadb.com/kb/en/mariadb/semisynchronous-replication-plugin-status-variables/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Binary Log FormatStatement Based (SBR) — Original Queries are Replicated
Least Data Sent over Wire and Tested for Years Non-Deterministic Statements Executed on Slave — Slave Load is Increased
Row Based (RBR) — Table Rows are Replicated
Only Non-Deterministic Statements Executed on Master — Slave Load is Reduced vs. SBR
More Data sent over Wire — Not Supported by All Engines Mixed (default) - Smart Switching between SBR and RBR
Checksum (--binlog-checksum) in Binary and Relay Logs to detect Errors
Includes Errors in Memory, Disk, Network and Database Can be Implemented for each Slave
42
Documentation on Binary Log Format: https://mariadb.com/kb/en/mariadb/binary-log-formats/
[mysqld]
binlog-format=MIXED
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Annotate RowsBinary Logging can Annotate Row
Events — Useful for Florensic Information Add the --binlog-annotate-row-events
Option — There's No Variable to use with SET
43
Documentation on https://mariadb.com/kb/en/mariadb/annotate_rows_log_event/
[mysqld] binlog-annotate-row-events
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Register New Semi-Synchronous SlaveSlave will Continue as Asynchronous
Unless Restarted Properly Restart Slave IO Thread to Register as
Semi-Synchronous
44
STOP SLAVE IO_THREAD;
START SLAVE IO_THREAD;
Executed on slave.
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Timeout for Semi-Synchronous ReplicationMaster Reverts to Asynchronous
when Time Exceeded for Acknowledgement
Default Time is 10,000 milliseconds
Status Variable rpl_semi_sync_master_status Set Automatically to OFF
45
SET rpl_semi_sync_master_timeout = ON;
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Debugging Semi-Sync ReplicationFour Tracing Levels for Semi-
Synchronous 1 — General Level (e.g., Time Function
Failures) 16 — Detailed, Verbose Level 32 — Net Wait Level (e.g., Information
about Network Waits) 64 — Function Level (e.g., Information
about Function Entries and Exits) Default Value is 32
46
SET rpl_semi_sync_master_trace_level = 32;
Documentation on trace levels: https://mariadb.com/kb/en/mariadb/semisynchronous-replication/#rpl_semi_sync_master_trace_level
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Waiting Related to Slave CountMaster Reverts to Asynchronous when No Slaves
(i.e., rpl_semi_sync_master_clients) Enable rpl_semi_sync_master_wait_no_slave to
Wait for Timeout
47
SET rpl_semi_sync_master_wait_no_slave = ON;
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Waiting for SynchronizationSet Parameters for Master to Wait for Acknowledgement
AFTER_SYNC — Transaction Written to Slave's binlog All Clients see Same Data at Same Time Failover is Lossless if Master Crashes
AFTER_COMMIT — After a Slave Acknowledges and Master Commits to Storage Engine
Other Clients may see Committed Transaction Clients may see Data Loss if Master Crashes
48
SET rpl_semi_sync_master_wait_point = AFTER_SYNC;
Documentation on rpl_semi_sync_master_wait_point: https://mariadb.com/kb/en/mariadb/semisynchronous-replication/#rpl_semi_sync_master_wait_point
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
MariaDB Replication Manager
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
MariaDB Replication ManagerAble to Handle Failovers of MariaDB
Replication with GTID Failover on a Dead Master Promote a Slave to Master
Arbitrator and Proxy Peacemaker Resource Use a Client Proxy to Route Queries to
New Hosts
50
Documentation on MariaDB Replication Manager: https://github.com/mariadb-corporation/replication-manager/blob/0.7/README.md
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
InstallationDownload a Binary Package from
Github Use yum or another Utility to
Install Binary
51
Download MariaDB Replication Manager: https://github.com/tanji/replication-manager/releases Article on Installing MariaDB Replication Manager: https://mariadb.com/resources/blog/mariadb-automatic-failover-maxscale-and-mariadb-replication-manager
wget https://github.com/tanji/replication-manager/.../xx.xx.version.rpm yum install xx.xx.version.rpm
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Switch Over ManuallyUse switchover Option to Switch
Servers Selects Most Suitable Node in --
hosts to be Leader — Decision based on which is Most Up-to-Date
Include --interactive for Interactive Mode to Confirm New Topology
52
replication-manager switchover \ —hosts=node1,node2,node3 \ —user=root:pass —rpluser=replicator:pass \
--interactive
Node A Node CNode B
MariaDBReplication Manager
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Procedural Command-Line ExampleNon-Interactive Failover Mode
Use root for Management Use repl for Replication Switchover Give Failover Script Set Maximum Slave Delay before Switchover
53
replication-manager failover \ --hosts=db1:3306,db2:3306,db2:3306 \
--user=root:pass --rpluser=repl:pass \ --pre-failover-script="/usr/local/bin/vipdown.sh" \ --post-failover-script="/usr/local/bin/vipup.sh" \ --verbose --maxdelay=15
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
MonitoringUse a Client Proxy to Monitor the Servers Add a Script Entry to Start Replication
Manager User Executing Script must have ALL
Privileges repluser must have at least REPLICATION
SLAVE Privilege
54
Download MariaDB Replication Manager: https://github.com/mariadb-corporation/replication-manager Donwload Go Programming Language: https://golang.org/dl/
[MySQL Monitor] type=monitor
module=mysqlmon servers=%%ENV:SERVERS_LIST%% user=root passwd=%%ENV:MYROOTPWD%% monitor_interval=1000
detect_stale_master=true
script=/usr/local/bin/replication-manager --user root:admin
--rpluser repluser:replpass --hosts $INITIATOR,$NODELIST --failover=force --interactive=false events=master_down
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Monitoring with Console Mode
55
replication-manager monitor / --hosts=db1:3306,db2:3306,db2:3306 / --user=root:pass --rpluser=repl:pas
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
HTTP ServerStart in Background to Monitor a
Cluster with HTTP Server Controlling Daemon
Accessible through Web Browser on http://localhost:10001 by Default
56
replication-manager monitor \ --hosts=db1:3306,db2:3306,db2:3306 \
--user=root:pass --rpluser=repl:pass \ --daemon --http-server
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
MariaDB Enterprise Cluster
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Advantages of MariaDB Enterprise ClusterParallel Slave Applying Practically No Slave Lag Instant Trivial Failover Automatic Node Provisioning Works Well in WAN
58
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
NuancesUses Only InnoDB Primary Keys are Necessary Commit Latency Transaction Size Limited to 2GB —
Unlimited in Future Versions
DEADLOCK on COMMIT
AUTO_INCREMENT Handled Differently
No Cluster-Wide Read Locks
59
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
MariaDB Replication — Server Centric StyleAsynchronous Replication No Conflict Detection Multi-Master Replication
If Node C Crashes, Does Cluster Survive?
If Node B Crashes and Clients Switch to C, How does Node B Rejoin?
Which Node has Data X? How do you Back-Up Cluster?
60
MasterServer
SlaveServer
Node A
Node C
Node B
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Galera Approach — Data Centric StyleData Doesn’t Belong to a Node Nodes Belong to Data Data is Synchronized among Two or
More Servers
61
Server 1 Server 2 Server 3 Server N
DataSet
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Galera ApproachGalera Nodes are Anonymous — All are
Equal Galera Cluster is One Large Distributed
Master A DataSet Needs an Identifier DataSet Identier is a Cluster Identifier
00295a79-9c48-11e2-bdf0-9a916cbb9294
62
DataSet Cluster
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Global Transaction Identifier (GTID)DataSet plus Sequence of Atomic
Changes equals GTID
63
DataSet00295a79-9c48-11e2-bdf0-9a916cbb9294:64201
00295a79-9c48-11e2-bdf0-9a916cbb9294:64201
64201
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Global Transaction Identifier (GTID)Initial DataSet
64
00295a79-9c48-11e2-bdf0-9a916cbb9294:0
00295a79-9c48-11e2-bdf0-9a916cbb9294:1
00000000-0000-0000-0000-000000000000:-1
First Change and Transaction
Undefined GTID
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Global Transaction Identifier (GTID)MySQL 5.6 GTID
65
00295a79-9c48-11e2-bdf0-9a916cbb9294:64201
8182213e-7c1e-11e2-a6e2-080027635ef5:12345
Galera GTID
0-10-12345
MariaDB 10 GTID(server identifier : transaction processed by server)
(data & cluster Identifier : data change in cluster)
(domain - server identifier - data change in asynchronous cluster)
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Global Transaction Identifier (GTID)Visible in MySQL 5.6
66
8182213e-7c1e-11e2-a6e2-080027635ef5:12345 8182213e-7c1e-11e2-a6e2-080027635ef5:12346 8182213e-7c1e-11e2-a6e2-080027635ef5:12347
New Master Promoted
f4e3bf7a-a91f-11e2-4e02-3f8dbcffaed8:1 f4e3bf7a-a91f-11e2-4e02-3f8dbcffaed8:2 f4e3bf7a-a91f-11e2-4e02-3f8dbcffaed8:3
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Global Transaction Identifier (GTID)Visible in MariaDB 10
67
0-10-12345 0-10-12346 0-10-12347
New Master Promoted
0-20-12348 0-20-12349 0-20-12350
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Global Transaction Identifier (GTID)Visible in Galera
68
00295a79-9c48-11e2-bdf0-9a916cbb9294:64201 00295a79-9c48-11e2-bdf0-9a916cbb9294:64202 00295a79-9c48-11e2-bdf0-9a916cbb9294:64203
New Master Promoted
00295a79-9c48-11e2-bdf0-9a916cbb9294:64204 00295a79-9c48-11e2-bdf0-9a916cbb9294:64205 00295a79-9c48-11e2-bdf0-9a916cbb9294:64206
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Master or SlaveNot a Node Role or Function A Relation Between a Node and a Client
69
master1slave2
slave1slave2
slave1master2
slave1slave2
slave1slave2
Cluster
master for client1 master for client1
client2client1
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Cluster Addresswsrep_cluster_address
70
10.0.0.1
10.0.0.610.0.0.2
10.0.0.3 10.0.0.4
10.0.0.5
Documentation on Galera Cluster Addresses: https://mariadb.com/kb/en/mariadb/galera-cluster-address/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Cluster Addresswsrep_cluster_address
71
10.0.0.1
10.0.0.610.0.0.2
10.0.0.3 10.0.0.4
gcomm://10.0.0.6
10.0.0.5handshake
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Cluster Address
72
10.0.0.1
10.0.0.610.0.0.2
10.0.0.3 10.0.0.4
10.0.0.5
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Cluster Address
73
10.0.0.1
10.0.0.610.0.0.2
10.0.0.3
10.0.0.4
10.0.0.5
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Cluster Addresswsrep_cluster_address = gcomm://node1,node2
Try to Connect to Members (node1, node2)
Can only Join a Running Cluster
74
10.0.0.5
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Node Synchronization (State Transfer)
75
10.0.0.1
10.0.0.610.0.0.2
10.0.0.3
10.0.0.4
10.0.0.5
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Node Synchronization (State Transfer)
76
joined
syncedsynced
desync
synced
undefined
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Node Synchronization (State Transfer)
77
catch-up
UNDEFINED
JOINER
JOINED
SYNCHED
catch-up
SYNCHED
DONOR
JOINED
SYNCHED
New Node Cluster Old Node
Verify Node Synched
Donor Found — Wait Be a Donor
Transfer Missing Data (Private Channel)
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Avoiding Split BrainDistinguishing Server Crash from
Network Failure in Shared Nothing Architecture
Decision Algorithm Used to Avoid Spilt Brain
Absolute Majority Needed in Galera Uneven Number of Nodes Safer
78
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Primary Component
79
Primary
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Primary Component
80
Primary
Non-Primary
Continues Working
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Primary Component
81
Primary
Non-Primary
Tries to ReconnectContinues Working
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Primary Component
82
Primary
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Split Brain
83
Non-Primary Non-Primary
Split Brain Possible with Even Number of Nodes
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Synchronous PenaltiesGalera Copies Data Buffer to All Cluster
Members on COMMIT from Client (~1 RTT added latency)
Connection throughput equals 1/RTT trx/sec
Total throughput equals 1/RTT trx/sec ✕ #connections
A Given Row Can't be Modified More Than 1/RTT times a second
84
Round Trip Time (RTT) is Length of Time for a Signal to be sent and Receipt of Acknowledgement
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Galera Synchronous Penalty in WAN (EC2)
85
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Galera Synchronous Penalty in WAN (EC2)
86
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Slave Lag in Galera
87
Client Master Node
START TRANSACTION
COMMIT Replicate Write Set
OK
Slave Node
COMMIT
PROCESS
APPLY
COMMIT
Acknowledge
SELECT (stale data)
Certify
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Client Proxies
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Client Proxy Objectives & FeaturesObjectives High Availability Minimal Latency Read Scalability Highly Scalable
Features Transparent to Applications Extendible Lightweight with Small
Footprint Authentication Required
89
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Popular Client ProxiesHAProxy — TCP/HTTP Load Balancer
Distributes Traffic across Multiple Servers to Improved Performance and Resource Usage
MaxScale Database Proxy to Forward SQL Statements to Multiple
Database Servers Provides Transparently to Applications, Load Balancing
and High Availability Functionality Scalable and Flexible Architecture that Supports different
Protocols and Routing
Hardware Load Balancer
90
Tutorial on haproxy: https://mariadb.com/resources/blog/setup-mariadb-enterprise-cluster-part-3-setup-ha-proxy-load-balancer-read-and-write-pools Documentation on MaxScale: https://mariadb.com/kb/en/mariadb-enterprise/mariadb-maxscale-21/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Proxy as an IntermediaryApplication-to-Database
Insulates Applications from Complexities of Backend Database Cluster
91
Proxy
MariaDB Master
Application
MariaDBSlave
MariaDBSlave
MariaDBSlave
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Load Balancing — Read ScalingMariaDB Master-Slave Replication GTID Unique across Independent
Replication Streams Multi-Source Replication Optimistic Parallel Replication Slave Execution of Triggers
92
Proxy
MariaDB Master
MariaDBSlave
MariaDBSlave
Client Client Client
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Load Balancing — Write ScalingMariaDB Enterprise Cluster Multi-Master Replication for Write
Scalability
93
Proxy
MariaDB
MariaDB
Client Client Client
MariaDB
Galera Cluster
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Monitoring & Health CheckLoad Balancer should Monitor Often Hardware Load Balancer can Health Check through inetd Script
Limited CPU Available in Appliances
MaxScale Parameter monitor_interval — Default 500 milliseconds
94
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Automatic FailoverAutomatic Detection of Master
Failure MaxScale Monitor Launches Script
upon Master Failure Promotes a Slave as New Master Instructs Other Slave of New
Master
95
MaxScale
Master
Slave Slave Slave
Monitor Detects Event: master_down
Execute Failover Script Promote a Slave to Master CHANGE MASTER on Slaves
Master Fails
binlog Cache
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Optimizing
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Optimizing Overview
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Value in Performance TuningEfficient Use of Hardware Resources Tune Only when Needed Anticipate Needs
Increase in Traffic leads to Exponential Load Growth
User Data Grows Over Time
98
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Tuning RoutineStart Early in Development Life Cycle Regularly Review Production Systems
Schema Changes, etc.
Monitor System Resources Emergency Response Tuning
99
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
HardwareOne Service per Server is Ideal to Prevent Contention More CPU Cores is generally Good More Disk is usually Better
Large Datasets, Fast Disks are Ideal
RAM is usually Best — Traffic Dependent
More of Dataset in Memory, Fewer Slow Disk Operations
100
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Schema Tuning
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Train DevelopersConsistent Naming Conventions Appropriate & Accurate Data Type (e.g., INT for Numbers)
Variable Length Fields are Often Padded
Use NOT NULL where Practical A NULL field uses slightly More Disk and Memory (Depends on
Storage Engine)
Use PROCEDURE ANALYSE( )
102
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Tuning Tables OverallMinimize Table Size on Disk and in Memory Archive Table Data if Possible and Appropriate Remove Duplicate or Unused Indexes Use Appropriate Data Types — Smaller is Better
Consider Sharding Large Tables across Multiple Servers
103
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Number Data TypesUse UNSIGNED when Appropriate
INT(n) Specifies Display Precision, Not Storage Precision
Size and Precision is Storage Engine Dependent Define Handling of Out-of-Range Values with sql_mode
Default Mode: Values are Truncated Silently Strict Mode: Errors are Generated
104
Documentation on Integer Data Types: https://mariadb.com/kb/en/mariadb/data-types/
TINYINT
SMALLINT
MEDIUMINT
INTEGER,INT
BIGINT
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
String Data TypesAll String Data Types have a Character Set CHAR(n) — number of characters, not bytes, wide VARCHAR(n)— Changes to CHAR in Implicit Temporary Tables and mysqld internal buffers
TEXT —Not Supported by the MEMORY Storage Engine; Implicit Temporary Tables may Convert to MyISAM
105
Documentation on String Data Types: https://mariadb.com/kb/en/mariadb/string-data-types/
CHAR
VARCHAR
TINYTEXT
TEXT
MEDIUMTEXT
LONGTEXT
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Character Set & CollationCharacter Set may be Global or for Schema, Table
or Column Multi-Byte Character Sets Increase Disk Storage
and Working Memory Requirements (e.g, UTF-8 Requires 3 or 4 bytes per Character)
Collations affect String Comparison (Character Order) Collations can be Changed for Query
106
Documentation on Character Sets and Collation: https://mariadb.com/kb/en/mariadb/data-types-character-sets-and-collations/
SELECT * FROM table1 ORDER BY col1 COLLATE latin1_german2_ci;
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Binary Data TypesBINARY and VARBINARY are Case-Sensitive Versions of CHAR and VARCHAR
No Character Set and Collation for Binary Types - ordered by bytes
Blobs are Used often to Store Files in a Database Files on Disk are often Faster Blobs are Included in Transactions, Replication, and
Backups Blobs Inflate mysqld memory usage
107
Documentation on BINARY Data Type: https://mariadb.com/kb/en/mariadb/binary/ Documentation on VARBINARY Data Type: https://mariadb.com/kb/en/varbinary/ Documentation on BLOB Data Type: https://mariadb.com/kb/en/blob/
BINARY
VARBINARY
TINYBLOB
BLOB
MEDIUMBLOB
LONGBLOB
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Date & Time Data TypesDATE — from 1000-01-01 to 9999-12-31 (YYYY-MM-DD)
TIME — from -838:59:59 to 838:59:59
DATETIME — Same Range with Time (YYYY-MM-DD HH:mm:ss)
TIMESTAMP — Unix timestamp, in seconds from 1970-01-01
Many Apps Store UNIX_TIMESTAMP() values in unsigned integer field YEAR — Accepts YY or YYYY
108
DATE
TIME
DATETIME
TIMESTAMP
YEAR
Documentation on Date and Time Data Types: https://mariadb.com/kb/en/mariadb/date-and-time-data-types/Documentation on Microseconds in MariaDB: https://mariadb.com/kb/en/mariadb/microseconds-in-mariadb/
SELECT CURTIME(4); +---------------+ | CURTIME(4) |
+---------------+ | 05:33:09.1061 | +---------------+
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Listing Acceptable ValuesENUM is an Enumerated List of
String Values — uses a 2-byte integer index
109
Documentation on ENUM: https://mariadb.com/kb/en/mariadb/enum/ Documentation on SET: https://mariadb.com/kb/en/mariadb/set-data-type/
CREATE TABLE colors (primary_colors ENUM('red', 'yellow', 'blue'));
CREATE TABLE colors
(primary_colors SET('red','yellow','blue'));
INSERT INTO colors VALUES('red'), ('red,blue');
Simple Example for an ENUM Column
Simple Example for a SET Column
SET is a Specified List of String Values — Can Hold Multiple Specified Values
Not Indexable
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Keys and IndexesMake PRIMARY KEY as Small as Practical
InnoDB Stores Copy of PRIMARY KEY with each Index Entry Secondary Indexes Grow with PRIMARY KEY
Use Surrogate PRIMARY KEY, Not Natural Ones AUTO_INCREMENT is like a Sequence AUTO_INCREMENT UNSIGNED NOT NULL Careful of AUTO_INCREMENT Contention for Pre-Plugin InnoDB
Consider Partial (Prefix) Index for String Indexes With Good Index Selectivity, Won't Affect Performance Much Without Good Selectivity is a Different Problem
110
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
File StorageStoring Files in BLOB Useful
Included in Back-Up and Replication Guaranteed to be Consistent with Other Data
Storing Files in BLOB has Problems Database gets Too Large MariaDB Can't do Partial Field Read — MyISAM Reads Entire File InnoDB can Skip if Field Not Referenced in Query
Minimize by putting Files in Separate Table (1:1 relationship)
Store Files on File System — Put File Name and Path in Table Database is Smaller and More Efficient Organize Separate Back-Up, possibly with External Locking for
Consistency
111
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Normalization of TablesProcess of Optimizing and Factoring a Schema
1:1, 1:n and n:n 1NF / 2NF / 3NF
Makes a Data Set Smaller by Eliminating Redundant or Duplicated Data
Problem if Data is Smaller than Key
Ensures Data Set Consistency and Integrity Often Improves Concurrency by Reducing Locking
Overhead — Not Always
Increases Number of Tables and Associated Maintenance
112
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Denormalization of TablesComplete Normalization can Slow Queries Complex JOINs are Drains and Maintainence Problem
Add Redundant Data Back to Simplify with JOIN Queries Combine 1:n Relationships Add Pre-Computed Columns Use Fake Materialized Views
Denormalization Adds Redundant Data to Schema Write Queries become More Complex or Numerous as
Multiple Locations must be Maintained
Easier to Normalize First, Then Denormalize when Appropriate
113
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Dynamic ColumnsStores Key/Value Pairs within a
BLOB Column with COLUMN_CREATE()
Used to Store many Attributes which are Unknown in Advance
Manipulated with Special Dynamic Column Functions:
114
SELECT item_name, COLUMN_GET(item_attributes, 'color' AS CHAR) AS color FROM clothes;
+-----------------+-------+ | item_name | color | +-----------------+-------+
| MariaDB T-shirt | blue | +-----------------+-------+
COLUMN_GET()
COLUMN_ADD()
COLUMN_LIST()
COLUMN_DELETE()
Documentation on Dynamic Columns: https://mariadb.com/kb/en/dynamic-columns/ Tutorial on Dynamic Columns: http://radar.oreilly.com/2015/04/dynamic-columns-in-mariadb.html
CREATE TABLE clothes
(item_name VARCHAR(32) PRIMARY KEY,
item_attributes BLOB);
INSERT INTO clothes
VALUES ('MariaDB T-shirt',
COLUMN_CREATE('color', 'blue', 'size', 'XL'));
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Virtual ColumnsTwo Virtual Column Types:
PERSISTENT (stored) VIRTUAL (generated only)
All Data Types Supported Use PERSISTENT for Indexes - Cannot
be Primary Key
Used with InnoDB, Aria, MyISAM, CONNECT
115
Documentation on Virtual Columns: https://mariadb.com/kb/en/virtual-columns/
CREATE TABLE sales
(sales_id INT AUTO_INCREMENT KEY, qty INT, price DECIMAL(6,2), total DECIMAL(6,2) AS (qty * price) VIRTUAL);
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
InnoDB In-Depth
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Advantages & Disadvantages of InnoDB
117
Read Uncommitted Read Committed Repeatable Read (no fantom rows) Serializable
Fully Transactional,
ACID Compliant
Data & Indexes Cached by mysqld
in Buffer Pool
Row-Level Locking for High-
Concurrency
Supports Four Isolation Levels
Supports Foreign Keys and Multi-
Version Concurrency
Control
Reliable Crash Recovery
Slower than MyISAM, but
Higher Concurrency
Advantages Disadvantages
Isolation Levels
Documentation for InnoDB & XtraDB: https://mariadb.com/kb/en/mariadb/xtradb-and-innodb/
117
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Isolation LevelsInnoDB Supports All Four Transaction Isolation Levels*
READ UNCOMMITTED READ COMMITTED REPEATABLE READ SERIALIZABLE
* Listed in Order of Increasing Performance Potential
Increasing Isolation Protection means More Locks Needed
Reduces Query Concurrency Increases Memory Usage
Set Isolation Level on Per-Session Basis for Transactions Requiring More Protection
Leave Bulk of Traffic to Operate at Lower Isolation Level
118
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
InnoDB Architecture
119
Additional Memory Pool
Buffer Pool *
Data Files * **
Redo Log (File 1/2)
Redo Log (File 2/2)
Memory
Disk
* Cached Data & Indexes, Stored in Pages ** Transaction Data Writes *** Data, Index, and Undo Log Files
INSERT, UPDATE, DELETE
1
COMMIT
CHECKPOINT
Log Buffer * *
2
3
119
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Buffer PoolLarger is Generally Better for innodb_buffer_pool_size
For Servers with Only InnoDB Tables, 80% of Available Memory
MyISAM is Used Internally and Needs Resources along with OS
Dirty Buffer Pool Pages are Flushed in Background Allows Changes to be Merged and Written Sequentially to Disk Applications with Plenty of Writes, or Write Spikes, may need
Adjusting Use innodb_max_dirty_pages_pct to Control Frequency of
Flushing Dirty Pages
InnoDB Plugin can Disable the Adaptive Hash Index Mostly Improves Performance, but Not Always
120
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
InnoDB Clustered Index
121
root
node node
leafleaf leaf
PAGE (16KB) PAGE (16KB) PAGE (16KB)
B-Tree
HEADER PRIMARY KEY ROW DATA
leaf
121
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
InnoDB Secondary Index
122
root
node node
leafleaf leaf
PAGE (16KB) PAGE (16KB) PAGE (16KB)
B-Tree
HEADER VALUES OF INDEXED COLUMNS PRIMARY KEY
leaf
122
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
IndexesInnoDB Indexes are Clustered
Row Data is Stored with Primary Key
Use Short Surrogate Primary Keys (INT UNSIGNED AUTO_INCREMENT)
Secondary Index Entries hold a Copy of PK — Large PK Increases All
Table Index Statistics are Vague for InnoDB Gathered by a Series of Random Index Dives It's Possible for Stats to be Inaccurate and Cause
Optimizer to Choose Bad Query Plan Run Periodically ANALYZE TABLE to Regenerate Stats
123
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
InnoDB Lock MonitoringINNODB_LOCKS
INNODB_TRX
INNODB_BUFFER_POOL_STATS
INNODB_TABLE_STATS
INNODB_INDEX_STATS
SHOW TABLES LIKE 'INNODB%';
SELECT r.trx_id AS waiting_trx_id, r.trx_mysql_thread_id AS waiting_thread,
r.trx_query AS waiting_query, b.trx_id AS blocking_trx_id, b.trx_mysql_thread_id AS blocking_thread, b.trx_query AS blocking_query FROM information_schema.innodb_lock_waits w
INNER JOIN information_schema.innodb_trx b ON b.trx_id = w.blocking_trx_id INNER JOIN information_schema.innodb_trx AS r ON r.trx_id = w.requesting_trx_id;
Locked & Locking Queries
124
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
InnoDB MutexesProcess of Locking and
Latching Resources for InnoDB
InnoDB Keeps Track of All Mutexes
125
SHOW ENGINE INNODB MUTEX;
+--------+------------------------------+---------------+
| Type | Name | Status |
+--------+------------------------------+---------------+
| InnoDB | &rseg->mutex | os_waits=1 |
| InnoDB | &dict_sys->mutex | os_waits=1 |
| InnoDB | &log_sys->mutex | os_waits=116 |
| InnoDB | &buf_pool->flush_state_mutex | os_waits=3913 |
| InnoDB | &buf_pool->LRU_list_mutex | os_waits=16 |
| InnoDB | combined &block->mutex | os_waits=7 |
| InnoDB | &dict_operation_lock | os_waits=167 |
| InnoDB | &log_sys->checkpoint_lock | os_waits=2239 |
| InnoDB | &btr_search_latch_arr[i] | os_waits=6 |
| InnoDB | combined &block->lock | os_waits=47 |
+--------+------------------------------+---------------+
Documentation on SHOW ENGINE INNODB MUTEX: https://mariadb.com/kb/en/mariadb/show-engine/#show-engine-innodb-mutex
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Transaction LogConverts Transactions from Random I/O to Sequential
Transaction Data is Written to Transaction Log for Durability and Speed - Not Directly to Data Files
Log Files Rotate as a Circular Buffer with Data Merged to Disk in Background
126
fsync() FrequencyVariable Setting Write and Flush Action
innodb_flush_log_at_trx_commit = 0 Write at Commit, but No Flush
innodb_flush_log_at_trx_commit = 1 Flush at Commit (Default)
innodb_flush_log_at_trx_commit = 2 Write at Commit, Flushed Every Second
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Transaction LogLarge innodb_log_file_size (Default 50 MB) Results
Usually in Less I/O and Better Performance Crash Recovery Delay Increases with Log File Size
Increasing innodb_log_buffer_size (Default 1MB) May Reduce Log File I/O for Large Transactions
127
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
The InnoDB CoreVariable innodb_flush_method Controls How Data is
Written to Disk Default is a Normal fsync( ) Operation Leads to "Double Buffering" of Data (Buffer Pool and OS Disk
Cache)
O_DIRECT Allows InnoDB to Use Direct I/O Bypasses any OS Cache
Changing Flush Method should be Tested Thoroughly Some Combinations of Device, Filesystem, OS may Not
Benefit
128
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Collecting Information
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Tools & StatisticsIdentify Accurately and Carefully Trouble
Spots Gather Performance Stats with MariaDB
and OS Tools SHOW Statements PERFORMANCE_SCHEMA CPU, Disk, Network, Memory, & Swap Stats
Retain Snapshots of Multiple Stats Data from a Single Point Shows Little
Automate Collection of Stats into Logs Useful for Emergency Tuning
130
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
SHOW PROCESSLISTSnapshot of mysqld Activity
mysqld is Multi-Threaded, One Thread per Client Connection — a "process" is a "thread"
Accumulate SHOW PROCESSLIST Snapshots for History of Thread Activities
131
Time: Length of Time Thread has been in Current State State: Indicates Thread Activity
SHOW PROCESSLIST \G
****** 1. row ******** Id: 9546 User: monyog Host: localhost:40600 db: NULL
Command: Sleep Time: 192 State: Info: NULL Progress: 0.000
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Thread StatesAfter Create Analyzing Checking Permissions Checking Table Cleaning Up Closing Tables Converting HEAP to MyISAM Copy to tmp Table Copying to Group Table Copying to tmp Table Copying to tmp table on Disk Creating Index
Creating Sort Index Creating Table Creating tmp Table Deleting from Main Table Deleting from Reference
Tables discard_or_import_tablespace End Executing Execution of init_command Flushing Tables Freeing Items FULLTEXT initialization
Init Killed Locked Logging Slow Query Login Manage Keys NULL Opening Tables, Opening
Table Optimizing Preparing Purging Old Relay Logs Query End
132
Documentation on General Thread States: https://mariadb.com/kb/en/mariadb/general-thread-states/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Thread StatesReading from Net
Removing Duplicates Removing tmp Table
Rename Rename Result Table
Reopen Tables Repair by Sorting
Repair Done
Repair with Keycache Rolling Back
Saving State Searching Rows for Update
Setup
Sorting for Group Sorting for Order
Sorting Index Sorting Result
Statistics System Lock
Table Lock
Updating Updating Main Table
Updating Reference Tables User Lock
User Sleep
Waiting for (tables|tables|table flush) Waiting for All Running Commits to
Finish
Waiting for Commit Lock Waiting for Global Read Lock
Waiting for lock_type Lock Waiting for Release of Read Lock
Waiting on Cond
Waiting to Get Read Lock Writing to Net
133
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Check System StatusUse SHOW STATUS for List of Internal Counters
GLOBAL for System Status Since Start
SESSION for Client Connection
FLUSH STATUS Resets Local Counters
Monitor Changes to Counters to Identify Hot Spots Collect Periodically Status Snapshots to Profile Traffic
134
Documentation on SHOW STATUS: https://mariadb.com/kb/en/mariadb/show-status/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Connection & Thread StatusClients & Connections Aborted_clients
Aborted_connects
Connections
Max_used_connections
Ssl_*
Threads Delayed_insert_threads
Threads_cached
Threads_connected
Threads_created
Threads_running
Bytes & Writes Bytes_received
Bytes_sent
Compression
Delayed_errors
Delayed_writes
Uptime*
135
Docmentation on Server Status Variables: https://mariadb.com/kb/en/mariadb/server-status-variables/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Status CountersTable Status Created_tmp_disk_tables
Created_tmp_files
Created_tmp_tables
Opened_table_definitions
Opened_tables
Table_locks_immediate
Table_locks_waited
Queries Last_query_cost
Not_flushed_delayed_rows
Prepared_stmt_count
Qcache_*
Queries
Questions
Select_*
Slow_launch_threads
Slow_queries
Sort_*
Transaction Coordinator Tc_log_max_pages_used
Tc_log_page_size
Tc_log_page_waits
Other Status Counters Flush_commands
Handler_*
Innodb_*
Key_*
Open_*
Opened_files
136
Docmentation on Server Status Variables: https://mariadb.com/kb/en/mariadb/server-status-variables/ Documentaton on InnoDB Status Variables: https://mariadb.com/kb/en/mariadb/xtradbinnodb-server-status-variables/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Binary Log Status VariablesFile Information Binlog_snapshot_file
Binlog_snapshot_position
Binlog_bytes_written
Caches Binlog_cache_disk_use
Binlog_cache_use
Binlog_stmt_cache_disk_use
Binlog_stmt_cache_us
Commits Binlog_commits
Binlog_group_commit_trigger_count
Binlog_group_commit_trigger_lock_wait
Binlog_group_commit_trigger_timeout
Binlog_group_commits
137
Docmentation on Replication Status Variables: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-status-variables/Docmentation on Replication Status Variables: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-status-variables/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Replication Status CountersMaster Status Master_gtid_wait_count
Master_gtid_wait_time
Master_gtid_wait_timeouts
Rpl_status (deprecating)
Slave Status Slave_connections
Slave_heartbeat_period
Slave_open_temp_tables
Slave_received_heartbeats
Slave_retried_transactions
Slave_running
Slave_skipped_errors
Slaves_connected
Slaves_running
Slave Changes Com_change_master
Com_slave_start
Com_slave_stop
Com_start_all_slaves
Com_start_slave
Com_stop_all_slaves
Com_stop_slave
138
Docmentation on Replication Status Variables: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-status-variables/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
PERFORMANCE_SCHEMAMonitors MariaDB Server Events
Function Calls, Operating System Waits, Internal Mutexes, I/O Calls
Detailed Query Execution Stages (Parsing, Statistics, Sorting)
Some Features Storage Engine Specific
Monitoring Lightweight and Requires No Dedicated Thread
Not a Simple, Quick Fix
Interpreting Output Requires Knowledge of Internals Designed to be Used Iteratively with Successive
Refinement
139
Documentation on performance_schema: https://mariadb.com/kb/en/mariadb/performance-schema/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
PERFORMANCE_SCHEMA Tablescond_instances
events_waits_current
events_waits_history
events_waits_history_long
events_waits_summary_by_instance
events_waits_summary_by_thread_by_event_name
events_waits_summary_global_by_event_name
file_instances
file_summary_by_event_name
file_summary_by_instance
mutex_instances
performance_timers
rwlock_instances
setup_consumers
setup_instruments
setup_timers
threads
140
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
ProfilingUse Profiling to Measure Differences in
Execution between Queries Enable by Setting PROFILING to ON
141
SHOW PROFILE FOR QUERY 1;
+----------------------+----------+
| Status | Duration |
+----------------------+----------+
| starting | 0.000047 |
| checking permissions | 0.000005 |
| Opening tables | 0.000015 |
| After opening tables | 0.000004 |
| System lock | 0.000004 |
| Table lock | 0.000011 |
| init | 0.000017 |
| optimizing | 0.000007 |
| statistics | 0.000012 |
| preparing | 0.000014 |
| executing | 0.000003 |
| Sending data | 0.008304 |
| end | 0.000006 |
| query end | 0.000003 |
| closing tables | 0.000002 |
| Unlocking tables | 0.000007 |
| freeing items | 0.000004 |
| updating status | 0.000015 |
| cleaning up | 0.000014 |
+----------------------+----------+
SET PROFILING=ON;
SELECT * FROM company.employees;
SHOW PROFILES \G
***** 1. row ***** Query_ID: 1
Duration: 0.00849380 Query: SELECT * FROM company.employees
Documentation for SHOW PROFILE: https://mariadb.com/kb/en/mariadb/show-profile/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Slow Query LogContains Queries that took Longer
than long_query_time Seconds to Execute — Off by Default
Activate when First Using an Application to Monitor Slow Queries
Default Location is in Data Directory Use mysqldumpslow to Process Log
Use also pt-query-digest as an Alternative to Process Log
142
[mysqld] log-slow-queries = ON long-query-time = 2
log-queries-not-using-indexes log-slow-admin-statements log-slow-vervosity = query_plan,explain
Documentation on Slow Query Log: https://mariadb.com/kb/en/mariadb/slow-query-log-overview/ Documentation on mysqldumpslow: https://mariadb.com/kb/en/mariadb/mysqldumpslow/ Documentation on pt-query-digest: https://www.percona.com/doc/percona-toolkit/2.1/pt-query-digest.html
Excerpt from /etc/my.cnf.d/server.cnf
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
External ToolsApacheBench htop httperf innotop iostat / vmstat / dstat/top Percona Toolkit MyBench
MariaDB Benchmark Suite (sql-bench) mysqladmin (extended-status) mysqlslap mytop statpack Super Smack SysBench
143
ApacheBench: http://httpd.apache.org/docs/2.4/programs/ab.html InnoTop: https://github.com/innotop/innotop MyBench: http://jeremy.zawodny.com/mysql/mysql-query-benchmarking/img13.html mysqlslap: https://mariadb.com/kb/en/mariadb/mysqlslap/ Percona Toolkit: https://www.percona.com/software/database-tools/percona-toolkit
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Benchmarking
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Tests to RunNumber of Queries Expected per Unit of
Time Number of Concurrent Client
Connections Expected Average Number of Clients Peak Number of Concurrent Clients
Length of a Client Connection Expected — In Milliseconds
Size of Dataset — Fit in Memory When all Requirements will Double
145
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
SysbenchOpen-Source Benchmarking Tool Contains oltp and mutex Tests
146
Documentation on sysbench and oltp: https://www.percona.com/docs/wiki/benchmark_sysbench_oltp.html
mysql -e "CREATE DATABASE sbtest"
sysbench --mysql-user=root --test=oltp prepare
sysbench --mysql-user=root --test=oltp run
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Establish a Base LineUse Current Configuration Get Stable Results Use Same Hardware and Configuration
if Possible Use Same Size and Distribution Dataset
147
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Construct TestsDetermine System Resource Limits and Design
Test Should Run Normally Should Exceed Server Resource Limits Should Simulate Nightly Back-up Load without
Users Noticing
Examine Impact of Configuration Changes Remove or Restrict Access to CPU core or Hard
Drives Start another Process via cron that Consumes
Suddenly Memory Observe What Happens When a Caching Layer
goes Offline
Use Tests to Identify Easy Solutions
148
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Examine Test ResultsTest System Identical to Production
System Same Operating System (Version, Libraries,
Patches)
Hardware (Number of Cores, Physical Memory, Disk Speed)
Other Resources (Memory Available to MariaDB, Competing Services)
Test System Simulating Real Data and Traffic
Number of Clients (Concurrently, per Hour per Day, Peak Traffic)
Any Caching that might Skew Results
149
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Query Tuning
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
The Query Process
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
MariaDB Query Process
Connection Handling
Archive
InnoDB
Aria
Memory
Storage Engines
Query Cache
SQL Parser
Optimizer
Clients
152
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Thread Cache
Thread Cache
Query Cache
SQL Parser
Optimizer
Storage Engines
Thread is Assigned to Each Connection
Threads may be Reused from the Thread Cache or Created
153
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Thread Pool
Thread Pool
Query Cache
SQL Parser
Optimizer
Storage Engines
Main Thread Listens for Client Connections User Authentication based on Host, User, and
Password Client Buffers for Session Variables and Network
Communications
154
thread_handling = pool-of-threads thread_pool_size = 3
Excerpt from /etc/my.cnf.d/server.cnf
Documentation on Thread Pool: https://mariadb.com/kb/en/mariadb/thread-pool-in-mariadb/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Query Cache
Thread Pool
Query Cache
SQL Parser
Optimizer
Storage Engines
Documentation on Query Cache: https://mariadb.com/kb/en/mariadb/query-cache/
SHOW VARIABLES LIKE 'query_cache_type';
Possible to Stipulate in a Query
Stores SELECT Query Result Sets
Useful for High Read, Low Write Servers
Cache Purged when Related Data Changed
SELECT SQL_CACHE * FROM ...
SELECT SQL_NO_CACHE * FROM ...
155
No Longer Recommended
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
SQL Parser
Thread Pool
Query Cache
SQL Parser
Optimizer
Storage Engines
Converts SQL Text to Binary Format Parses SQL into Tokens (i.e., keyword, table,
field, value) Applies Grammar Rules to check
Validity (Lexical Scanner, Grammar Rules Module)
Construct a Parse Tree to be passed to Optimizer
156
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Optimizer
Thread Pool
Query Cache
SQL Parser
Optimizer
Storage Engines
157
The Optimizer Reads the Parse Tree and Determines an Execution Plan
Locate any Related Indexes Compare Efficiency of Index Access to
Table Scan Determine JOIN order of Table Eliminate Unnecessary Tables and WHERE
clauses Find Indexes that can handle GROUP BY
and ORDER BY
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Query Tuning
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Query Tuning OverviewDon't Query Tune on Production Server
Use Test Server with same Hardware and OS
Use Copy of Production Data Set
Query Frequency and Query Speed are Both Important
Frequent Slow Queries are often Worse than Rarely Run Very Slow Query
159
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
IndexingIndexes Improve Read Performance
Without Index, MariaDB Must Read Every Row
With Index, MariaDB Jumps to Requested Rows
Indexes Improve Writes since they Require Reads
Index for Speed but Avoid Indexing Excessively or Arbitrarily
Remove Unused or Redundant Indexes
160
Article on Getting Started with Indexes: https://mariadb.com/kb/en/mariadb/getting-started-with-indexes/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Index SizeKeep Indexes as Small as Practical
Faster if Fits in Memory Rebuilds Faster after Writes
Use Partial Prefix-Index for String Columns
Reduces Index Size — May Slow Searches Use Index Cardinality (Uniqueness Measure)
if Necessary — Re-evaluate as Data Grows Low Cardinality Indicates many
Duplicates High Cardinality is More Useful
161
Documentation on Index Statistics: https://mariadb.com/kb/en/mariadb/index-statistics/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Query AnalysisUse EXPLAIN to see how
Troublesome Query Executed Shows if Indexes are Used
Use EXPLAIN EXTENDED and SHOW WARNINGS to see how Query Rearranged before Execution
162
Documentation on EXPLAIN: https://mariadb.com/kb/en/mariadb/explain/ Documentation on EXPLAIN EXTENDED: https://mariadb.com/kb/en/mariadb/explain/#explain-extended Documentation on mysqldumpslow: https://mariadb.com/kb/en/mariadb/mysqldumpslow/
EXPLAIN SELECT * FROM company.clients WHERE name_first = 'John' AND name_last = 'Smith' \G
id: 1 select_type: SIMPLE table: clients type: ref
possible_keys: names key: names key_len: 15 ref: const,const rows: 1
Extra: Using where
EXPLAIN EXTENDED SELECT ...; SHOW WARNINGS;
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
ANALYZE vs. EXPLAINEXPLAIN is Expected Execution Plan
ANALYZE Executes and Shows Effect Writes are Executed
163
ANALYZE SELECT * FROM company.clients
WHERE name_first = 'John'
AND name_last = 'Smith' \G
***** 1. row *****
id: 1
select_type: SIMPLE
table: clients
type: ref
possible_keys: names
key: names
key_len: 15
ref: const,const
rows: 1
r_rows: 1.00
filtered: 100.00
r_filtered: 100.00
Extra: Using where
See Upcoming Slides for Lists of Select Types and Query Types
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Fields from EXPLAINField Descriptionid Query identifierselect_type Type of SELECT performed
table The table in use (each table in a JOIN will use one row)
type The table access strategy
possible_keys Any available indexes that could resolve the query
key Index MariaDB chose from possible_keyskey_len The portion of the chosen index to be used (width in bytes)ref Columns from the index, or a constant, usedrows Number of rows the query is expected to touchExtra Extra information relating to the access strategy
164
Documentation on EXPLAIN: https://mariadb.com/kb/en/mariadb/explain/ Documentation on EXPLAIN EXTENDED: https://mariadb.com/kb/en/mariadb/explain/#explain-extended
See Next Slide for List of Types
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Scan or Index Method Types
Field Description
const A constant value can be read once and cached
eq_ref One index access per outer query row
ref Multiple index accesses per outer query row
index_merge Multiple indexes used, merging into a single result set
range Multiple index accesses to return all rows within a range
index Full index scan (every index entry is read sequentially)
all Full table scan (every record is read sequentially)
Method Used to Access Table
165
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Select TypesType of SELECT being Performed
166
Type of SELECT Description
SIMPLE Normal SELECT Statement
PRIMARY First (Outermost) SELECT in JOIN
UNION Subsequent Query in JOIN
DEPENDENT UNION Subsequent Query Dependent on an Outer Query Result
UNION RESULT Query is the Results Set from a UNION
SUBQUERY First SELECT in a Sub-Query
DEPENDENT SUBQUERY A Sub-Query that must be Re-Executed for Each Combination of Values in Outer Query
DERIVED Sub-Query in a FROM Clause
UNCACHEABLE SUBQUERY Sub-Query that must be Re-Executed for Every Iteration of an Outer Query
UNCACHEABLE UNION Subsequent Query in a UNION belonging to an Uncacheable Subquery
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Extra Explanationsconst row not found
Distinct
Full scan on NULL key
Impossible HAVING
Impossible WHERE noticed after reading const tables
Impossible WHERE
No matching min/max row
No matching row in const table
No tables used
Not exists
Open_frm_only
Open_full_table
Open_trigger_only
Range checked for each record (index map: N)
Scanned N databases
Select tables optimized away
Skip_open_table
Unique row not found
Using filesort
Using index
Using index condition
Using index condition(BKA)
Using index for group-by
Using intersect(…)
Using sort_union(…)
Using temporary
Using where
Using WHERE with pushed condition
Using buffer
167
Documentation on Extra Field of EXPLAIN: https://mariadb.com/kb/en/mariadb/explain/#extra-column
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Selecting All ColumnsSELECT with * Performs Badly
Increases Size of Results Set — Memory and Network Overhead on Client and Server
Increases Disk Activity for Storage Engines that can Read Partial Records (not MyISAM, but InnoDB)
Causes Development and Usage Problems
Ambiguity with JOIN tables having same Column Names
Problems when Schema has been Changed
168
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
MariaDB OptimizerOptimizer calculates Best Plan
Can't Take too Long or Defeats Purpose
Many Possible Plans for Large Joins Greedy Searches used to Reduce
Possibilities Possible to Adjust Decision Parameters
Limit Number of Tables Considered (optimizer_search_depth)
Skip Plans Based on Row Count Estimates (optimizer_prune_level)
169
Documentation on optimizer_search_depth: https://mariadb.com/kb/en/mariadb/server-system-variables/#optimizer_search_depth Documentation on optimizer_prune_level: https://mariadb.com/kb/en/mariadb/server-system-variables/#optimizer_prune_level Article on Setting Optimizer Search Depth: https://mariadb.com/resources/blog/setting-optimizer-search-depth-mysql
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Optimizer HintsGiving Hints can Affect the Optimizer’s
Choices Index Choices USE INDEX IGNORE INDEX FORCE INDEX
Buffering Methods for GROUP BY SQL_BUFFER_RESULT SQL_SMALL_RESULT SQL_BIG_RESULT
170
SELECT * FROM people FORCE INDEX names WHERE name_first = 'Bob';
Documentation on Forcing Indexes: https://mariadb.com/kb/en/mariadb/index-hints-how-to-force-query-plans/ Documentation on Indexing with GROUP BY: https://mariadb.com/kb/en/mariadb/index-hints-how-to-force-query-plans/#forcing-an-index-to-be-used-for-order-by-or-group-by
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Joining TablesJOIN is a Cartesian Product of Table
Subsets with Variations The Optimizer can Help
Orders Tables for Minimum Cost — Fewest Rows Examined
Converts OUTER to INNER when Possible
Ensure Columns in ON and USING Clauses are Indexed
Restrict GROUP BY and ORDER BY to Columns in same Table
Consider JOIN Decomposition
171
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Sub-QueriesMariaDB uses Semi-Join Optimizations
— Queries within IN( ) Subquery Cache used to Optimize
Evaluation of Correlated Subqueries Views & Sub-Queries are Essentially the
Same Potential Performance Drain
172
Documentation on Subquery Optimization: https://mariadb.com/kb/en/mariadb/subquery-optimizations/ Documentation on Semi-Join Optimization: https://mariadb.com/kb/en/mariadb/semi-join-subquery-optimizations/ Documentation on Subquery Cache: https://mariadb.com/kb/en/mariadb/subquery-cache/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Limiting RowsLIMIT is Evaluated at End of Query Plan
ORDER BY with LIMIT needs to Sort All Rows
LIMIT(o, n) will still Parse o + n Rows Scan in Index Order and Apply LIMIT
vs. Sort and LIMIT
173
Documentation on LIMIT Clause: https://mariadb.com/kb/en/mariadb/select/#limit Documentation on LIMIT ROWS EXAMINED: https://mariadb.com/kb/en/mariadb/limit-rows-examined/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Ordering and AggregatingGROUP BY Triggers an Implicit ORDER BY (filesort)
ORDER BY NULL Avoids It
174
EXPLAIN SELECT Continent, COUNT(*) FROM country GROUP BY Continent \G
... Extra: Using temporary; Using filesort
EXPLAIN SELECT Continent, COUNT(*) FROM country
GROUP BY Continent ORDER BY NULL\G ... Extra: Using temporary
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Advanced Index Usage
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Faster ResultsUnderstanding of Indexes will Help to
Get More from Them Clustered Index InnoDB Secondary Indexes
Indexes can't Resolve Flawed Logic in SQL Statements
176
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
B-Tree IndexesBalanced Binary Search Trees holding
Pointers to Rows Precision Depends on Storage Engine Some Store Pointers to Rows (Offsets) InnoDB Primary Key Stores Row — Other
Indexes Store Copy of Key
Good for Many Operations Equality (= <>) Range Access (> < >= <= BETWEEN)
Partial String (i.e., LIKE) — String Constants without Leading %
Sorting (ORDER BY) — Indexes are Pre-Sorted
177
Documentation on Indexes Used by Storage Engines: https://mariadb.com/kb/en/mariadb/storage-engine-index-types/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
InnoDB Secondary Index
178
root
node node
leafleaf leaf
PAGE (16KB) PAGE (16KB) PAGE (16KB)
B-Tree
HEADER VALUES OF INDEXED COLUMNS PRIMARY KEY
leaf
178
Article Comparing InnoDB Primary and Secondary Indexes: https://mariadb.com/resources/blog/innodb-primary-key-versus-secondary-index-interesting-lesson-explain
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Composite IndexesSpans Multiple Columns Increases Search Speed Best when All Columns in Index
Specified in Query Not Used with Latter Columns of
Index Only One Index Used per Table
179
CREATE TABLE people
(name_first CHAR(20), name_last CHAR(20), city CHAR(20), KEY names(name_first,name_last));
SELECT * FROM people WHERE name_first = 'Bob' AND name_last = 'Smith';
SELECT * FROM people WHERE name_first = 'Bob';
Which Use an Index
SELECT * FROM people WHERE name_last = 'Smith';
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Ordering with IndexesUsing an ORDER BY Clause can
Cause an Index Not to be Used
180
CREATE TABLE students (name_first CHAR(20), name_last CHAR(20), city CHAR(20),
INDEX names(name_first, name_last), INDEX city (city));
SELECT * FROM table1
WHERE city = 'Boston' ORDER BY name_first;
SELECT * FROM table1 WHERE name_first = 'Bob'
ORDER BY name_last;
Which Use an Index
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Aggregating with IndexesUsing a GROUP BY Clause can
Cause an Index Not to be Used
181
CREATE TABLE students (name_first CHAR(20), name_last CHAR(20), city CHAR(20),
INDEX names(name_last, name_first), INDEX city (city));
SELECT name_first, COUNT(*)
FROM students WHERE name_last = 'Smith' GROUP BY name_first;
SELECT city, COUNT(*) FROM students WHERE name_first = 'Bob' GROUP BY city;
Which Use an Index
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Straddling Columns with IndexesUsing AND in WHERE is Fine when
All Columns in Index can Cause an Index Not to be
Used
182
CREATE TABLE students (name_first CHAR(20), name_last CHAR(20), city CHAR(20), country CHAR(2)
INDEX name_city (name_first, name_last, city));
SELECT * FROM table1
WHERE name_first = 'Bob' AND name_last = 'Smith';
SELECT * FROM table1 WHERE name_first = 'Bob'
AND city = 'Boston';
Which Use an Index
SELECT * FROM table1 WHERE name_first = 'Bob' AND country = 'us';
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Pagination by SkippingIndexes can Improve Query
Performance Performance can be Lost with
some Clauses (e.g., LIMIT) for Returning Pages of Results
Check Status of rows_read to see clearly Index Usage
183
FLUSH STATUS;
SELECT name_last FROM people WHERE name_first = 'Vito' LIMIT 100, 5;
SHOW STATUS LIKE 'rows_read' \G ... Value: 218907
Add an Index
Without Index
CREATE INDEX name ON people (name_first);
FLUSH STATUS;
SELECT ... LIMIT 100,5; SHOW STATUS LIKE 'rows_read' \G ... Value: 105
Only 100 More than Needed
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Pagination by IndexIndexes can Improve Query
Performance Performance can be Lost with
some Clauses (e.g., LIMIT) for Returning Pages of Results
Check Status of rows_read to see clearly Index Usage
184
SELECT people_id , name_last FROM people WHERE name_first = 'Vito'
ORDER BY people_id LIMIT 5;
[results]
SELECT name_last FROM people WHERE name_first = 'Vito' AND people_id > [last_id] ORDER BY people_id LIMIT 5;
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Handling Duplicate Row ValuesIndexes can Improve Query
Performance Performance can be Lost with
some Clauses (e.g., LIMIT) for Returning Pages of Results
Check Status of rows_read to see clearly Index Usage
185
SELECT people_id , salary, name_last FROM people WHERE name_first = 'Vito'
ORDER BY salary DESC LIMIT 5;
[results - last 3 same salary]
SELECT people_id , salary, name_last FROM people WHERE name_first = 'Vito' AND salary <= [last_salary] AND people_id NOT IN([id_same_salary])
ORDER BY salary DESC LIMIT 5;
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Diagnosing Systems
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Common Bottlenecks
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Lack of CachingCaching can Occur at Many Levels
188
Applications User Session Data Pre-Compiled, Static Content
Database System Query Cache Storage Engine
Hardware File System Cache Proxy Server Content Distribution Network memcached
Documentation on Query Cache: https://mariadb.com/kb/en/mariadb/query-cache/ Documentation on InnoDB Buffer Pool: https://mariadb.com/kb/en/mariadb/xtradbinnodb-buffer-pool/
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Buffer Pool SizeInnoDB Buffer Pool is a Key
Component for Optimizing MariaDB
Stores Data and Indexes Keeping in Memory Reduces Disk
IO — a Main Bottleneck
Buffer Pool keeps Frequently Used Blocks in the Buffer — Recently Used (63%) and Older Information (37%)
189
Documentation on InnoDB Buffer Pool: https://mariadb.com/kb/en/mariadb/xtradbinnodb-buffer-pool/
SHOW STATUS LIKE 'innodb_buffer_pool_read%';
+---------------------------------------+----------+
| Variable_name | Value |
+---------------------------------------+----------+
| Innodb_buffer_pool_read_ahead | 8848 |
| Innodb_buffer_pool_read_ahead_evicted | 185 |
| Innodb_buffer_pool_read_ahead_rnd | 0 |
| Innodb_buffer_pool_read_requests | 24549645 |
| Innodb_buffer_pool_reads | 177895 |
+---------------------------------------+----------+
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Slow StartSlow at Start could be from Cold Index
& Data Caches — Try Preloading InnoDB Clustered Index InnoDB without PRIMARY Indexes
For Speed, Try Preloading PK Ranges in Parallel
190
SELECT COUNT(*) FROM table1 WHERE idx_col LIKE '%0%';
SELECT COUNT(*) FROM table1 WHERE non_idx_col=0;
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Temporary Tables on DiskMEMORY Converts to MyISAM on
Disk if Temporary Table Not Possible
Check if Temporary Disk Tables is Incrementing using SHOW STATUS
Execute SHOW PROCESSLIST to find Threads Creating tmp table on disk
191
SHOW STATUS LIKE 'Created_tmp_%';
+-------------------------+-------+ | Variable_name | Value |
+-------------------------+-------+ | Created_tmp_disk_tables | 0 | | Created_tmp_files | 45 | | Created_tmp_tables | 1 | +-------------------------+-------+
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Resolve Disk Temporary TablesMove TEXT and BLOB Columns to
another Table Make Large VARCHAR Columns as Small
as Possible Increase tmp_table_size and max_heap_table_size
192
SHOW VARIABLES
WHERE Variable_Name LIKE 'tmp_table_size' OR Variable_Name LIKE 'max_heap_table_size';
+---------------------+----------+ | Variable_name | Value | +---------------------+----------+ | max_heap_table_size | 16777216 | | tmp_table_size | 16777216 |
+---------------------+----------+
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Maximum Connections ExeededCreating Connections has Little Impact
if Threads are Cached Use Thread Pool for Short-Lived
Workloads Session Buffers Allocated Only when
Needed
193
Documentation on Handling Excessive Connections: https://mariadb.com/kb/en/mariadb/handling-too-many-connections/
SHOW STATUS WHERE Variable_Name
LIKE 'Threads_connected' OR Variable_Name LIKE 'max_used_connections';
+----------------------+-------+ | Variable_name | Value | +----------------------+-------+ | Max_used_connections | 10 | | Threads_connected | 2 |
+----------------------+-------+
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Domain Name ServiceDNS and Reverse DNS Used for IP and
Hostname with Client Connections Slow DNS Server can Slow MariaDB
Client Connection Local DNS Server Improves Lookup
Speed and Security Disable DNS Lookup
Use Only IP Addresses in Permissions Tables
194
[mysqld] skip-name-resolve
Disable DNS Lookup
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
InnoDB Adaptive Hash IndexInnoDB performs like In-Memory
Database with AHI Leads to High CPU Usage
Mutex for Adaptive Hash Index
195
SHOW ENGINE INNODB MUTEX;
+--------+------------------------------+---------------+
| Type | Name | Status |
+--------+------------------------------+---------------+
| InnoDB | &rseg->mutex | os_waits=1 |
| InnoDB | &dict_sys->mutex | os_waits=1 |
| InnoDB | &log_sys->mutex | os_waits=210 |
| InnoDB | &buf_pool->flush_state_mutex | os_waits=5095 |
| InnoDB | &buf_pool->LRU_list_mutex | os_waits=41 |
| InnoDB | combined &block->mutex | os_waits=11 |
| InnoDB | &new_index->lock | os_waits=2 |
| InnoDB | &new_index->lock | os_waits=16 |
| InnoDB | &new_index->lock | os_waits=1 |
| InnoDB | &dict_operation_lock | os_waits=221 |
| InnoDB | &log_sys->checkpoint_lock | os_waits=2920 |
| InnoDB | &btr_search_latch_arr[i] | os_waits=6 |
| InnoDB | combined &block->lock | os_waits=62 |
+--------+------------------------------+---------------+
[mysqld] innodb_adaptive_hash_index=OFF
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Sort BufferMemory Allocated for Sorting
Results Check the sort_merge_passes
Status to get Number of Merge Passes Performed
Set sort_buffer_size to Higher Value if Merge Passes is High
196
SHOW STATUS
LIKE 'sort_merge_passes';
+-------------------+-------+
| Variable_name | Value |
+-------------------+-------+
| Sort_merge_passes | 0 |
+-------------------+-------+
SHOW VARIABLES LIKE 'sort_buffer_size';
+------------------+---------+
| Variable_name | Value |
+------------------+---------+
| sort_buffer_size | 2097152 |
+------------------+---------+
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Troubleshooting
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Error LogContain Startup, Shutdown, Error
Messages Unix uses stderr, Sent to host_name.err in datadir
Some Unix systems Redirect to System Log
Windows uses host_name.err in datadir, or System Event Log
198
Set log-warnings to 2 for Verbose
Documentation on Error Log: https://mariadb.com/kb/en/mariadb/error-log/
Error Log
General Query Log
SQL Query Log
Slow Query Log
Binary Log
mysqld
[mysqld]
log-error = /path/file_name log-warnings = 1
Excerpt from my.cnf or my.ini configuration file
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
General Query LogAll Queries Received from All Clients Order Received — Not Execution Potential Problems with Log
Possible Disk I/O bottleneck on High-Traffic Servers
Log can Quickly become Huge Contains Queries in Plain Text — a
Security Risk
199
Documentation on General Query Log: https://mariadb.com/kb/en/mariadb/general-query-log/
Error Log
General Query Log
SQL Error Log
Slow Query Log
Binary Log
mysqld
Excerpt from my.cnf or my.ini configuration file
[mysqld] general-log
general_log_file='/path/host.log'
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
SQL Error LogLogs SQL Errors
Error Messages SQL Statement
Part of MariaDB Audit Plugin Used for Detecting SQL Injections
Logs User, Host, & Time
200
Documentation on SQL Query Log: https://mariadb.com/kb/en/mariadb/sql_error_log-plugin/
Error Log
Slow Query Log
Binary Log
mysqld SQL Error Log
General Query Log
INSTALL PLUGIN sql_error_log SONAME 'sql_errlog';
Requires INSERT Privilege on mysql.plugin Table
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Binary LogWrites in the Order Executed Useful for Recovering Lost Data Used for Replication Move from to Separate Disk for
Security Set Expiration or Purge Occasionally
201
Documentation on Binary Log: https://mariadb.com/kb/en/binary-log/
Error Log
General Query Log
Binary Log
Slow Query Log
SQL Error Logmysqld
[mysqld] log-bin
expire_logs_days = 7
Excerpt from my.cnf or my.ini configuration file
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Out of Memory (OOM Killer)
202
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Galera Self Leaves
203
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Stack Traces
204
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Find Bug by BacktraceFind Identifying Part Search Issue Tracker (https://jira.mariadb.org)
Confirm Results and Watch Issue Check Comments for Work-Around
205
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Conclusion
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
On-Line ResourcesDocumentation Knowledge Base Open-Source Community
Post Questions on Forums Ask Questions on IRC — FreeNode
207
MariaDB Documentation and Knowledge Base: http://mariadb.com/kb
Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence
Advanced MariaDB for DBAs
Other Training CoursesPublic In-Person Courses Private In-Person Courses Live Virtual Training Courses
208
Training Courses: https://mariadb.learnupon.com/store Upcoming Courses: https://mariadb.learnupon.com/store/sessions
Advanced MariaDB
Introduction
MariaDB Replication
GTIDComplex Scenarios
Semi-Synch
Replication Manager
Client ProxiesOptimizing Overview
Schema Tuning
InnoDB In-Depth Collecting Information
Benchmarking
The Query Process
Query Tuning
Advanced Index UsageCommon Bottlenecks
Troubleshooting
Hacking with GNU Debugger
Conclusion
Thanks for Participating