Deep Dive on MySQL Databases on Amazon RDS Dive on MySQL...Deep Dive on MySQL Databases on Amazon...

Deep Dive on MySQL Databases on Amazon RDS

Chayan Biswas

Sr. Product Manager

Amazon RDS

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Amazon RDS is . . .

Cloud native engine Open source engines Commercial engines

RDS platform

• Automatic fail-over

• Backup & recovery

• X-region replication

• Isolation & security

• Industry compliance

• Automated patching

• Advanced monitoring

• Routine maintenance

• Push-button scaling


Amazon RDS for open source MySQL engines are . . .


RDS platform











Agenda

Why run MySQL

Why run managed MySQL on Amazon RDS

Migrating to RDS

Tips and tricks from one of the world’s largest MySQL operators


1. Popular

2. Innovative

3. Flexible

Image credit: By Mackphillips - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=55946550

https://commons.wikimedia.org/w/index.php?curid=55946550


MySQL is the world’s most popular databaseAll respondents

Source: Stack Overflow Developer Survey Results 2018 ( https://insights.stackoverflow.com/survey/2018/#technology )

https://insights.stackoverflow.com/survey/2018/#technology


MySQL is the world’s most popular databaseProfessional Developers

Source: Stack Overflow Developer Survey Results 2018 ( https://insights.stackoverflow.com/survey/2018/#technology )

https://insights.stackoverflow.com/survey/2018/#technology


“Most popular” buys you . . .

Large ecosystem of ISVs, Tools, Implementation and Support Partners

Highly exercised, stable code

Large community of users and community-driven resources and a larger DBA talent pool


1. Popular

2. Innovative

3. Flexible




MySQL 8.0 highlights - FUNCTIONALITY

• Common Table Expressions

• Window functions

• JSON improvements

• 5108 Spatial Reference Systems

• utf8mb4


Common Table Expressions (CTEs; WITH statement)

WITH derived AS (subquery)

SELECT ...

FROM derived, t1 ...

SELECT ...

FROM (subquery) AS derived, t1 ...

Common Table Expressions

Derived Tables

vs.

• Is more readable

• Can be referenced more than once

• Can be chained

• Has better performance

• Can be recursive


JSON improvements

• JSON_TABLE(): accepts JSON, returns relational table

• ->> (inline path) operator: Equivalent to JSON_UNQUOTE() + JSON_EXTRACT()

• JSON aggregation functions: JSON_ARRAYAGG() and JSON_OBJECTAGG()

• JSON_MERGE_PATCH(): accepts two JSON objects, merges them into one

• Range support in XPath expressions: for example $[1 to 5], $[last]

• Utility functions: JSON_PRETTY(), JSON_STORAGE_SIZE(), JSON_STORAGE_FREE()


MySQL 8.0 highlights - AVAILABILITY

• Instant ADD COLUMN

• Unified, transactional data dictionary

• Crash-safe, atomic DDL


Instant ADD COLUMN

ALTER TABLE table_name [alter_specification], ALGORITHM=INSTANT;

• Only metadata changes are made in the data dictionary

• No need to acquire metadata lock during SE changes and we don’t touch the data of the table

• Currently supports the following operations (with some limitations*):

• Change index option

• Rename table (in ALTER way)

• SET/DROP DEFAULT

• MODIFY COLUMN

• Add/drop virtual columns

• Add columns (non-generated) – also called instant ADD COLUMN

• Fails fast for operations that do not support INSTANT algorithm


Unified, transactional metadata dictionary

• Simplicity• Centralized, uniform data dictionary

• Removal of file-based metadata storage

• Uniform and centralized caching for dictionary objects

• Transactional, crash-safe storage of dictionary data

• Simpler, improved implementation for INFORMATION_SCHEMA tables

• Atomic DDL

Benefits


MySQL 8.0 highlights - PERFORMANCE

• 2x-5x higher performance versus MySQL 5.7

• Hot-spot management

• Descending indexes

• Invisible indexes

• Improved optimizer cost model

• Resource Groups

• Improved replication


2x-5x higher performance versus MySQL 5.7

0

1000

2000

3000

4000

5000

6000

2 2 2 2 2 2 2 2 2 2 2 2 4 4 4 4 4 4 4 4 4 4 4 4 8 8 8 8 8 8 8 8 8 8 8 8 161616161616161616161616323232323232323232323232646464646464646464646464 2

Transactions per second, with varying number of threads

MySQL 5.6 MySQL 5.7 MySQL 8.0

Multi-AZ configuration, r4.16xlarge, 30K IOPS, 1TB database


Resource groups

Dynamically control resource consumption by threads in a group

Currently supports vCPUs as a manageable resource

CREATE RESOURCE GROUP ReportingTYPE = USERVCPU = 5-6THREAD_PRIORITY = 6;

CREATE RESOURCE GROUP BatchTYPE = USERVCPU = 7-8THREAD_PRIORITY = 4;

SET RESOURCE GROUP Reporting FOR 1866;

SELECT /*+ RESOURCE_GROUP(Reporting) */ <col>FROM …

ALTER RESOURCE GROUP ReportingVCPU = 5THREAD_PRIORITY = 19;


MySQL 8.0 highlights – SECURITY, MANAGEABILITY

• Roles

• Password strength

• Open SSL as default TLS/SSL library

• Enhanced observability


MariaDB 10.2 highlights

• Common Table Expressions

• Window Functions

• JSON and GeoJSON functions

• Replication improvements• Delayed replication; throttling; compressed binlog

• Oracle compatibility improvements• Multi-trigger; CHECK constraint expressions; EXECUTE

IMMEDIATE, 38-digit DECIMAL etc.




MariaDB 10.3 highlights

• Oracle compatibility• PL/SQL compatibility parser

• Sequences

• INTERSECT and EXCEPT to complement UNION

• New ROW type and TYPE OF stored functions

• Invisible Columns

• Cursor with parameters

• Temporal data processing

• User-defined aggregates

• ADD INSTANT COLUMN for InnoDB




Oracle compatibility - PL/SQL compatibility parser

Easier migration from Oracle to MariaDB

No need to migrate Oracle PL/SQL to SQLP/PM

Simply SET sql_mode=’oracle’ to create a stored procedure using Oracle PL/SQL syntax

Parser addresses syntax differences between a large subset of PL/SQL and SQL/PSM

• Stored Procedures and Stored Functions

• Cursors

• LOOP

• Variables

• Exceptions

• BEGIN Blocks

Oracle and MariaDB syntax based stored procedures can be used in the same database

• ELSIF, UNIQUE, TRUNCATE

• Functions

• Prepared Statements

• Synonyms for Basic SQL Types

• Packages

• NULL Handling


1. Popular

2. Innovative

3. Flexible




Amazon RDS for open source MySQL engines are . . .


RDS platform











1. Popular

2. Innovative

3. Flexible


Amazon RDS: hundreds of thousands of customers


Popular buys you . . .

Unrivalled operational excellence

Highly exercised, stable code


Who can you trust?

Automated remediation

World’s most experienced operators

Automation


1. Popular

2. Innovative

3. Flexible




Amazon RDS highlights - AVAILABILITY

• Automated, 0-RPO failover across AZs

• Managed x-region replicas for DR

• Automated backups, manual snapshots

• Point-in-time recovery

• Elastic volumes up to 16 TB


Automated, 0-RPO failover across AZs

DNS

Primary Standby

EC2 #1

EC2 #2

EBS #1

EBS #2

Each host manages set of EBS volumes with a full copy of the data

Instances are monitored by an external observer to maintain consensus over quorum

Availability zone 1 Availability zone 2

App



DNS

EC2 #1

EC2 #2

EBS #1

EBS #2



Failover initiated by automation or through RDS API


App

Primary Standby



DNS

Primary

EC2 #1

EC2 #2

EBS #1

EBS #2




Redirection to the new primary instance is provided through DNS


App



DNS

EC2 #1

EC2 #2

EBS #1

EBS #2




Redirection to the new primary instance is provided through DNS


App

Standby Primary


Backups, Snapshots, and Point-in-time restore

TransactionLogs

EBSVol EBS Snapshot

AmazonS3

App

Two options – automated backups and manual snapshots

EBS snapshots stored in Amazon S3

Transaction logs stored every 5 minutes in S3 to support Point in Time Recovery

No performance penalty for backups

Snapshots can be copied across regions or shared with other accounts

Availability zone 1

Availability zone 2

Region 1

EBS vol.

EBS vol.

Region 2

EBS vol.


Amazon RDS highlights – SECURITY, MANAGEABILITY

• IAM DB Authentication


• Automated OS and database upgrades


• Managed binlog replication

• Log upload to CloudWatch Logs

• Recommendations


Recommendations

Example issues flagged by the feature:

• Engine version outdated

• Pending maintenance available

• Automated backups disabled

• Enhanced Monitoring disabled

• Encryption disabled

• …


Amazon RDS highlights – PERFORMANCE

• New! M5 database instance family

• Performance insights


Performance Insights

• Measures DB Load

• Identifies bottlenecks (top SQL, wait events)

• Adjustable time frame (hour, day, week, longer)


1. Popular

2. Innovative

3. Flexible



“All we have to seeIs I don’t belong to youAnd you don’t belong to me.”

George Michael

Freedom! ’90


© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Migration paths to RDS MySQL

On-premise

MySQL/MySQL-compatible

Commercial DBs Non-relational

Heterogeneous

MySQL Tools & Amazon S3 AWS Data Migration Service



Amazon EC2 On-premise Commercial DBs Non-relational

MySQL/MySQL-compatible Non MySQL-compatible

Backup

MySQL Tools

AWS DMS

Recommended UnsupportedSupported Usable



MySQL databases to RDS with Backup

S3MySQL

Backup Managed restore

Replication

Ingestion from S3of Percona XtraBackup files

High performanceFaster restores with binary backups

Binlog replicationfor near-zero downtime migration

EC2 or On-premises





Backup

MySQL Tools

AWS DMS




MySQL databases to RDS with MySQL tools

Source DB S3 Client (EC2 instance)Source-side Client

Logical dumpLogical import

Replication

MySQL tools

• mysqldump / mysqlimport: MySQL-native tools.

• mydumper, myloader: Multi-threaded, flexible

MySQL commands

• SELECT INTO OUTFILE, LOAD DATA INFILE



Client location

Source Database

Client Instance

Logical dump Logical import

Close proximityto database servers

Same VPC and AZas the server

Two client machinesIf the source database is in a different VPC or on-premises



Client instance considerations

• 1 CPU/export or import thread for real-time processing e.g. compression/decompression

• Use EC2 instance types that support enhanced networking

• Ensure client instances and underlying EBS volume can sustain I/O throughput

• Rely on OS diagnostics to detect CPU, storage or network I/O bottlenecks



RDS instance considerations

• Provision enough capacity to ingest or dump data

• 2 CPU/thread for imports, and 1 CPU/thread for exports

• Point-and-click compute scaling through AWS console to adjust compute capacity



Large table exports and imports

• Use mydumper for a multithreaded export of table data.

• For large tables, specify the --rows parameter to break it into chunks

• Use myloader for multithreaded imports

• Adjust the number of threads with the “--thread” parameter for instances larger than r4.2xlarge

--rows : Split table into chunks of these many rows, default unlimited



sudo sh -c 'for x in /sys/class/net/eth0/queues/rx-*; do echo ffffffff, ffffffff > $x/rps_cpus; done'

sudo sh -c "echo 4096 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt"

sudo sh -c "echo 4096 > /sys/class/net/eth0/queues/rx-1/rps_flow_cnt

Client performance tweaks

sudo sh -c "echo 32768 > /proc/sys/net/core/rps_sock_flow_entries"

Enable Receive Packet Steering (RPS)

Enable Receive Flow Steering (RFS)





Backup

MySQL Tools

AWS DMS



AWS Database Migration Service (AWS DMS)

DMS migrates databases to AWS easily and securely with minimal downtime. It can migrate your data to and from most widely used commercial and open-source databases –and for as little a $3 for TB DB.

Amazon Aurora



Database migration processStep 1: Convert or Copy your Schema

Source DB

AWS SCT

Native Tool

Step 2: Move your data

Source DB AWS DMS

Data



How does DMS work?

Customer

premises

Application users

AWS

Internet

VPN

• Start a replication instance

• Connect to source and target databases

• Create Tasks

• Let AWS DMS create tables, load data, and keep them in sync

• Switch applications over to the target at your convenience

AWS

DMS


6 frequently asked questions . . .

• Which instance family should I choose?

• When should I use Multi-AZ vs. Read Replicas?

• When should I use automated backups vs. snapshots?

• How do I secure my database?

• How do I monitor my database?

• Can I know when service events happen?


Which instance family should I choose?

M2/M4/M5

• General purpose

• 2vCPU, 8GiB RAM to 96vCPU, 384 GiB RAM

• High perf. networking

R3/R4

• Memory optimized

• 2 vCPU, 16 GiB RAM to 64 vCPU 488 GiB RAM

• High perf. networking

T2/T3

• Burstable

• 1vCPU, 1GiB RAM to 8vCPU, 32 GiB RAM

• Moderate networking

Dev/test and smaller workloads CPU intensive workloads (for example, WordPress)

Query intensive workloads with high connection count


When should I use Multi-AZ vs. Read Replicas?

Read-replica

• General purpose

• Replica can serve read workload

• No backups configured by default

• Can be within AZ, cross-AZ, or cross-region

• DB engine upgrades independent of primary

• Manual promotion to standalone database

Multi-AZ

• Synchronous replication – highly durable

• Secondary is idle

• Backups can be taken from secondary

• Always in two AZs within a Region

• DB engine upgrades on primary

• Automatic failover when issue is detected


When should I use backups vs. snapshots?

Manual snapshots

• Create manually via AWS console, CLI or API

• Kept until you delete them

• Restores to saved snapshot

• Good for data-retention compliance, checkpoint before making large changes, spawning test databases, and final copy before deleting a database

Automated backups

• Specify retention window (7 day default)

• Kept until out of window (35 day maximum) or instance is deleted

• Supports Point in Time Restore

• Good for disaster recovery


How do I secure my database?

Amazon VPC: network Isolation

AWS IAM: resource-level permission controls

AWS KMS: encryption at rest

SSL: protection for data in transit


How do I monitor my database?

Amazon CloudWatch metrics: CPU, storage, memory, swap usage, I/O, latency, throughput, replica lag

Amazon CloudWatch logs: Audit log, General log, Slow-query log, and Error

Amazon CloudWatch alarms: Similar to on-premises monitoring tools

Enhanced monitoring: 50+ CPU, memory, file system and disk I/O metrics

Other: Performance Insights, 3rd party tools


Can I know when service events happen?

Amazon SNS: email, text, call to HTTP endpoint

Amazon CloudWatch Events: run AWS Lambda or Amazon ECS tasks.

6 different source types: DB Instance, DB Parameter Group, DB Security Group, DB Snapshot DB Cluster, DB Cluster Snapshot

17 different event categories: availability, backup, configuration change, etc.

Thank you!

Chayan Biswas

[email protected]

Rate My Session

Deep Dive on MySQL Databases on Amazon RDS Dive on MySQL...Deep Dive on MySQL Databases on Amazon...

Documents

Transcript of Deep Dive on MySQL Databases on Amazon RDS Dive on MySQL...Deep Dive on MySQL Databases on Amazon...