MySQL Cluster: An introduction - · PDF fileComponents of a Cluster MySQL (mysqld) NDB PHP...

MySQL Cluster: An introduction

Geert Vanderkelen

2006-06-24

MySQL AB

Agenda

• Quick MySQL Introduction• Overview• Cluster components• High Availability and Scalability• Small example: Web Sessions• New features• Q&A

Who am I?

• Geert Vanderkelen– Support Engineer for MySQL AB– MySQL Cluster– Belgian, based in Germany

MySQL Quick Introduction

Quick intro to MySQL

• MySQL is a DBMS running on most OS• Reputation for speed, quality, reliability and easy to

use• Storage Engines (MyISAM, InnoDB, ..)• Current GA is 5.0:

– SQL 2003 Stored procedures– Triggers, Updatable Views, Cursors– Precision math– Data dictionary (INFORMATION_SCHEMA database)– and more..

• Lots of Connectors and API available

MySQL Architecture

Connection Pool(Authentication, limits, caches,..)

API(PHP,Java,C/C++,.NET,..)

SQL Interface

DML/DDLStored Procedures, Triggers, Views,..

Parser

SQL Translation,Object Privileges

Optimizer

Execution Plan, Statistics

Caches & Buffers

Query Cache, global & per engine

Storage Engines

MyISAM InnoDB Memory Federated NDB

(custom)ArchiveCVS BlackholeMerge

MySQL Cluster Overview

What is MySQL Cluster?

• In-memory storage– data and indices in-memory– check-pointed to disk

• Shared-Nothing architecture• No single point of failure

– Synchronous replication between nodes– Fail-over in case of node failure

• Supports transactions– READ COMMITTED only

• Row level locking• Hot backups• Online software upgrade

Background

• Designed for telecom environment• 99.999% Availability• Performance

– reponse time about 5-10ms– 10000 transactions/sec

• Scalability– many concurrent applications– load balancing

Cluster components

Cluster Nodes

• Participating processes are called ‘nodes’– Nodes can be on same computers

• Two tiers in Cluster: – SQL layer

• SQL nodes (also called API nodes)– Storage layer

• Data nodes• Management nodes

• Nodes communicate through transporters:– TCP (most common)– Shared memory– SCI (Scalable Coherent Interface)

Components of a Cluster

MySQL(mysqld)

Application.NetJBossPHP

Management(ndb_mgmd)

SQL Nodes

Data Nodes

Management Nodes

ndbd ndbd

ndbdndbd

Data Nodes

MySQL(mysqld)

Sql Layer

Storage Layer

• Contain data and index• Used for transaction coordination• Each data node is connect to the others• Shared-nothing architecture• Up to 48 data nodes

Data nodes

SQL nodes

• Usually MySQL servers• Also called API nodes• Each SQL node is connected to all data nodes• Applications access data using SQL• Native NDB application (e.g. ndb_restore)• Client application written using NDB API

Management nodes

• Controls setup and configuration• Needed on startup for other nodes• Cluster can run without• Can act as arbitrator during network partitioning• Cluster logs• Accessible using ndb_mgm CLI

ndb_mgm CLI

[geert@ndbsup-1 5.1bk]$ ./bin/ndb_mgm-- NDB Cluster -- Management Client --ndb_mgm> showConnected to Management Server at: ndbsup-priv-1:1406Cluster Configuration---------------------[ndbd(NDB)] 2 node(s)id=4 @10.100.9.8 (Version: 5.1.12, Nodegroup: 0, Master)id=5 @10.100.9.9 (Version: 5.1.12, Nodegroup: 0)

[ndb_mgmd(MGM)] 2 node(s)id=1 @10.100.9.6 (Version: 5.1.12)id=20 (not connected, accepting connect from ndbsup-priv-2)

[mysqld(API)] 4 node(s)id=10 @10.100.9.6 (Version: 5.1.12)id=11 (not connected, accepting connect from ndbsup-priv-2)id=12 (not connected, accepting connect from any host)id=13 (not connected, accepting connect from any host)

ndb_mgm>

Physical Requirements

• Need at least 3 machines• Data nodes

– need lots of memory– ndb_size.pl (http://forge.mysql.com)– not CPU bound, single threaded– MySQL 5.1 brings disk based

• SQL/API nodes– usually more than data nodes– more CPU bound, multi threaded

• Dedicated network– TCP/IP communication used– separate data node traffic– protected from outside!

High Availability & Scalability

High Availability and Scalability

• Data Distribution– Fragmentation– Synchronous Replication

• Failure handling:– for MySQL nodes– for Data nodes– for Management nodes

• Backup– hot backups– restoring

• Cluster replication

Data Distribution

Fragmentation & Replication

• Fragmentation (or partitioning)– tables fragmented horizontally– if we have 4 data nodes, data fragmented in 4 parts– primary node for one fragment

• Replication– each node has secondary fragment (replica)– synchronous: committed everywhere or not at all– nodes having same data are grouped

Data Fragmentation: 2 nodes, 2 replicas

Horizontal partitioningof table into 2 parts

Node Group 1

ndbd 1

ndbd 2

Parts distributed ondata nodesFx - primary replicaFx - secondary replica

Data Fragmentation: 4 nodes, 2 replicas

F1Horizontal partitioningof table into 4 parts

Node Group 1 Node Group 2

Parts distributed ondata nodesFx - primary replicaFx - secondary replica

ndbd 1

ndbd 2

ndbd 3

ndbd 4

Host A

ndbd 1

ndbd 3

Host B

ndbd 2

ndbd 4

Failure Handling

A Configuration[ndbd default]NoOfReplicas = 2DataMemory = 400MIndexMemory = 32MDataDir = /var/lib/mysql/cluster

[ndb_mgmd]DataDir = /var/lib/mysql/clusterHostName = 192.168.0.42

[ndbd]HostName = 192.168.0.40

[ndbd]HostName = 192.168.0.41

[mysqld]HostName = 192.168.0.42

[mysqld][api]

MySQL+ ndb_mgmd

ApplicationApplication Application

Data NodesNode Group 1

ndbd ndbd

Failure: MySQL server

• Applications can use other• mysqld reconnects

MySQL+ ndb_mgmd

ndbd ndbd

Failure: Data Node

• Other data nodes know• Transaction aborted• Min. 1 node per group needed• 0 nodes in group = shutdown MySQL

MySQL+ ndb_mgmd

ndbd ndbd

Failure: Management Node

• Continued operations• Needed for startup• Best to have 2• Can run on MySQL Nodes MySQL

NDB API

MySQL+ ndb_mgmd

NDB API

Data Nodes

ndbd ndbd

Node Group 1

Backups

• Hot Backups– no interruption– backups made at same point– locally on each data node– using ndb_mgm CLI– backups can be centralized

• Restoring– using ndb_restore application– can be done from any API node– first 1 time meta information, then data– needs single user mode– should never be needed

Scalability

• Total of 64 nodes• Max. 48 data nodes• Typical setup:

– 2 ndb_mgmd (management)– 4 data nodes– Up to 58 MySQL server (SQL nodes)

• Adding nodes need restart– adding data nodes can not be done online

Cluster Replication

• MySQL 4.1/5.0– statement based replication– all updates to data should go to 1 MySQL server– updates from NDBAPI application ignored

• MySQL 5.1– row based replication– updates can go to all MySQL servers– injector thread– NDBAPI applications not ignored– multiple replication channels

Cluster Replication in 5.1

Data NodesNodeGroup 1

ndbd ndbd

MySQL(master)

NDBBinLog

Application

BinLog

mysql> UPDATE...

mysql> INSERT...update()

MySQL(slave)

Data NodesNodeGroup 1

ndbd ndbd

BinLog

replication

MySQL(slave)

replication

Cluster Usage Example

Web Sessions

• Use in web applications– shopping carts– personalized web pages– tracking of users

• Sample table for PHP sesssions

CREATE TABLE `store`.`sessions` ( `id` CHAR(32) PRIMARY KEY, `data` TEXT, `usetime` TIMESTAMP , ) ENGINE = InnoDB;

Web Sessions

MySQL(mysqld)

Apache/PHP

Load balancer

• Without Cluster– One MySQL server holding data– Single point of failure

Web Sessions using Cluster

• Changing existing session table:

• No changes to the web application• Each webserver runs a MySQL server• Storing Gigabytes of session data• Using inexpensive hardware

ALTER TABLE `store`.`sessions` ENGINE = NDBCLUSTER;

Web Sessions

Apache + PHPmysqld

Load balancer

• Wit Cluster– MySQL distributed– No Single point of failure– Shared storage, but redundant

Apache + PHPmysqld

ndbd ndbd

ndbdndbd

Some New Features in 5.1

• Disk based storage (none indexed data)• Auto-discovery tables and databases• Cluster replication• Variable-sized attributes (save space!)• User defined partitioning• Online add/drop index• Better documentation

Get ready to Cluster!

• Docs: http://dev.mysql.com/manual/• Mailinglist: cluster@lists.mysql.com• Download: http://dev.downloads/mysql/5.0.html

– or go latest MySQL 5.1 (beta currently)• Standard binaries ‘mysql-max’ version• RPM based:

– Linux x86 generic RPM (dynamically linked)– MySQL-Max– The MySQL-ndb* packages

• From source:– ./configure --with-ndbcluster

And Thanks!

MySQL Cluster: An introduction - · PDF fileComponents of a Cluster MySQL (mysqld) NDB PHP...

Documents

Transcript of MySQL Cluster: An introduction - · PDF fileComponents of a Cluster MySQL (mysqld) NDB PHP...

MySQL Reference Manual - HZDR1 General Information About MySQL MySQL.

Why Use MySQL · 2.MySQL Cluster Community Edition 3.MariaDB/Percona Server • Commercial 1.MySQL Classic Edition 2.MySQL Standard Edition 3.MySQL Enterprise Edition 4.MySQL Cluster

The Galaxy eSolutions Opportunitywhitepaper.galaxy-esolutions.com/GES_Slide_Deck.pdf · NDBD: Currently a B2C (Business-to-Consumer) site evolving to become a marketplace with approved

MySQL 8 : The Best gets even better · 2018-07-27 · MySQL 5.7 MySQL InnoDB Cluster - MySQL Group Replication - MySQL Router - MySQL Shell MySQL 8.0 - NoSQL Document Store - JSON

IP Addressing/Subnetting Simplified6 Nodes 23 Nodes 19 Nodes 17 Nodes 14 Nodes 10 Nodes. VLSM Exercise 1 192.168.20.0 /24. Internet. 14 Nodes

Administrar MySQL Server con MySQL Workbench, crear ...yolandaam.com.es/images/mysql/02-Administrar MySQL Server con M… · Administrar MySQL Server con MySQL Workbench, crear usuarios,

Clueless Nodes to Network-Cognizant Smart Nodes: Achieving

MySQL, PHP, Stuff - Jeremy Zawodnyjeremy.zawodny.com/mysql/phpcon/2003/MySQL-PHP-Stuff.pdf · MySQL, PHP, Stuff PHPCon East 2003 ... MySQL 4.0 • Query optimizer improvements ...

Upstream Downstream - FOSDEM · HA Audit MySQL 5.6 MySQL Workbench 6.1 MySQL Utilities MySQL Applier for Hadoop MySQL Workbench 5.2 & 6.0 MySQL Enterprise Oracle Certifications Windows

Ndbd Formation, en quelques motsformation.ndbd.fr/wp-content/uploads/2020/04/Ndbd... · 05 59 29 83 18 / 06 36 07 56 48 > Maîtriser les ressources à disposition de l’action publique.

MySQL - kuliah.unnes.ac.idkuliah.unnes.ac.id/~hardy/mysql/MySQL-komlan.pdf · mysql>flush privileges ! Reloads the privileges from the grant tables in the database mysql ! An alternative

MySQL Replication Update MySQL 5.5 (GA) & MySQL 5.6.2 (Dev ...akipp.se/larsthalmann.com/110412.thalmann.MySQL_UC_2011.pdf · MySQL Replication Update MySQL 5.5 (GA) & MySQL 5.6.2

Datastore Axes Choosing your scalability direction · 2019. 2. 26. · MySQL – many servers each with all data, read only Clusters – more nodes in cluster Increased copy number

MySQL Introduction to the MySQL products. Agenda Company Overview Open Source & MySQL Momentum Why MySQL? MySQL OEM, Community & Enterprise offerings.

Using MySQL for Big Data Advantage - · PDF fileUsing MySQL for Big Data Advantage Integrate for Insight ... MySQL Cluster 7.2 MySQL Cluster Manager 1.4 MySQL Utilities 1.0.6 MySQL

MySQL Replication: Latest Developments · MySQL 5.1.30 row-based replication MySQL 4.0.12 MySQL 4.1.7 MySQL 5.5.8 semi-sync replication MySQL 5.6.10 crash-safe replication metadata

MySQL Strategy - itreegroup.eu · MySQL Strategy Morten Andersen, MySQL Enterprise Sales . ... MySQL Workbench 6.2 MySQL Database 5.7 DMRs * MySQL Cluster 7.4 DMR *Development Milestone

MySQL 5.7入門 インストール、アーキテクチャ基礎編 Database 5.6 All GA! MySQL Cluster 7.3 1.3 MySQL Enterprise Monitor 3.0 MySQL Fabric MySQL Workbench 6.3 MySQL Cluster

MySQL HA - heinlein-support.de · Percona Xtrabackup: konsistente Backups ohne Downtime. 7 MySQL HA - Grundsätzliches Flavours of MySQL. 8 MySQL HA - Grundsätzliches MySQL != InnoDB

MySQL開発最新動向 Oracle MySQL Cloud Serviceの最新情報もご … · 2019-08-06 · MySQL InnoDB Cluster (GA) - MySQL Group Replication - MySQL Router - MySQL Shell MySQL

MySQL 5.7入門インストール、アーキテクチャ基礎編 Database 5.6 All GA! MySQL Cluster 7.3 1.3 MySQL Enterprise Monitor 3.0 MySQL Fabric MySQL Workbench 6.3 MySQL Cluster