MMUG18 - MySQL Failover and Orchestrator

MMUG18

MySQLFailoverandOrchestratorSimonMudd [email protected]

17th May2017

GraciasaTuenti

117/05/2017 MadridMySQLUsersGroup- MMUG18

• Por permitir eluso desus oficinas paraesta presentación

Content

• HandlingfailoverwithMySQL• Downtime&Requirements• MySQLClusteringsolutions• Non-clusterings solutionsandconsiderations

• Orchestrator• Questions


IsDowntimeAcceptable?

• Doyouhaveasystemthatneedstorun24x7?• Noteveryonedoes• Ifyouhaveawebsitethengenerallydowntimeisnotacceptable


Requirements

Goal:Run24x7x365withno downtime• Isthisreallynecessary?• Ifyouaskmanagementthey’llalwayssayyes…• Whatisthecost?• Shorterdowntimerequirementsmeanmoreeffortspenttoachievethat• Howdoyoureliably detectfailure?Hardproblemtosolve

Ifyouacceptdowntimehowmuchcanyoureallytolerate?• 1s,5s,30s,1min?


Whatoptionsareavailable?

• MySQLCluster• carriergrade• veryhighuptime• NotInnoDB – specialised workloads

• Galera• Oftenwithasynchronousreplicationbetweendatacentres

• InnoDB Cluster• Verynew

• Allrequireclientstotakeactiononfailureofanode• Ifyouuseaproxythatcanfailtoo…



“Clustersolutions”• Donotworkwellcross-DCduetolatency• Ifyouacceptwritesintomultiplemastersthere’sachanceofconflict• Slowsthingsdown• InnoDB Clusternowdoesnotrecommendthisbehaviour – requirescare

• Onlysmallsetupsworkinasingledata-centre soadaptationhereisalsoneeded• Clustersetupsdonotscaleeasilyto10ormoreservers



• StandardMySQL,MariaDB,AmazonRDS,GoogleCloudSQL,…• Readscale-out• Asynchronousreplication• Semi-synchelpsimproveperformanceandensuredatais“somewhereelse”whenacknowledgingatransaction

• Ifyouareoutofthecloudthen:differentsetups• SBRorRBR?• NoGTID,OracleorMySQLGTID?• Optionalsemi-sync?

• Ifyouareoutofthecloudthen:doityourself• MHA• MariaDB ReplicationManager• Orchestrator


Orchestrator


Orchestrator

• Handlesmasterfailover,butmore…• GUItomanageandvisualise topology– veryhandy• CLItodothesamethings– goodforscripting• APIcallstorunatadistance(moregenericinterface)• NeedsaDBbackendtostorestate.• NormallyMySQLbutcanbeSQLite


Orchestrator

Whatfailuresdoesithandle?• Masterfailures– needstotalktoexternalsystems• Intermediatemasterfailures– canhandleonitsown• Doesnot careaboutslavesorapplications• WorkswithGTID:OracleorMariaDB• WorkswithoutusingGTID:CanaddPseudo-GTID (eventsinjectedonthemasterareusedtofindamatch)sononeedtomigratetoGTIDifnotwanted• Handlesmulti-leveltopologies


OrchestratorGUI


OrchestratorCLI

Over100commandsyoucanuse• E.g.

• relocate• discover• begin-downtime,end-downtime• topology


OrchestratorCLI

17/05/2017 MadridMySQLUsersGroup- MMUG18 15

FailureNotifications

• Usingthehookscantalktojabberoremailtoadviseoftheactionstaken:


FailureAuditing


OrchestratorSetup

• Sourceatgithub.com/github/orchestrator• Binarieswritteningo• Daemonrunswebserviceanddiscovery,clientoneachMySQLserver• StatestoredinMySQL/SQLite• Singlejson configurationfile:/etc/orchestrator.conf.json

• Howtoreachbackenddatabase(storesstate)• Howtorecognise delay• Mostdefaultsaregoodtogetyougoing• Whichsystemsyouwanttotriggerrecoveryon• Hookstohandlerecovery(talkingtoexternalsystems)• Ifyouneedhelppleaseask


OrchestratorCharacteristics

• Discover oneserverinyourclusterandorchestratorwillfindtheothers• Detectsnewserversintheclusterautomatically• Notifiesyouofproblemsseen• Recoveryisoptional(percluster)• Optionalselectionofcandidatemastersorserverstoblacklist• GlobalON /OFF switch– handyifseveralfailureshappenatonce• ForparanoidDBAs,sofarorchestratorhasalwaysdonetherightthing


OrchestratorHA?

OrchestratorcanberuninHAmode• Multipledaemonswillco-operatesoifonefailsanotheronetakesover(theysharethedatabasebackend)• UsealoadbalancertoprovideanHAGUIservice• Usenginx (orsimilar)forauthenticationandTLSifneeded• Upgradesareeasier• ReplicatetheorchestratorMySQLbackendtonotlosedata


DoesitScale?

Yes• Booking.com hasalargeinstallationwithasingleclustermonitoringthousandsofMySQLservers• RecommendedbyYouTubeformanagingVitess servers• Quiteanumberofotherusersbuttheyarenotveryvisible


Futurework

• Simplifyconfigurationandsetupsomorepeoplecanuseit• Improvescalability• MakeitworkonAmazonRDS• Spreadtheword…


Furtherhelpneeded?

• github.com/github/orchestrator• forIssues(Problems/Questions)andPullRequests(patches)

• GoogleGroup:OrchestratorMySQL• https://groups.google.com/forum/#!forum/orchestrator-mysql

• FeelfreetocontactmeandIwilltrytohelpprovidepointers


Oh,andBooking.com ishiring!

• Almostanyrole:• MySQLEngineer/DBA• SystemAdministrator• SystemEngineer• SiteReliabilityEngineer• Developer• Designer• TechnicalTeamLead• ProductOwner• DataScientist• Andmanymore…

• https://workingatbooking.com/


Questions

?


MMUG18 - MySQL Failover and Orchestrator

Technology

Transcript of MMUG18 - MySQL Failover and Orchestrator