A Support Engineer Walkthrough on pt- stalk - Percona · 1 © 2017 Percona Marcos Albe A Support...
Transcript of A Support Engineer Walkthrough on pt- stalk - Percona · 1 © 2017 Percona Marcos Albe A Support...
©2017Percona1
MarcosAlbe
ASupportEngineerWalkthroughonpt-stalk
PrincipalSupportEngineer- Percona
©2017Percona2
WhoisSpeaking?• Marcos Albé
- Principal Support Engineer @ Percona
- 7 years here- Focus on performance and observability
©2017Percona3
Agenda
©2017Percona4
Agenda
1. WhatdoweUSE
2. Who’sstalkin’?
3. Howpt-stalkcanbeUSEd
4. Walkthrough
©2017Percona5
WhatdoweUSE
©2017Percona6
Weuseoneofthese...
WhatdoweUSE
©2017Percona7
Utilization
WhatdoweUSE
©2017Percona8
UtilizationSaturation
WhatdoweUSE
©2017Percona9
Utilization
SaturationErrors
WhatdoweUSE
©2017Percona10
WhatdoweUSE
©2017Percona11
Who’sstalkin’?
©2017Percona12
Who’sstalkin’?
check out those resources!
©2017Percona13
Who’sstalkin’?•Aspersa stalk+Aspersa collectinone!(thanksDaniel&Baron)•#Thename'stalk'isbecause'watch'isalreadytaken,and'stalk'isfun.•Nowpartofpt-toolkit•Monitorsvariableorascript’ssingle-valuedoutput•Triggerscapturewhensetthresholdisreached- filesfrom/proc:diskstats,interrupts,meminfo,slabinfo,procstat,procvmstat.- vmstat,iostat,top,netstat,pmap,mpstat,sysctl,df,hostname,mysqladmin ext,mysqladmin debug,lsof).- SHOWENGINEINNODBSTATUS,SHOWENGINEINNODBMUTEX,SHOWGLOBALVARIABLES,SHOWFULLPROCESSLISTandSHOWOPENTABLES.
•Productionsafe!•Customtriggersandcustomcaptures•Companiontools(pt-sift,pt-mext,pt-diskstats,pt-summary);Bashdissection
©2017Percona14
Who’sstalkin’:commonparameters•Stalkingpart- --daemonize |--no-stalk- --function=status- --interval=1- --variable=Threads_running- --threshold=25- --cycles=5•Collectingpart- --run-time=30- --sleep=300- --iterations=- --plugin=- --dest=/var/lib/pt-stalk
•Valuesshownarethedefaultsforeach(wheregiven)
©2017Percona15
Who’sstalkin’:workflow
©2017Percona16
Who’sstalkin’:workingexamplesICollectdataduring60secondsfor3consecutivetimes,startingimmediately.
pt-stalk--no-stalk\--dest=$HOME/percona \--user=root\--ask-pass\--run-time=60--sleep=60--iterations=3
©2017Percona17
Who’sstalkin’:workingexamplesIIRunasadaemonandtriggercollectionifInnoDB hasmorethan5pendingfsyncs.
pt-stalk--daemonize \--function=status\--variable=Innodb_data_pending_fsyncs \--threshold=5\--cycles=2\--interval=1\--iterations=5\--plugin=/usr/local/bin/raid_card_status.sh
©2017Percona18
Who’sstalkin’:customtriggers#!/bin/bash# Save this bash snippet as /root/delayed_slave_trg.sh;trg_plugin() {
mysql $EXT_ARGV -Ee "SHOW SLAVE STATUS" | \grep "Seconds_Behind_Master" | \awk "{ print $2 }";
}#uselikept-stalk --function=/root/delayed_slave_trg.sh --etc --etc
©2017Percona19
Who’sstalkin’:customcapture#!/bin/bash# save as /root/collect_perf.shbefore_collect() {
# run perf stat; match sleep length to --run-time
/usr/bin/perf stat --all-cpus \--output=${d}/${p}-stat_a \-- sleep ${OPT_RUN_TIME};
}#uselikept-stalk --plugin=/root/collect_perf.sh --etc --etc
©2017Percona20
Howpt-stalkcanbeUSE’d
©2017Percona21
Howpt-stalkcanbeUSE’d
Linu
x K
erne
l CPU
CPUInterconnect
DRAM
I/O Bridge
I/O Controller Network Controller
I/O Bus
ExpanderInterconnect
Disk Disk Swap Port Port
Applications
System libraries
System Call Interface
VFS SocketsScheduler
File Systems TCP/UDP
Volume Manager IP Virtual Memory
Block Device Interface Ethernet
Device Drivers
Memory Bus
Interface Transports
©2017Percona22
Linu
x K
erne
l CPU
CPUInterconnect
DRAM
I/O Bridge
I/O Controller Network Controller
I/O Bus
ExpanderInterconnect
Disk Disk Swap Port Port
Applications (MySQL!)
System libraries
System Call Interface
VFS SocketsScheduler
File Systems TCP/UDP
Volume Manager IP Virtual Memory
Block Device Interface Ethernet
Device Drivers
Memory Bus
Interface Transports
ps
df
/proc/diskstats
SHOWENGINEINNODBSTATUS
/proc/interrupts
iostat
lsof /proc/meminfo
mpstat
SHOWENGINEINNODBMUTEX
mysqladminext
SHOWOPENTABLES
pmap
/proc/procstatSHOWFULLPROCESSLIST
/proc/vmstat/proc/slabinfo
sysctl
SHOWENGINETOKUDBSTATUS
top
SHOWGLOBALVARIABLES
vmstat
SHOWSLAVESTATUS
netstat
strace
tcpdump
gdb(backtraces)
oprofile
©2017Percona23
Howpt-stalkcanbeUSE’d:HereBeDragons!
strace tcpdump
gdb(backtraces)oprofile
©2017Percona24
Awalkthrough
©2017Percona25
AwalkthroughUsingpt-stalktosolveproblemswith...●...HighCPU●...HighIO●...Network
©2017Percona26
Demo
©2017Percona27
GetYourTicketsforPerconaLiveEurope!
ChampioningOpenSourceDatabases▪ MySQL,MongoDB,OpenSourceDatabases▪ TimeSeriesDatabases,PostgreSQL,RocksDB▪ Developers,Business/CaseStudies,Operations▪ September25-27th,2017▪ RadissonBlu RoyalHotel,Dublin,Ireland
LastYear’sConferenceSoldOut!ReserveyourspotASAP.
©2017Percona28
TalktoPerconaExpertsatAWSRe:Invent!DatabasePerformanceforCloudDeployments▪PerconaSupportandManagedServices• AmazonRDS,Aurora,RollYourOwn•MySQL/MariaDB/MongoDB• Reducecostsandoptimizeperformance
▪PerconaMonitoringandManagementDemos• Point-in-timevisibilityandhistoricaltrendingofdatabaseperformance• Detailedqueryanalytics
▪Booth#1138