Post on 30-May-2020
© 2017 ISC
ISC's Performance Lab
DNS-OARC Workshop 26 - Madrid
1
© 2017 ISC
Goals of the Project
▪ Provide a framework for automated performance testing of BIND– Test experimental code– Look at long-term trends to avoid
regressions
2
© 2017 ISC
System Features #1
▪Multiple configurations– git branch or tag– ./configure settings– named.conf options– named command line options– etc
3
© 2017 ISC
System Features #2
▪Multiple authoritative zone configurations– root zone– 1M delegations (small TLD) – 1M small zones (web hoster)– 1 zone with 1M A+AAAA
▪ dnsperf query sets for the above▪ (Some) Recursive support4
© 2017 ISC
System Features #3
▪ HTML5 Web UI– real time updates over WebSockets
▪ Round-robin scheduler–with ad-hoc priority bump
5
© 2017 ISC
System Features #4
▪ Graphs▪ CSV Output▪ A/B comparison with Student's t-test▪ Extensible for other servers▪ NSD 4, Knot 2 already supported▪ Dummy 'echo' server support, too
6
© 2017 ISC
Technology
▪MongoDB ▪ NodeJS (ES6)–Promises– Express
▪ HTML5–Bootstrap–Angular JS (ES5)–WebSockets
7
© 2017 ISC
QUICK DEMO…
© 2017 ISC
Logical Architecture
DNS Server
Web Client
dnsperfREST API
Server
Test Agent
MongoDB
9
© 2017 ISC
Physical Architecture
Server under Test
Traffic Generator Backend
10 GE
Gig-E Management LAN
Apache Reverse Proxy
10
© 2017 ISC
Test Methodology
▪ Build and start server▪ Run dnsperf 30 times▪ 30 seconds each time▪ first run ignored for statistic (allows
for cache and buffer warmup)
11
© 2017 ISC
Challenge - Test Variability
▪ The results from 30x dnsperf tests against a single run of BIND are generally consistent
▪ Testing a new run of BIND often shifts the mean significantly, by several standard deviations
12
© 2017 ISC
Challenge - Test Variability
13
Master Branch Test Variance Detail
610
615
620
625
630
635
640
kQPS
© 2017 ISC
(Some) Mitigation
▪ Tune for stability, not peak performance:–Disable Hyperthreading– Lock CPU clock rate•Disable Intel SpeedStep•Disable Turbo mode
– Lock Client and Server CPU core affinity– Lock NIC RX/TX queue CPU core affinity– Set NIC queue flow hashing to use
deterministic ports14
© 2017 ISC
Challenge - Test Variability #2
15
Dual Core Variance Detail
660
680
700
720
740
760
780
800
kQPS
●
●
●
●●
●●
●
●
●●●
●
●
●●
●
●
●●●
●
●●●
●●
●
●●
●
●
●●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●●
●●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●●
●●
●
●
●●
●●
●
●●
●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●●●
●●
●●
●
●
●
●
●●
●
●
●
●●●
●
●●●●●
echo server two threads locked to CPU#0 and #1
© 2017 ISC
Challenge - Test Variability #2
16
Dual Core Variance Detail
660
680
700
720
740
760
780
800
kQPS
●
●
●
●●
●●
●
●
●●●
●
●
●●
●
●
●●●
●
●●●
●●
●
●●
●
●
●●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●●
●●
●●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●●
●
●●
●●
●●
●●
●
●
●●
●●
●
●●
●
●
●
●●●
●
●
●
●
●
●●●
●
●
●
●●●
●
●●
●
●
●
●
●
●
●
●
●
●●●●
●
●
●
●
●
●
●●
●●
●
●●
●●
●
●●
●
●
●
●●●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●●●
●●
●●
●
●
●
●
●●
●
●
●
●●●
●
●●●●●
echo server two threads locked to CPU#0 and #1
Huh?!
© 2017 ISC
Coming Soon
▪ In 9.11.2 and back-ported to 9.10– optimised name compression– optimised owner case preservation
▪ In 9.12– "minimal-responses" on by default– "glue cache" replaces "acache"• improves delegation performance
17
© 2017 ISC
2012 2013 2014 2015 2016 2017 2018
0
200
400
600
800
BIND Performance − 1M Delegations
12−core Intel Xeon E5−2680 v3 @ 2.50GHz
Release Date
kQPS
9.9 9.10 9.11 dev
●●
● ● ● ●
● ● ●
● ●
●
● ●●
●
●
● ●●
●
18
© 2017 ISC
2012 2013 2014 2015 2016 2017 2018
0
200
400
600
800
BIND Performance − 1M Delegations
12−core Intel Xeon E5−2680 v3 @ 2.50GHz
Release Date
kQPS
9.9 9.10 9.11 dev
●●
● ● ● ●
● ● ●
● ●
●
● ●●
●
●
● ●●
●
19
Change to default "-U" setting
© 2017 ISC20
2012 2013 2014 2015 2016 2017 2018
0
100
200
300
400
BIND Performance − Root Zone
12−core Intel Xeon E5−2680 v3 @ 2.50GHz
Release Date
kQPS
● ● ●●
● ●
● ● ●
● ●
● ● ● ●
● ●● ●
9.9 9.10 9.11
© 2017 ISC
2012 2013 2014 2015 2016 2017 2018
0
100
200
300
400
BIND Performance − Root Zone
12−core Intel Xeon E5−2680 v3 @ 2.50GHz
Release Date
kQPS
9.9 9.10 9.11 dev
● ● ●●
● ●
● ● ●
● ●
● ● ● ●
● ●● ●●
●
21
6x faster!
© 2017 ISC
How Do I Get It?
https://github.com/isc-projects/perflab▪ This is not supported software▪ It's not turn-key - installation is likely
to require lots of fiddling▪ Pull requests welcomed…
22
© 2017 ISC
QUESTIONS?