balancers with HAProxy and Python Intent-driven, fully ...
24
Intent-driven, fully automated deployment of anycasted load balancers with HAProxy and Python DENOG 11 Maximilian Wilhelm 1 / 24
Transcript of balancers with HAProxy and Python Intent-driven, fully ...
Intent-driven, fully automated deployment of anycasted loadbalancers with HAProxy and Python
DENOG 11
Maximilian Wilhelm
1 / 24
Agenda1. Who's who2. Context3. The past4. The Idea5. The now6. Q & A
2 / 24
Who's who Maximilian WilhelmNetworkerOpenSource HackerFanboy of
(Debian) Linuxifupdown2
Occupation:By day: Senior Infrastructure Architect, Uni PaderbornBy night: Infrastructure Archmage, Freifunk HochstiftIn between: Freelance Solution Architect for hire
Contact@[email protected]
3 / 24
Who's who
Context
Context
4 / 24
Who's who
ContextContext
Paderborn University20.000 students2.500 employees
Lots of central IT servicesIDM (LDAP, Kerberos, AD, …)Mail (SMTP, IMAP, PMX, Mailman, Exchange)An aweful lot of websiteseLearning things (Moodle, PAUL, …)SharePointFile servicesThe Internet...
5 / 24
Who's who
Context
The Past
The Past
6 / 24
Who's who
Context
The Past
7 / 24
Who's who
Context
The Past
The PastCisco Nexus based L2 fabric
VLANs for service / backend networks2x F5 Viprion 2400 LBs
Router / default gateway for all service networksPrefixes for VIPs statically routed to VRRP IPPrefixes for backend networks statically routed to VRRP IPNo ACLs between service networksOut-of-everything end of 2018
Manually configuredEven monitoring
8 / 24
Who's who
Context
The Past
9 / 24
Who's who
Context
The Past
The Idea The Idea
10 / 24
Who's who
Context
The Past
The Idea
The big picture
11 / 24
Who's who
Context
The Past
The Idea
The ideaA service as central config elementCan be balanced by
AnycastHAProxy
If balancedService VIPs announced via BGPShould be Active/Active
Monitoring configured automaticallyChecks for frontends / VIPs as well as backends
Config of webserver(s) generatedShould additionally allow
H/A clustersCaching layer for web stuff
Subnets of service nodes should be routed by DC routerswith ACLs
12 / 24
Who's who
Context
The Past
The Idea
Working DC network setup
All VLANs everywhereBGP capable DC routers
Heavy automation for Linux boxes
bcfg2Written in PythonEasily extendableConfig generators for Icinga2Basic Apache2 templating
People not afraid of automation
On the contrary
What was in the cards?
13 / 24
Who's who
Context
The Past
The Idea
Now what IS a serviceHas an FQDN
resolves to IP and/or Legacy-IP addressesHas a proto and service
proto derived from service, if possiblee.g. tcp/http or tcp/80
Is provided by hosts of $bcfg2_groupe.g. kdc-production
May be anycastedMay be balanced
And the LBs anycastedMay be a web thing
With special http confige.g. template, redirects and stuff
May have special monitoring config
14 / 24
Who's who
Context
The Past
The Idea
How does it look like?mwilhelm@kili:/bcfg2/etc/services/imt/infrastructure/anycasted$ cat kerberos-kdc.srv
anycast: Truestatus: produktiv
name: kerberos-kdc
fqdn: kerberos.srv.imt.uni-paderborn.deservice: kerberos
bcfg2_srv_group: kdc-slave
monitoring: virtual_bcfg2_groups: - kdc - imt-master
15 / 24
Who's who
Context
The Past
The Idea
Well OK, it has a defaulting mechanism, toomwilhelm@kili:/bcfg2/etc/services/imt/infrastructure/anycasted$ cat defaults.yaml
anycast: Truestatus: produktiv
mwilhelm@kili:/bcfg2/etc/services/imt/infrastructure/anycasted$ cat kerberos-kdc.srv
name: kerberos-kdc
fqdn: kerberos.srv.imt.uni-paderborn.deservice: kerberos
bcfg2_srv_group: kdc-slave
monitoring: virtual_bcfg2_groups: - kdc - imt-master
16 / 24
Who's who
Context
The Past
The Idea
The NowThe Now
17 / 24
Who's who
Context
The Past
The Idea
The Now
18 / 24
Who's who
Context
The Past
The Idea
The Now
Lessons learnedBad NIC firmware is bad
BGP timeouts are longRecovery times are bad when L2 is a black holeBFD will solve this
HAProxy configuration is complex
Lots of switches have effect on other switchesNo way to ask HAProxy what config options are active
19 / 24
Who's who
Context
The Past
The Idea
The Now
The goodBackends with support for Proxy Protocol
Apache2Cyrus IMAPDovecotEximNginxPostfixVarnish...
20 / 24
Who's who
Context
The Past
The Idea
The Now
The badOpenLDAP
No support for Proxy ProtocolHas to be DNATed by HAProxy when slapd should see client IPsTherefore LDAP backends have to be routed by HAProxy
Exchange
Funny problems with timeouts (solved)Funny problems with Outlook for Mac clients
SharePoint
Funny problems when you don't use tcp mode for some vHostsI want this hour of my life back
21 / 24
Who's who
Context
The Past
The Idea
The Now
Bonus level: Packet �lter con�gurationWe know what ports a service is usingWe know where (backend, frontend)Let's generate netfilter rulesLimiting access to source prefixes just came on topSpecifying additional_ports, too
mwilhelm@kili:/bcfg2/etc/services/imt/infrastructure/anycasted$ cat proxy.srv
name: proxy
fqdn: proxy.srv.imt.uni-paderborn.deservice: proxyprotos: tcpport: 3128
bcfg2_srv_group: proxy-server-produktiv
acl: allow_from: - imt_thinclients - imt_fw_mgmt
22 / 24
Who's who
Context
The Past
The Idea
The Now
Links
Further ReadingBGP / networking basics
https://myfirst.network
Anycast with Cisco Nexus 7000 and Debian Linux
https://blog.sdn.clinic/2018/02/anycasted-services-with-debian-bird-anycast-healthchecker-and-cisco-nexus-7000/
Anycast all the things
https://www.slideshare.net/BarbarossaTM/anycast-all-the-things
23 / 24
Who's who
Context
The Past
The Idea
Outlook
Links
Questions?
Questions?
24 / 24
