200,000 Lines Later: Our Journey to Manageable Puppet Code
-
Upload
david-danzilio -
Category
Software
-
view
340 -
download
1
Transcript of 200,000 Lines Later: Our Journey to Manageable Puppet Code
2 0 0 , 0 0 0 L I N E S L AT E RO U R J O U R N E Y T O M A N A G E A B L E P U P P E T C O D E
D AV I D D A N Z I L I O C O N S TA N T C O N TA C T
W H O A M I ?
• Puppet Evangelist at Constant Contact
• Joined Constant Contact in 2014
• SysAdmin background but more of a developer these days
• Second time speaking at PuppetConf
W H AT I S C O N S TA N T C O N TA C T ?BACKGROUND
• Founded in 1995
• Headquarters in Waltham, MA
• Offices in San Francisco, NYC, London, among others
• ~ 1,500 Employees, ~ 350 developers, ~ 70 operations
• 160+ apps (SOA - microservices for hipsters)
• We’re hiring! (See me after for details)
W H AT ’ S T H I S TA L K A B O U T ?
• An overview of our environment and our challenges
• Our successes, failures, and lessons from trying to transform Puppet in a large enterprise
• Plans for achieving the Puppet singularity
• My quest to understand and affect change in a massively complex organization
R U N N I N G P U P P E T AT S C A L E I S H A R D
TL ;DR
A L O N G T I M E A G O I N A G A L A X Y FA R FA R A W AY
E A R LY A D O P T E R S Y N D R O M E
• Started using Puppet in 2009 with version 0.24.8
• You name the bug, we’ve probably seen it
• Forge didn’t exist so we had to write everything
• Puppet talent was (still is) hard to come by
• Writing Puppet was a little harder back then…
Feature 0.23.x 0.24.x 0.25.x 2.6.x 2.7.x 3.x 3.2.x 3.4.xDynamic Scope X X X X X Appending to attributes in class inheritance (+>) X X X X X X X XMulti-line C-style comments X X X X X X XArrays of resource references allowed in relationships X X X X X X XOverrides in class inheritance X X X X X X XAppending to variables in child scopes (+=) X X X X X X XClass names starting with 0-9 X X X Regular expressions in node definitions X X X X X XAssigning expressions to variables X X X X X XRegular expressions in conditionals/expresions X X X X X Xelsif in if statements X X X X XChaining Resources X X X X XHashes X X X X XClass Parameters X X X X XRun Stages X X X X XThe “in” operator X X X X X$title, $name, and $module_name available in parameter lists X X X X XOptional trailing comma in parameter lists X X X XHyphens/dashes allowed in variable names * X Automatic class parameter lookup via data bindings X X X“Unless” conditionals X X XIteration over arrays and hashes X XThe modulo (%) operator X X$trusted hash X
• Lots of the code we wrote back then is still around…
• The company was growing so fast, nobody had time to focus on Puppet
• Ops mandated that all configuration and app deployment had to be done with Puppet
• Lots of push back
• Puppet became a dirty word
T H E G O O D
• Everything is in Puppet
• Everything is in Git
• Our infrastructure is built to be ephemeral
• We can provision physical nodes about as fast as we can spin up a virtual machine
• Moving an app to OpenStack is just a matter of running Puppet
T H E B A D
• Stuck on Puppet 2.7.3; lots of the code was written before parameterized classes
• Most of our code lives in a single Git repo
• Testing is up to the developer
• No design beyond the initial implementation; most of the code just evolved organically
• Propensity for home-grown solutions, and a habit of thinking only in terms of the current use case
T H E U G LY
• Massive codebase (more on this in a second)
• Frightening complexity
• Tightly coupled codebase (massive understatement)
• Very few tests (effectively none)
• Very little documentation
• Too easy to get into production
• No dependency management
F I L E S L A N G U A G E B L A N K C O M M E N T C O D E8 , 2 6 7 P U P P E T 4 3 , 0 5 9 2 1 , 9 3 3 1 9 8 , 9 0 11 , 0 7 6 E R B 2 5 , 1 9 8 6 6 1 4 3 , 7 2 4
3 9 1 B O U R N E S H E L L 6 , 7 6 1 6 , 5 3 1 5 5 , 1 6 21 9 3 X M L 2 , 7 1 4 5 , 1 1 1 3 7 , 3 6 05 8 P E R L 4 , 1 9 9 6 , 5 2 0 2 3 , 8 6 6
5 4 2 R U B Y 4 , 9 2 2 1 , 7 2 0 2 2 , 6 6 85 1 P Y T H O N 2 , 9 5 4 1 , 6 2 2 1 1 , 2 6 86 7 S Q L 4 5 2 1 9 7 9 , 9 9 9
4 6 6 YA M L 6 2 3 8 4 2 8 , 7 9 91 0 0 O T H E R 2 , 5 3 9 6 , 0 3 4 8 , 3 2 8
1 1 , 2 1 1 T O TA L 9 3 , 4 2 1 5 0 , 5 7 6 5 2 0 , 0 7 5
F I L E S L A N G U A G E B L A N K C O M M E N T C O D E2 , 3 7 6 R U B Y 5 2 , 3 5 2 3 0 , 4 6 3 2 2 9 , 6 5 4
4 0 E R B 6 1 7 1 1 2 , 2 7 17 X M L 5 0 2 1 , 5 5 4
5 6 YA M L 1 6 4 1 , 5 1 13 2 B O U R N E S H E L L 4 0 2 4 7 9 1 , 6 3 81 6 J S O N 7 5 0 1 , 1 7 58 2 P U P P E T 1 5 9 4 6 6 9 42 3 S M A R T Y 3 8 0 3 6 92 L I S P 4 4 9 6 3 0 0
1 4 O T H E R 1 3 3 1 1 3 6 2 92 , 6 4 8 T O TA L 5 3 , 8 8 6 3 1 , 2 1 4 2 3 9 , 7 9 5
T H E J O U R N E Y B E G I N S
U N D E R S TA N D
I D E N T I F Y O U R U S E R S
• Consumers use Puppet to deploy their apps. They just want Puppet to work.
• Developers may write Puppet modules or profiles. They may consume Forge modules.
• Architects help shape the strategic direction of Puppet in the organization.
• Consumers have a very different use case than Architects or Developers.
• Most of our users are Consumers, so we should focus on them.
• Consumers aren’t necessarily interested in learning Puppet, but they need to use it to deploy their apps.
• Interfaces are very important to these people.
T E A C H
T E A C H I N G P U P P E T
• Most people working with Puppet didn’t really know the language well
• But they had to get shit done
• Rampant propagation of bad patterns
• Wrote a 4-hour intro class and started teaching
• Get everybody on a common baseline, a good foundation to build good habits
• Offer more advanced training to people who want it
U P G R A D E
A M O D E R N V E R S I O N O F P U P P E T
• Puppet 2.7.3 is buggy and slow
• Couldn’t use any modules from the Forge out of the box because of trailing commas
• Puppet 2.7 was holding us back
• Offload file serving to Puppet 3 masters, cheap performance win: bit.ly/1VfSwYM
• Problem: code isn’t compatible with Puppet 3
classmysql_site{
#Setsomedefaults$mysql_log_rotate_enable=0$mysql_log_rotation='daily'$mysql_log_retention='14'$mysql_slowlog_rotate_enable=0$mysql_slowlog_rotation='weekly'$mysql_slowlog_retention='4'$mysql_errlog_rotate_enable=0$mysql_errlog_rotation='monthly'$mysql_errlog_retention='6'$mysql_genlog_rotate_enable=0$mysql_genlog_rotation='daily'$mysql_genlog_retention='14'
...ANDSOON…
}
classmysql_site::devinheritsmysql_site{
#OverridesforDevenvironment$mysql_log_rotate_enable=1$mysql_slowlog_rotate_enable=1$mysql_errlog_rotate_enable=1$mysql_genlog_rotate_enable=1
}
classmysql_site::dev::appinheritsmysql_site::dev{
#Overridesforappindevenvironment$mysql_slowlog_rotation='monthly'$mysql_errlog_rotation='daily'
includemysql
}
M I N I M U M V I A B L E R E FA C T O R
• We knew the code was bad, but we didn’t have the resources to focus on a redesign
• Refactor just enough to get to Puppet 3
• Fix templates
• Eliminate dynamic scope
• Hiera played a big role in this
• Move hosts to Puppet 3 as code becomes compatible
TA K I N G T H E P L U N G E
• We built out a Puppet 3 infrastructure alongside the legacy infrastructure
• Shared CA, so the clients could talk to both sets of masters
• Move hosts over as the code becomes compatible
• Turns out this is really hard when there’s no standard way to classify hosts
• We started with the easiest stuff: our own hosts
• Moved on to our CD apps
• In process of moving legacy apps now
M E A N W H I L E …
• We couldn’t stop time and do all this work in a vacuum, life was still happening around us
• New modules are put into their own repos and deployed with r10k
• We’ve kept Puppet 3 environment up to date. Started on 3.4.3, now on 3.8.3
• New modules must have tests and documentation
• We’re using roles and profiles for new things
• CI workflow and infrastructure
• Testing environments, harnesses, and helpers
• Deployment tooling
• Knowledge of testing frameworks
• No official images
N E X T S T E P S
T H I N G S W E ’ R E TA C K L I N G N E X T
• Node classification
• Roles and Profiles
• Decoupling our code
• Desire for bespoke everything
• Writing modules with a single entry point
• Dependency management
• The Forge test
T H E F O R G E T E S T / P U P P E T A P P R O V E D T E S T / D E F I N I T I O N O F D O N E
• Can this module be deployed to the Forge?
• Is there any proprietary data embedded?
• Is it modular enough to meet unforeseen use cases?
• Does it have a robust suite of tests?
• Does it conform to community style expectations?
• Is it fully documented?
I T ’ S A L L A B O U T T H AT I N T E R FA C E
• Make it easier to use our modules
• Our consumers care about the interface, not the implementation
• Design better and fewer interfaces
• Modules with well defined interfaces for composability and better dependency management
• Documentation and tests as part of the definition of done
T E S T I N G
• Make testing easier
• Make testing automatic
• Make testing a part of the culture
• Teach people about BDD
• Show them how much easier and more reliable an automated test can be than a manual one
• Eventually, make testing required
L E S S O N S L E A R N E D
• Know your audience
• Don’t try and please everybody
• It’s always going to take longer than you think
• Don’t forget about the supporting infrastructure
• People, process, and technology. In that order
• It’s never going to move as fast as you want it to. Walk, don’t run
• Don’t let yourself become paralyzed by complexity
• Take notes, write down things you need to work on, you’ll get to them eventually
• Always take the easy wins
• Focus on high impact improvements, help as many people as you can
• Small improvements add up
There’s nothing more successful than success
L E S S O N S T O B E L E A R N E D
• How do we get people to feel invested in the design?
• How can we reenforce good behaviors?
• How can we make sure our massive codebase is being regularly evaluated?
• How can (should?) we hold people accountable for their Puppet code?
• How can we get operations folks to realize they’re software developers?
T H E E N D
H O W T O F I N D M E
• @djdanzilio on Twitter
• danzilio on Freenode (you can usually find me in #puppet-community, #puppet-dev, and #puppet)
• danzilio on GitHub and The Forge
• david (at) danzil (dot) io
• blog.danzilio.net
I M A G E C R E D I T S
• The space photos are from the NASA Image Galleries and are either in the public domain or licensed under CC-BY 2.0
• All other photos are mine or my employer’s