Knee deep in the undef - Tales from refactoring old Puppet codebases

Knee deep in the undefTales from refactoring old Puppet codebases@petersouter

Knee deep in the undef: Tales from refactoring old

Puppet codebasesPeter Souter

Senior Professional Services Engineer | Puppet@petersouter

Knee deep in the undefTales from refactoring old Puppet codebases@petersouter 2

Who am I?

@petersouter

Senior Professional Services Engineer

5 years using Puppet

2 years @ Puppet Inc

Help customers deploy Puppet

Teach Puppet classes

Contribute to the community and

open-source

petems IRC/Slack/GitHub


Warning: I speak quicklyAnd I have a different accent...

3


“Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.”

- http://www.retrospectives.com/pages/retroPrimeDirective.html

4

http://www.retrospectives.com/pages/retroPrimeDirective.html

http://www.retrospectives.com/pages/retroPrimeDirective.html


Show of hands in the roomWhat version of Puppet are you on now?

5


Show of hands in the roomHow old is your Puppet codebase? When was your first commit?

6


The King is Dead… Long live the king!Puppet 3.X is now EOL


Some customers we hadn’t talked to in a while...This is a good time for an upgrade or refactor of code

8


Noticed a few repeating anti-patterns and issues with a lot of these customersAnd similar issues with people asking for community help in Slack and IRC

9


commit bbdecbeb1c199b7132f94c33ee44a66e265fa456

Author: Jane Doe <[email protected]>

Date: Tue Jun 4 10:46:13 2013 +0100

Initial commit


commit ac4b7c3312d75c3584ef069c272f7ad4ff610eb4

Author: John Doe <[email protected]>

Date: Thu Mar 3 15:46:59 2011 +0000

Import

git-svn-id: https://svn.example.com/svn/puppet/trunk@1

3d7401f0-959d-0410-a61e-fbe730d1da08


I’m not going to talk about the Puppet 4 upgrade stuffLots of talks on that… but it’s a common time for people to look over their codebases when upgrading

12


1. Hiera overload/Too much data2. Lack of validation/CI

3. Reinventing the wheel4. Lack of VCS best practises

5. Make the newbie experience better

5 Key Areas that came up over again


Hiera overload/Too much dataHiera is simultaneously the best and worst thing to happen to Puppet

14


YAML management is the biggest pain point in the DevOps world...Not just limited to Puppet...

15


Why is this a problem?Two main issues...

16


First: Context switching is the mind killerHumans suck at multitasking

17


“The trick here is that when you manage programmers, specifically, task switches take a really, really, really long time. That's because programming is the kind of task where you have to keep a lot of things in your head at once. The more things you remember at once, the more productive you are at programming. A programmer coding at full throttle is keeping zillions of things in their head at once: everything from names of variables, data structures, important APIs, the names of utility functions that they wrote and call a lot, even the name of the subdirectory where they store their source code.”

Human Task Switches Considered Harmful - Joel Spolskyhttp://www.joelonsoftware.com/articles/fog0000000022.html

http://www.joelonsoftware.com/articles/fog0000000022.html

http://www.joelonsoftware.com/articles/fog0000000022.html


● Get the people who are maintaining the code to look at what's written

● How long does it take them to find out how something’s done in your code?

● Ask them if they can easily understand what’s happening and why

To fix it: Get an outsider's perspective


Second: Maintaining this data is costlyBoth technically and mentally

20


$ find . -type f | wc -l 587

Hiera can become bloated over time


● Minimise node specific data: it’s the hardest to maintain and easiest to get stale

● Abstract information into a hierarchy to DRY it up● Purge irrelevant data ● Remember, it’ll be in the git history still!

Clean up your hiera!


PR I’ve been meaning to resurrect: maybe at the contributor summit?

- https://github.com/voxpupuli/puppet-syntax/pull/57

https://github.com/voxpupuli/puppet-syntax/pull/57

https://github.com/voxpupuli/puppet-syntax/pull/57


Lack of validation/CIThe earlier you can catch errors, the cheaper it is to fix them.

24


Gareth’s talk about the Future of Puppet testing

https://speakerdeck.com/garethr/the-future-of-testing-puppet-code





● Puppet-syntax checks puppet, yaml, erb and epp files● When you get that under control, use puppet-lint for

style checks● Then rspec-puppet for unit tests● Then beaker/testkitchen for acceptance tests● If you don’t have CI, fix that! Or at least use git pre/post

commit hooks as a basic gate

At its most basic, a syntax error should not be deployable


Testing with containers makes tests fast and repeatable

http://cfgmgmtcamp.eu/schedule/testing/andy-henriod.html




Lack of VCS best practisesMonolithic repos make it hard to upgrade modules and for multiple people to work on the module at one time

28


Make the newbie experience betterPeople change teams, new people join. Make sure they can contribute as soon as possible

29


● If you want people to follow specific workflows or requirements make it easy as possible

● Prevent risky newbie mistakes with CI and tooling (puppet-lint, rubocop, rake tasks etc)

● Pair-programming is not just about the code: it’s about learning the processes and improving the experience for both new and old users

30

The bar should be low enough to trip over


● Comments can be a crutch● The best comment is a good name for a method or

variable● Sometimes your hand is forced: make sure you

comment clearly and concisely● Give the full context in the commit message

Code should be self-documenting


https://blog.codinghorror.com/code-smells/

“There's a fine line between comments that illuminate and comments that obscure. Are the comments necessary? Do they explain ‘why’ and not ‘what’? Can you refactor the code so the comments aren't required? And remember, you're writing comments for people, not machines.”




● Separate subject from body with a blank line● Limit the subject line to 50 characters● Capitalize the subject line● Do not end the subject line with a period● Use the imperative mood in the subject line● Wrap the body at 72 characters● Use the body to explain what and why vs. how

Git Commit best practices


http://chris.beams.io/posts/git-commit/

“Re-establishing the context of a piece of code is wasteful. We can't avoid it completely, so our efforts should go to reducing it [as much] as possible. Commit messages can do exactly that and as a result, a commit message shows whether a developer is a good collaborator.”




Commit Often, Perfect Later, Publish OnceLearn how rebasing and amending works


An example of good git commits at workhttp://www.philandstuff.com/2014/02/09/git-pickaxe.html

http://www.philandstuff.com/2014/02/09/git-pickaxe.html

http://www.philandstuff.com/2014/02/09/git-pickaxe.html


The Git PickaxeThe reason I care about commit messages is because I'm an avid user of the git pickaxe. If I'm ever confused about a line of code, and I want to know what was going through the mind of the developer when they were writing it, the pickaxe is the first tool I'll reach for. For example, let's say I was looking at this line from our puppet-graphite module:

exec <%= @root_dir %>/bin/carbon-cache.py --debug start

That --debug option looks suspect. I might think to myself: "Why are we running carbon-cache in --debug mode? Isn't that wasteful? Do we capture the output? Why was it added in the first place?" In order to answer these questions, I'd like to find the commit that added the switch.

https://github.com/gds-operations/puppet-graphite/blob/778da32e3e1e735a7aecdcfbc603477b16b9527d/templates/upstart/carbon-cache.conf#L15


commit 5288d5804a3fc20dae4f3b2deeaa7f687595aff1

Author: Philip Potter <[email protected]>

Date: Tue Dec 17 09:33:59 2013 +0000

Re-add --debug option (reverts #11)

The --debug option is somewhat badly named -- it *both* adds debug

output, *and* causes carbon-cache to run in the foreground. Removing the

option in #11 caused the upstart script to lose track of the process as

carbon-cache started unexpectedly daemonizing.

Ideally we want to have a way of running through upstart without the

debug output, but this will fix the immediate problem.

Ta-dah, mystery solved!


Reinventing the wheelThere are existing modules, use them!

39


The more existing boilerplate and modules you can leverage:● The less work you have to do (Yay for you!)● The more maintainable it will be (Yay for your team!)● The more supportable it will be (Yay for everyone!)

Start off on the right foot


Start with the puppet control repo

41https://github.com/puppetlabs/control-repo

https://github.com/puppetlabs/control-repo

https://github.com/puppetlabs/control-repo


Use as supported modules as your number 1 choice

https://forge.puppet.com/modules?endorsements=supported




Use approved modules as your number 2 choice

43https://forge.puppet.com/modules?endorsements=approved

https://forge.puppet.com/modules?endorsements=approved

https://forge.puppet.com/modules?endorsements=approved


If there’s no supported or approved, use your best judgement...

44https://forge.puppet.com/modules?utf-8=%E2%9C%93&sort=downloads&q=limits

https://forge.puppet.com/modules?utf-8=%E2%9C%93&sort=downloads&q=limits

https://forge.puppet.com/modules?utf-8=%E2%9C%93&sort=downloads&q=limits


When writing new modules, use a custom module skeleton

https://github.com/petems/petems-puppet-module-skeleton 45

https://github.com/petems/petems-puppet-module-skeleton

https://github.com/petems/petems-puppet-module-skeleton


Write as little as possibleCode is like a puppy: anything you write you are responsible for


SummaryWhat have we learnt?


Get your hiera under controlClean up old files, validate and try and abstract


Get validation and CI on your codebasesyntax, lint, spec, acceptance


Don’t reinvent the wheelTry and reuse as much as possible,


Make sure you’re VCS is top notchGit is the primary place to show your intentions and the history of the code


Ensure your newbie experience is greatMake the bar low enough to fall over


● PuppetConf 2016: Enjoying the Journey from Puppet 3.x to 4.x – Rob Nelson, AT&T 71 viewshttp://www.slideshare.net/PuppetLabs/puppetconf-2016-enjoying-the-journey-from-puppet-3x-to-4x-rob-nelson-att

● Does Your Configuration Code Smell? - Tushar Sharma, Marios Fragkoulis and Diomidis Spinellis - Dept of Management Science and Technologyhttp://www.tusharma.in/wp-content/uploads/2016/03/ConfigurationSmells_preprint.pdf

● Roles and profiles: A complete examplehttps://docs.puppet.com/pe/2016.5/r_n_p_full_example.html

● Puppet Design Patterns - David Danzilio, Kovarushttp://www.slideshare.net/DavidDanzilio/puppet-design-patterns-puppetconf

53

Want to know more?

http://www.slideshare.net/PuppetLabs/puppetconf-2016-enjoying-the-journey-from-puppet-3x-to-4x-rob-nelson-att



http://www.tusharma.in/wp-content/uploads/2016/03/ConfigurationSmells_preprint.pdf



https://docs.puppet.com/pe/2016.5/r_n_p_full_example.html

https://docs.puppet.com/pe/2016.5/r_n_p_full_example.html

http://www.slideshare.net/DavidDanzilio/puppet-design-patterns-puppetconf

http://www.slideshare.net/DavidDanzilio/puppet-design-patterns-puppetconf


Q&A

54

Knee deep in the undef - Tales from refactoring old Puppet codebases

Technology

Transcript of Knee deep in the undef - Tales from refactoring old Puppet codebases