Linux Clusters Institute:Configuration Management

Garrett McGrath, Princeton Neuroscience Institute

gmcgrath@princeton.edu

Goals

● Detail what configuration management is and why it is useful● Review the current landscape of available tools● Be able to convey what configuration management is and why /

where it should be used to coworkers and managers

This will not be a deep dive into tools or a full course implementation of a solution. We will use puppet for some examples but many of the concepts transfer relatively broadly.

Defining Configuration Management

At its broadest level, the management and maintenance of operating system and service configuration via code instead of manual intervention.

More formally:

● Declaring the system state in a repeatable and auditable fashion and using tools to impose state and prevent systems from deviating

State

All system have a ‘state’ comprised of:

● Files on Disk● Running services

State can be supplied by:

● Installation / provisioning systems● Golden Images● Manual steps including direct configuration changes and setup

scripts

Modern Configuration Management Features

● Idempotency○ Declaration and management of files and services to reach a ‘desired

state’● Revision Control

○ Systems are managed with an ‘Infrastructure as code’ model● Composable and flexible

Why bother?

● Configuration management can operate as a ‘work multiplier’● Standard configuration steps can be made into components you

mix to define a machine● Tools can be used to verify a machines current state vs. the official

central manifest for the machine● Revision history allows you to track what changes were made to a

system, by who, and why

Benefits of configuration version control

•Built-in documentation (change logs, summaries, etc.)

•Peer review (issue tracking, merge requests, email alerts)

•Reverts (at least partially)

Benefits of configuration management

•Centralized catalog of all system configuration

•Automated enforcement of system state from an authoritative source

•Ensured consistency between systems

•Rapid system provisioning from easily-composed components

•Preflight tests to ensure deployments generate expected results

•Collection of system ‘ground truths’ for better decision making

Modern configuration-management systems

• Puppet• Ruby based DSL

• Chef Infra

• Ruby based DSL

• CFEngine

• C based DSL

• Salt

• Python based DSL

• Ansible

• Python based DSL

Getting started (puppet example)

● Pick a simple, small part of your primary system checklist○ sudoers○ resolv○ nsswitch

● Implement and Test (start with “no-op”)

Puppet Directory Structure

• production/• hiera/• modules/

• ntp

• manifests/

• Init.pp

• files/

• ntp.conf

• manifests/• site.pp

# modules/ntp/manifests/init.pp

class ntp {

package { 'ntp':

ensure => installed,

}

file { '/etc/ntp.conf':

source => 'puppet:///modules/ntp/ntp.conf',

owner => 'root',

group => 'root',

mode => '0644',

require => Package['ntp'],

}

service { 'ntp':

ensure => running,

enable => true,

require => File['/etc/ntp.conf'],

}

}

# manifests/site.pp

node 'node1' { include ntp}

Testing the prototype

# puppet apply --noop \ --modules modules manifests/site.pp

Next steps

•Top-level node roles

•Add features you need now (don't try to do everything at once)

•Convince, teach, and assist your team

•Continue until you have no more questions about your environment

• Find more modules on https://forge.puppet.com/

Puppet Workflow in PNI

● Primarily a Component Object Model○ Secondary configuration with Profiles

● Environment Folders○ production/dev/User workspaces

● Hiera for almost everything● Mercurial based version control with dev and production branches

○ Production env folder follows production branch○ Dev env folder follows dev branch

What does this workflow look like?

● Pull from dev to make sure you’ve got the latest dev code being shared by others

● Make changes in your env folder

● Test changes against test nodes with --environment <name> and --noop as appropriate

● commit

● Merge changes from dev to production if complete and ready to use everywhere

● Push from local env folder to ../dev and ../production as appropriate

● Issue ‘update’ commands in those folders so they are running the latest revs of the code.

Component Object Model

● Machines are self classifying○ Facilitated by ‘custom facts’; rules that specify things like ‘member of

cluster X, access to network Y’○ Host groups for smaller sets of configurations based simply on

hostnames○ Additional behaviors added based on hardware facts

■ Raid cards

■ Nvidia gpus

■ Virtual machine vs. physical

Puppet workflow in HCC

•Roles and profiles

•Hiera

•R10K

•Git

Puppet

R10KGit

Puppet

R10KGit

clone, commit, pushpull, merge….etc.

Puppetfile

environment = productionenvironment = test

You can add more:Gerrit,

Jenkins….

What does this workflow look like?

• git clone git@git-server:puppet

• git checkout –b mybranch

•… make some changes…• git add/commit/push

•On you test node: puppet agent –t –environment=mybranch

•Merge it to production!

Roles and Profiles

Advocating to colleagues

• Work is front-loaded, so early work seems much more costly

• System might undo work done by others

• Add comments at the top of managed config files

• Offer to help colleagues port

• Work with at least one other person

• Be as transparent as possible

• Commit emails

• Slack updates either automated or manually when pushing out changes to production

• Document how to port an existing host

Advocating to management

• Work more efficiently (get more done)• Properly tooled configuration management operates as a ‘work multiplier’ doing something one

time expands to doing the right thing on all systems every time

• Not an all-or-nothing proposition: start with a few systems and go slow

• Document and report success stories

• Deployment speed improvements

• Patch deployment improvements

• Peer review anecdotes

• Corrections made

Things to watch out for

•Also easy to make a mistake on several hosts at once• Test in isolation first, and with a no-op mode• Changes can have wide sweeping effects if not deployed carefully, resulting

in a denial of service failure if not carefully tested before deployment to production.

• It's easy to get lazy and allow systems to fall out-of-sync• This is avoidable with systems using active clients, but scaling this approach is

difficult

• It's easy to let perfectionism take over

Reference

•Puppet: https://puppet.com/

•Puppet forge: https://forge.puppet.com/

•R10k: https://github.com/puppetlabs/r10k

•Roles and profiles: http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/