Download - Puppet getting started by Dirk Götz

www.netways.de // blog.netways.de // @netways

Make IT do more with less

27 NOVEMBER 2013 | PUPPET CAMP

PUPPET GETTING STARTED

DIRK GÖTZ | NETWAYS GMBH



AGENDA

￭ Brief introduction

￭ Configuration management

￭ Components

￭ Design your environment

￭ Design your workflow

￭ Design your module



BRIEF INTRODUCTION



BRIEF INTRODUCTION TO NETWAYS

• Founded in 1995

• Open source since 1997

• 40 employees

• Specialised in open source systems

management and open source data

center infrastructure



Puppet Camp 2014

• 11 April 2014

• 110 attendees (April 2013)

• 170 attendees (November 2014)

Open Source Data Center Conference

• 8 – 10 April 2014

• 120 attendees (2013)

• 2 tracks of presentations & workshops

NETWAYS CONFERENCES



CONFIGURATION MANAGEMENT



Golden images

Software solutions

CONFIGURATION MANAGEMENT

Manual configuration

Self-made scripts

for i in $(cat host.cfg)dossh user@$i uname -a

done

Types of „Automation“



THE PUPPET WAY

1. Define

2. Simulate

3. Enforce

4. Report

Desired StateCurrent State



THE PUPPET WAY - DEFINE

￭ Using Puppet Domain Specific Language (DSL)

￭ Describe relationship between resources

￭ Create modular reuseable code



THE PUPPET WAY - SIMULATE

￭ Simulate deployment

￭ Without changes to your system

￭ Test and fix it before enforcing

$ sudo puppet apply ntp.pp --verbose --noopInfo: Loading facts in /var/lib/puppet/lib/facter/root_home.rbInfo: Loading facts in /var/lib/puppet/lib/facter/pe_version.rbInfo: Loading facts in /var/lib/puppet/lib/facter/vmware.rbInfo: Loading facts in /var/lib/puppet/lib/facter/last_run.rbInfo: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rbInfo: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rbInfo: Loading facts in /var/lib/puppet/lib/facter/puppi_projects.rbInfo: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rbError: Could not find template 'ntp/ntp.conf.erb' at /tmp/ntp.pp:9 on node puppet.localdomainError: Could not find template 'ntp/ntp.conf.erb' at /tmp/ntp.pp:9 on node puppet.localdomain



THE PUPPET WAY - ENFORCE

￭ Compares current state and desired state

￭ Changes to desired state

￭ Idempotency

$ sudo puppet agent -tInfo: Retrieving pluginInfo: Loading facts in /var/lib/puppet/lib/facter/root_home.rbInfo: Loading facts in /var/lib/puppet/lib/facter/pe_version.rbInfo: Loading facts in /var/lib/puppet/lib/facter/vmware.rbInfo: Loading facts in /var/lib/puppet/lib/facter/last_run.rbInfo: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rbInfo: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rbInfo: Loading facts in /var/lib/puppet/lib/facter/puppi_projects.rbInfo: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rbInfo: Caching catalog for puppet.localdomainInfo: Applying configuration version '1384768191‚Notice: Finished catalog run in 0.54 seconds

Desired StateCurrent State



THE PUPPET WAY - REPORT

￭ Report changes and metrics

￭ Many different presentation formats



COMPONENTS – BASIC TOOLS



PUPPET WORKFLOW

SSL-Encryption



FACTER

￭ Open source Ruby library by Puppet Labs

￭ Provides information on your host

￭ CLI returns key-value pairs

$ facterarchitecture => x86_64augeasversion => 0.9.0blockdevice_sr0_model => QEMU DVD-ROMblockdevice_sr0_size => 1073741312blockdevice_sr0_vendor => QEMUblockdevices => sr0,vdadomain => localdomainfacterversion => 1.7.3filesystems => ext4,iso9660fqdn => puppet.localdomain

$ facter osfamilyRedHat



PUPPET


￭ Client server based

• REST-API• X509 certificates

￭ Platform independent

• Puppet Domain Specific Language• Providers for different platforms



PUPPET DOMAIN SPECIFIC LANGUAGE

￭ Abstracts resources

￭ Resource types

• user, group, • file, package, service, • exec, • …and many more• possible to create your own

￭ Type and title/name pairs must be unique



PUPPET PROVIDERS

￭ Resource Abstraction Layer

• Resource types• Providers



PUPPET – MORE ABSTRACTION

￭ Class groups resources in one manifest

￭ Syntax constructs possible



PUPPET – SYNTAX CONSTRUCTS

￭ Variables (but are more like constants)

￭ Logical structures

• selectors• case statements• if/elseif/else statements

￭ Dependencies

• require/before• subscribe/notify

￭ Inheritance



PUPPET – MORE ABSTRACTION

￭ Module groups classes and corresponding files

modulename|-- files – static files|-- lib – custom facts, functions, resources|-- manifests – manifests containing classes|-- spec – tests for rspec-puppet|-- templates – dynamic files|-- tests – examples declaring classes



PUPPET FILESERVER

￭ File transfer from master

• puppet:///modules/modulename/filename

￭ “Magic“ mount point for modules

￭ Additional mount points possible

￭ Recursion for directories possible



PUPPET PLUGIN SYNC

￭ Option on the agent

￭ Syncs from all modules

• custom facts• custom resources (types and providers)

￭ Before the actual Puppet run occurs



PUPPET TEMPLATES

￭ Function executed on server

￭ ERB (Plain text with embedded Ruby)

• variables (including facts)• conditionals• iterations• access to tags and classes• usage of Puppet functions

￭ Combination of multiple templates possible



PUPPET NODE DECLARATION

￭ Connects system (node) with functionality (classes)

• Single hosts

• Mulitiple hosts (by regular expression)

• Default

• Inheritance possible



COMPONENTS – BASIC GUI



BASIC GUI – ADDED FUNCTIONALITY

￭ Reporting target

• Collects reports• Graphical presentation

￭ External Node Classifier (ENC)

• Graphical node declaration• Adds groups



PUPPET - REPORTING

￭ Enable on agent

￭ Choose target on server

• http/https• log• tagmail• store• rrdgraph• puppetdb

￭ Multiple targets possible

￭ Create your own



PUPPET - ENC

￭ External source for node declaration

• Script returning yaml• LDAP

￭ Merged with internal node declaration

$/etc/puppet/node.rb puppet.localdomain---environment: productionparameters: foreman_env: productionowner_name: Admin Userdomainname: ""owner_email: root@localdomainroot_pw: $1$default$hCkak1kaJPQILNmYbUXhD0puppetmaster: ""

classes: profiles::default:

dn: cn=testserver,ou=Hosts,dc=madstop,dc=comobjectClass: deviceobjectClass: ipHostobjectClass: puppetClientobjectClass: topcn: testserverenvironment: testingipHostNumber: 192.168.0.50description: My test serverl: dc1puppetClass: testingpuppetVar: owner_name=„Admin user“



PUPPET DASHBOARD

￭ Open source ruby web interface

• Version <= 1.2 by Puppet Labs• Version >= 2 by community

http://puppetlabs.com/presentations/story-dashboard-20



COMPONENTS – ORCHESTRATION



ORCHESTRATION

￭ Centralized execution on multiple systems

￭ Different solutions

• ssh loops• func• fabric• capistrano• mcollective• … and many more



MCOLLECTIVE

￭ Open source Ruby framework by Puppet Labs

￭ Facter for grouping

￭ Middleware for communication

$ mco package status httpd -F kernel=Linux

* [ ============================================================> ] 3 / 3

centos63a version = httpd-2.2.15-15.el6.centos.1ubuntu1204a version = httpd-purgedmaster version = httpd-purged

---- package agent summary ----Nodes: 3 / 3

Versions: 1 * 2.2.15-15.el6.centos.1, 2 * purgedElapsed Time: 0.12 s



PUPPET ENTERPRISE CONSOLE

￭ Ruby web interface based on dashboard

• Auditing• Live management



COMPONENTS – DEPLOYMENT



DEPLOYMENT

￭ Config management needs running agent

￭ (Agent needs running operating system)

￭ Different solutions:

• Kickstart / Autoyast / Preseed / Jumpstart• Cobbler / kickstand• Razor• Puppet Cloud Provisioner• Foreman• … and many more



RAZOR

￭ Open source Ruby library by Puppet Labs/EMC

￭ Client

• micro-kernel for PXE boot

￭ Server

• REST-API• CLI

￭ Policy connects

• host profiles• operating system• config management

{"name": "centos-for-small","repo": { "name": "centos-6.4" },"installer": { "name": "centos" },"broker": { "name": "noop" },"enabled": true,"hostname": "host${id}.example.com","root_password": "secret","max_count": "20","rule_number": "100","tags": [{ "name": "small", "rule": ["<=", ["num", ["fact",

"processorcount"]], 2]}]}



CLOUD PROVISIONER


￭ Instantiates cloud instances

• EC2• VMware• OpenStack• Google Compute Engine

￭ Installs Puppet using SSH



FOREMAN

￭ Open source Ruby web interface by Ohad Levy

(Red Hat)

• Reporting target• ENC• Provisioning



FOREMAN – SMART PROXIES

￭ Connects GUI and Backends

• puppet• puppetca• tftp• dhcp• dns



FOREMAN – COMPUTE RESOURCES

￭ Integrates virtualization and cloud platforms

• Libvirt• oVirt / RHEV• VMware• EC2• Google Compute Engine• OpenStack• Rackspace



COMPONENTS – MORE FEATURES



EXPORTED RESOURCES

￭ One node creates resource

￭ Another node realizes that resource

￭ Resource needs to be stored:

• Stored configs (deprecated)• PuppetDB

￭ Use cases:

• host entries• sshkey management• monitoring / backup• other centralized services



PUPPETDB

￭ PostgreSQL and Java based data warehouse solution

by Puppet Labs

￭ Collects

• Facts• Catalogs• Reports (optional)

￭ Used for

• Inventory service• Exported resources



HIERA


￭ Hierarchical data look-up separates code and data

￭ Integrated in Puppet >3 / addon for Puppet <3

$cat hiera.yaml---:backends:- yaml

:yaml::datadir: /etc/puppet/hieradata

:hierarchy:- `hosts/${::fqdn}´- `location/${::location}´- common

$cat hosts/specialhost.localdomainntp::server = 192.168.23.23

$ cat location/rz2ntp::server = 192.168.2.23yum:mirror = 192.168.2.42

$ cat commonntp::server = 192.168.0.23yum::mirror = 192.168.0.42proxy::server = [`192.168.0.237´,`192.168.0.238´]



COMPONENTS – SOFTWARE MANAGEMENT



SOFTWARE MANAGEMENT

￭ Provider needs defined sources for software

￭ Local mirror reduces traffic

￭ Many different tools

• rsync / createrepo• updian• Spacewalk / Red Hat Satellite / Suse Manager• pulp / katello• … and many more



PULP

￭ Open source Python repository management

by Red Hat

￭ Server:

• Import and upload of content• rpm and puppet modules• Publish web-based or ISO images

￭ Client:

• Server-side management and reporting



KATELLO

￭ Java web interface by Red Hat

￭ Combines:

• candlepin (subscription management)• pulp (software management)• Foreman + Puppet (configuration management)



DESIGN YOUR ENVIRONMENT



CHOICES

￭ Support / packages needed?

￭ Best method to report?

￭ Resources to export?

￭ Deployment needed?

￭ Software management needed?

￭ Node declaration or ENC?



SCALE UP

￭ Puppet easily scales up

￭ Start simple, grow with your environment

￭ Rule of thumb:

number of nodes * catalog compile time (in seconds)number of masters = ------------------------------------------------------------------------------

cores per master * run interval (in seconds)



SCALE UP

￭ Only one certficate authority

• Only needed for registration

￭ Load balance Puppet traffic

￭ GUI only required for users

• But ENC / Reporting always required

￭ Orchestration



SCALE UP – EXAMPLE: PUPPET ENTERPRISE



DESIGN YOUR WORKFLOW



EDITOR

￭ vim – text editor

• vim-puppet – synthax highlight• tabular – style guide conformity• puppet-lint – style guide conformity• syntastic – validation

￭ Geppetto – eclipse based IDE

synthax hightlight, style guide conformity, module creation



STYLEGUIDE

￭ Official guidehttp://docs.puppetlabs.com/guides/style_guide.html

• readability• work with and without features• simple and robust• shareable / useable by others

￭ Create your own

• based on official guide• make your own rules



TESTING

￭ puppet parser validate – syntax

￭ puppet-lint – style guide conformity

￭ puppet apply --noop – simulate

￭ vagrant – simulate

￭ rspec-puppet – expected results



VERSION CONTROL SYSTEM

￭ Pre-Commit / Pre-Receive

• use for validation and review

￭ Post-Commit

• checkout in environment

￭ Versioning for change management

￭ Solutions:

• Version control: svn, git, bazaar• Validation & Review: gerrit, jenkins/hudson



STAGING

￭ Puppet environments

• Different versions of modules

￭ Use your own versioning

• config_version: script returns version string



DOCUMENTATION

￭ Inline

• RDoc markup• Console output• HTML generation

￭ README

• Markdown

￭ Modulefile

• used by Puppet module tool



DESIGN YOUR MODULE



CONSIDERATIONS

￭ Where to start?

￭ Use cases?

• different platforms• different roles• small adjustments• one-time or multiple objects

￭ ‚Part of it‘ or separate?

• Dependencies

￭ Templates or files?

￭ Write your own?



PUPPET FORGE

￭ Community platform for modules

• Web platform• Command line tool

￭ Module information:

• Author• Project homepage and issue tracker• Tags• Releases and download count• Test results



PUPPET FORGE – EXAMPLE

example42

￭ 96 modules on forge / more on github

￭ always same layout

￭ supports: Red Hat, CentOS, Fedora, Ubuntu, Debian,

Mint, SLES, OpenSuSE, FreeBSD

￭ adds monitoring, firewall and puppi (own script library)



MODULE DESIGN – MORE ABSTRACTION

￭ Classes abstract resources

￭ Modules abstract classes

￭ Nodes contain the logic

Not maintainable!

node 'basil.puppetlabs.vm' {class { 'apache’:version => 'latest’,

}class { 'motd': }class { 'ssh’: }if $::operatingsystem == 'solaris' {class { 'users’:default_shell => '/bin/false’,

}}else {class { 'users': }

}Class['ssh’] -> Class['users’]

}






￭ Profiles contain the logic

• abstract the modules• separate implementation

from technology

class profiles::application {include tomcatinclude mysqlinclude componenta

}class profiles::application::x inheritsprofiles::application {include componentbcomponentb::resource { 'name':ensure => present,

}}class profiles::application::y inheritsprofiles::application {include componentcinclude componentd

}class profiles::application::z inheritsprofiles::application {include componentbinclude componentdinclude dependencyClass['dependency'] -> Class['componentd']

}






￭ Profiles contain the logic

￭ Roles contain business logic

• no logic, just profiles• separate business role from implementation

￭ Node has exactly one role

• No Puppet know-how neededfor node declaration

class role::webapp {include profiles::baseinclude profiles::customappinclude profiles::test_tools

}

node ‘web1.example.com’ {include role:webapp

}



QUESTIONS & ANSWERS



NETWAYS GmbH

Deutschherrnstrasse 15-19

90429 Nürmberg

Phone: +49 911 92885-0

Fax: +49 911 92885-77

Email: [email protected]

Website: www.netways.de

Twitter: twitter.com/netways

Facebook: facebook.com/netways

Blog: blog.netways.de

QUESTIONS & ANSWERS