Mwrc2011 cookbook design patterns

Copyright © 2010 Opscode, Inc - All Rights Reserved

Speaker:

‣ [email protected]‣ @jtimberman‣ www.opscode.com

Joshua Timberman Sr. Technical Evangelist

1

Cookbook Design Patterns

Thursday, March 17, 2011

Welcome

Copyright © 2010 Opscode, Inc - All Rights Reserved 2http://www.flickr.com/photos/anotherphotograph/2100904507/sizes/o/


http://www.flickr.com/photos/anotherphotograph/2100904507/sizes/o/

http://www.flickr.com/photos/anotherphotograph/2100904507/sizes/o/

Copyright © 2010 Opscode, Inc - All Rights Reserved 3

cider:~/dev/cookbooks% git log | grep -ic "timberman" 950


I write cookbooks

Copyright © 2010 Opscode, Inc - All Rights Reserved 4Thursday, March 17, 2011

Training, services and evangelism


Developers?Systems Administrators?Developers who do system administration?“Business” People?

http://www.flickr.com/photos/timyates/2854357446/sizes/l/

5Thursday, March 17, 2011

Enough about me, who are you?




Lets talk about chef

Copyright © 2010 Opscode, Inc - All Rights Reserved 7http://www.flickr.com/photos/tambako/4444066932/


Show of hands time! How many people....

http://www.flickr.com/photos/tambako/4444066932/

http://www.flickr.com/photos/tambako/4444066932/


...solo?


Who is using chef solo, Directly on your own or through a service like EY?


...server?


Who’s using their own Open Source chef server?


...Platform?


Who is using the Opscode Platform?


Chef enables infrastructure as code

Manage configuration as idempotent Resources.Put them together in Recipes.Track it like Source Code.Configure your servers.

11

package "haproxy" do action :installend

template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"end

service "haproxy" do action [:enable, :start]end


This isn’t a talk about how Chef works or the nitty gritty, we assume that you have some familiarity with Chef already. And there’s lots of resources for learning :).


http://www.flickr.com/photos/38299630@N05/3635356091/

wiki.opscode.comhelp.opscode.com

[email protected]@jtimberman


If you want to know more, find me


Chef provides an MVC framework


Chef is an MVC framework for building infrastructure. How is that?


{ "kernel": { "machine": "x86_64", "name": "Darwin", "os": "Darwin", "version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386", "release": "10.4.0" }, "platform_version": "10.6.4", "platform": "mac_os_x", "platform_build": "10F569", "domain": "local", "os": "darwin", "current_user": "jtimberman", "ohai_time": 1278602661.60043, "os_version": "10.4.0", "uptime": "18 days 17 hours 49 minutes 18 seconds", "ipaddress": "10.13.37.116", "hostname": "cider", "fqdn": "cider.local", "uptime_seconds": 1619358 }

Node attributes are the model


Models are data. Attributes are data. We’re going to process and mold the data to get to where we want to be.

Copyright © 2010 Opscode, Inc - All Rights Reserved http://www.flickr.com/photos/peterrosbjerg/3913766224/

A configured node is the view


And where we want to be is a configured node. Running a Rails app, database, middleware, whatever.

Copyright © 2010 Opscode, Inc - All Rights Reservedhttp://www.flickr.com/photos/roadsidepictures/2478953342/sizes/o/

Recipes are the controller


They do all the processing of the data to build the view.

http://www.flickr.com/photos/roadsidepictures/2478953342/sizes/o/

http://www.flickr.com/photos/roadsidepictures/2478953342/sizes/o/

Copyright © 2010 Opscode, Inc - All Rights Reserved http://www.flickr.com/photos/thisisbossi/3526698689/

Recipes are Ruby.


Since recipes are Ruby, that gives us a lot of power and flexibility.


Cookbooks are packages of recipes

18http://www.flickr.com/photos/riggenransom/4140166239


Recipes and supporting code, assets, etc.

http://www.flickr.com/photos/riggenransom/4140166239

http://www.flickr.com/photos/riggenransom/4140166239


Design patterns are applicable to cookbooks


Since recipes and other things are code, and we’re really talking about infrastructure as code, there’s good design patterns!


Cookbooks represent best practices.



Best practices are opinions.


Copyright © 2010 Opscode, Inc - All Rights Reserved 22http://www.flickr.com/photos/peyri/304354485


So lets talk about cookbooks

http://www.flickr.com/photos/peyri/304354485

http://www.flickr.com/photos/peyri/304354485


Of course I really mean Opscode’s cookbooks


Question: fork/clone/watch? Thanks!Contributed? You’re awesome, thank you!Tried to contribute and I told you to sign a CLA? Apache license requirement


http://www.opscode.com/blog/2009/08/11/why-we-chose-the-apache-license/


Please sign a CLA, its for you as much for us






This is the important part: it doesn’t assign copyright to Opscode, you enable us to bundle and redistribute your work, which means your code reaches a lot of people, yay!


But you can also have your code reach a lot of people through Opscode’s Chef Community site: Its like RubyGems for Chef cookbooks. You don’t need to sign a CLA.

Its also easier to find cookbooks than trawling through github.


Cookbooks package configuration

READMEattributes/definitions/files/libraries/metadata.rbproviders/recipes/resources/templates/


Cookbooks can have a bunch of stuff in them. Lets talk about this stuff.


Question: Who saw this blog post?

It talks about how writing your readme first to get clarity about the code, plus it gives a nice bit of documentation.

Copyright © 2010 Opscode, Inc - All Rights Reservedhttp://www.flickr.com/photos/bike/2263136517 30

No really, write the fine manual.


http://www.flickr.com/photos/bike/2263136517

http://www.flickr.com/photos/bike/2263136517


Writing Recipes


Okay, you got writing the README out of the way, or maybe you didn’t. Now write the recipes.


Contrived Example


A real world example


So I was at FOSDEM this year. Did anyone go?

Fosdem is a huge free conference in Europe, 5000+ people, 300 talks, 2 days.


GNU parallel

34

http://www.gnu.org/software/parallel/

It replaces xargs


One of the 300 talks was on GNU parallel.




I wrote a cookbook during the talk.

35http://www.flickr.com/photos/jenorton/2229437427/


http://www.flickr.com/photos/jenorton/2229437427/

http://www.flickr.com/photos/jenorton/2229437427/


This isn’t because I’m awesome.


Copyright © 2010 Opscode, Inc - All Rights Reserved 37http://www.flickr.com/photos/flikr/131433774/


Cats with laser eyes are awesome.

http://www.flickr.com/photos/flikr/131433774/



Nor because the talk was boring.



I wanted to write one anyway.



GNU Parallel is like almost any GNU software.

40

wget ftp://ftp.gnu.org/gnu/thing/thing.tar.gz

tar -zxvf thing.tar.gz

cd thing

./configure && make && make install

Rejoice!



Do that in a recipe

41

remote_file "/tmp/parallel-20110205.tar.bz2" do source "http://ftp.gnu.org/gnu/parallel/parallel-20110205.tar.bz2"end

bash "build gnu parallel from source" do cwd "/tmp" code <<-EOF tar -jxvf parallel-20110205.tar.bz2 (cd parallel-20110205 && ./configure) (cd parallel-20110205 && make && make install) EOFend


So do that in a recipe.


That wasn’t hard.


Big deal, thats not hard. I can do that in about 42 seconds. And I did.


But thats a horrible recipe.


Not because it installs from source. Unless..


'-. .-' _______________'-._________.-'______________ '-. _ '-. .-' _ .-' '-. (_) / \ (_) .-' '-. / \ .-' '-.____/ \____.-' \_ _ _ _ _ / //////////\\\\\\\\\ ///////////\\\\\\\\\\ |||| .-----------._|||| |||| '-|___|___|-' |||| \\\\ '---------' //// \\\|||||||||||||/// \\\\\\\\/////// \\\\\\///// jnh

http://triple-double-u.com/ascii/?s=ascii-art&y=weather&q=ab/beard.txt

http://ascii-art.de/info/copyright/


Unless you have a unix sysadmin beard :) and package everything.






Really, why is it bad?



Can’t download from a different location!

Can’t download a different version!

The file gets downloaded every time to verify the checksum.

Use customizable attributes

46

remote_file problems

remote_file "/tmp/parallel-20110205.tar.bz2" do source "http://ftp.gnu.org/gnu/parallel/parallel-20110205.tar.bz2"end


This resource doesn’t have an easy way for someone to customize the location, or the version, and the file has to be downloaded in order to verify the checksum every time.


Design Pattern 1: Attributes


So we reach our first design pattern, use attributes.


default['gnu_parallel']['url'] = 'http://ftp.gnu.org/gnu/parallel'default['gnu_parallel']['version'] = '20110205'default['gnu_parallel']['checksum'] = 'sha256s dont fit on slides'

In cookbooks/gnu_parallel/attributes/default.rb:


Wait, but I know what versions, and urls and everything in my infrastructure.

Yeah but if you want to share that, random folks on the internet don’t have the same infrastructure.


/tmp isn’t a great location

Some systems clear it on reboot.

This causes the file to be downloaded again.

Use Chef::Config[:file_cache_path].


Downloading to /tmp isn’t a great solution either.


Design Pattern 2: Exploit Chef internal

values


Second pattern, leverage the ability to access Chef’s Ruby objects in Recipes.


remote_file using attributes

51

version = node['gnu_parallel']['version']cache_path = Chef::Config[:file_cache_path]

remote_file "#{cache_path}/parallel-#{version}.tar.bz2" do source "#{node['gnu_parallel']['url']}/parallel-#{version}.tar.bz2" checksum node['gnu_parallel']['checksum'] mode 0644end


So now we have a remote file resource that uses some attributes, and internal Chef values.


bash script problems

52

Least of all is compiling from source :-)

bash "build gnu parallel from source" do cwd "/tmp" code <<-EOF tar -jxvf parallel-20110205.tar.bz2 (cd parallel-20110205 && ./configure) (cd parallel-20110205 && make && make install) EOFend


The other resource in the recipe has problems.


bash script solutions!

53

The version needs to be an attribute.

The default configure options may not be useful.

Moar attributes.

default['gnu_parallel']['configure_options'] = []


We should reuse the version attribute, which we saw earlier. What if we want to customize how things are compiled, or the install prefix?

Define that as an attribute and set it to an empty array, user can modify the attribute in a role or on the node.


Design Pattern 3: Sane defaults easily changed


Empty configure options is sane because there aren’t any of those enabled when you do a normal ./configure when compiling from source.


bash script using attributes

55

config_opts = node['gnu_parallel']['configure_options'].join(" ")

bash "build gnu parallel" do cwd Chef::Config[:file_cache_path] code <<-EOF tar -jxvf parallel-#{version}.tar.bz2 (cd parallel-#{version} && ./configure #{config_opts}) (cd parallel-#{version} && make && make install) EOFend


So here we add a little more ruby, but gain a lot more flexibility. We’re Rubyists, so we’re not scared of Ruby.


'-. .-' _______________'-._________.-'______________ '-. _ '-. .-' _ .-' '-. (_) / \ (_) .-' '-. / \ .-' '-.____/ \____.-' \_ _ _ _ _ / //////////\\\\\\\\\ ///////////\\\\\\\\\\ |||| .-----------._|||| |||| '-|___|___|-' |||| \\\\ '---------' //// \\\|||||||||||||/// \\\\\\\\/////// \\\\\\///// jnh




The README tells his guy how to modify the attributes to customize how to configure and where to install.

... Of course he’s going to make a package anyway.






case node['platform']when "centos" default['gnu_parallel']['install_method'] = 'package'else default['gnu_parallel']['install_method'] = 'source'end

Attribute conditional on platform in attributes file

include_recipe "gnu_parallel::#{node['gnu_parallel']['install_method']}"

package "gnu-parallel"

cookbooks/gnu_parallel/default.rb

cookbooks/gnu_parallel/package.rb


Select which recipe to use based on platform, on centos we’ll install from package using the package recipe


Design Pattern 12: Platform specific

conditionals


Its good to utilize chef’s ability to look up the node’s platform and select behavior or set attributes based on the platform.


INFO: remote_file[/var/cache/chef/parallel-20110205.tar.bz2]: Creating /var/cache/chef/parallel-20110205.tar.bz2INFO: Setting mode to 644 for remote_file[/var/cache/chef/parallel-20110205.tar.bz2]INFO: Ran bash[build gnu parallel] successfully


All that said, we can run chef on the node and get the source installed gnu-parallel, and share this cookbook with other users who can use it how they wish. Yay!


All that in ~30 minutes


Copyright © 2010 Opscode, Inc - All Rights Reserved 61http://www.flickr.com/photos/flikr/131433774/


Because I really am a Cat with Laser Eyes!



Copyright © 2010 Opscode, Inc - All Rights Reserved 62http://www.flickr.com/photos/rutty/438775617


Here’s a giant rubber duck. Quack.

http://www.flickr.com/photos/rutty/438775617

http://www.flickr.com/photos/rutty/438775617


Recipes


Lets talk about Recipes a bit more. We saw a contrived example recipe, lets look at how to best to utilize recipes in cookbooks. These are where we formulate our opinions.


Recipes

64

Separate by functionalitydefault

client

server

... etc


It is totally okay to have separate recipes split up by functionality of what they’re configuring. We saw some of this with the package vs source recipes earlier.


Recipes

Avoid hardcoding datanode attributes

data bags

chef search


Chef has a rich set of features that allow us to avoid hardcoding data in recipes

In addition to attributes that we saw earlier, when using the Chef Server we can use data bags and search.


Code Reuse!

66http://www.flickr.com/photos/dnorman/3314634378


Reduce, reuse, recycle.

http://www.flickr.com/photos/dnorman/3314634378/sizes/l/

http://www.flickr.com/photos/dnorman/3314634378/sizes/l/


Design Pattern 4: Separate recipes



Separate by functionality

default.rb

client.rb

server.rb


default - install common components, should do what one might expectclient - set up to talk to a server, use search to find the server based on a roleserver - set up the server part, search to find clients


Our Nagios cookbook uses this pattern.

http://www.flickr.com/photos/cote/163746456





%w{ nagios-nrpe-server nagios-plugins nagios-plugins-basic nagios-plugins-standard}.each do |pkg| package pkgend

remote_directory "/usr/lib/nagios/plugins" do source "plugins" owner "nagios" group "nagios" mode 0755 files_mode 0755end


nagios::default, packages and plugins.

Copyright © 2010 Opscode, Inc - All Rights Reserved 71http://www.flickr.com/photos/zigazou76/3702501888


Danger!

We’re going to talk about some Chef Server features.

http://www.flickr.com/photos/zigazou76/3702501888

http://www.flickr.com/photos/zigazou76/3702501888


search(:node, "role:#{node[:nagios][:server_role]}") do |n| mon_host << n['ipaddress']end

package "nagios-nrpe-server"

template "/etc/nagios/nrpe.cfg" do source "nrpe.cfg.erb" owner "nagios" group "nagios" mode "0644" variables :mon_host => mon_host notifies :restart, "service[nagios-nrpe-server]"end

service "nagios-nrpe-server" do action [:enable,:start]end


nagios::client is a bit more interesting, where we search for the system that is the monitoring server and then allow it to connect.


include_recipe "nagios::client"

nodes = search(:node, "hostname:*")

package "nagios3"

template "/etc/nagios3/hosts.cfg" do source "hosts.cfg.erb" owner "nagios" group "nagios" mode 0644 variables :nodes => nodes notifies :restart, "service[nagios3]"end

service "nagios3" do action [ :enable, :start]end


Similiarly in nagios::server we search for all the nodes to monitor.


But I use Chef Solo.


So all thats wonderful if you’re using a Server. But you’re not. You’re using Solo.


Benefits of Chef Server

75

Persistent node dataArbitrary infrastructure dataSearch indexesAPI


A sidebar about solo vs server flexibility and reuse you might be missing.


Design Pattern 6L:Check for Chef Solo.



if Chef::Config[:solo] node_list = nodes['mything']['node_list']else node_list = search(:node, "role:mything")end


Make a conditional check for solo before doing something like a search, or loading from a data bag or other server-only feature.


Hardcoding data: Anti-pattern

Customizable cookbooksDocument default attributes

README!

Let users override with rolesAbstract to data bags

non-role/non-node specific like application info

Play nice with chef-solo


Hardcoding recipes is an anti pattern. We already saw this.


Templates and Files


Lets talk about some of the good things you can do with cookbook assets - templates and files


File specificity

cookbooks/mysql/templatescentos/debian/default/redhat/ubuntu-10.04/ubuntu-8.04/ubuntu-9.10/

all contain my.cnf.erbrendered template picked based on node’s platform


File specificity is useful if your environment has multiple platforms, or if you’re using cookbooks from others that support platforms different than your own.

Generally, install the package, grab the default config file and stick it in the right directory.


Static vs Dynamic Resources

cookbook_file is statictemplate is dynamic

Duh :).


In recipes, use the appropriate resources.


You want dynamic

Easily sharableData drivenRich data from multiple sources


So you want templates. Use them. They’re ERB, so <3 for Rubyists. And they help others customize your cookbook for their environment with attributes, of course


Libraries, Definitions, Resources and

Providers


Cookbooks are more than just recipes and assets (and attributes). Lets talk about libraries, definitions, resources and providers.


Libraries

Recipe helpersLWRP helpersHeavyweight R/P


Extend chef with libraries, like enhance recipes with helpers.


Recipe Helpers

Use Chef::Recipe class

Methods are available directly in recipes

85

# in the libraryclass Chef class Recipe def radiant_edge? node[:radiant][:edge] end endend

# in the recipeif radiant_edge? deploy "/srv/radiant" do repository "git://github.com/radiant/radiant.git" endend


Extend the Chef::Recipe class.


LWRP Helpers

Don’t repeat yourself!

Abstract API calls

86

module Opscode module Aws module Ec2 def ec2 @@ec2 ||= RightAws::Ec2.new( new_resource.aws_access_key, new_resource.aws_secret_access_key, { :logger => Chef::Log } ) end end endend

# in provider:include Opscode::Aws::Ec2 ... ec2.describe_addresses.find{|a| a[:public_ip] == ip}...


in our aws cookbook


Heavyweight Resources & Providers

Full Ruby classes like those in Chef itselfAllow behaviors not available in LWRPs‣ inherit/extend existing resources/provider

Distribute as gems‣ chef-deploy


Sometimes you might want to write full resources and provides.


Definitions

88http://www.flickr.com/photos/thestorylady/4326274437


Don’t use definitions anymore. They look like resources, but they’re actually replaced by the resources they contain and they don’t send/receive notifications. Instead for more awesome, use...

http://www.flickr.com/photos/thestorylady/4326274437

http://www.flickr.com/photos/thestorylady/4326274437


Lightweight Resources & Providers

89http://www.flickr.com/photos/lucynieto/2769594798


Aka LWRPs, these are a lightweight DSL for creating new resources and providers in your cookbooks.


Resource DSL

actionsattributes

validation parameters


Resources really just contain two things, actions and attributes.


Validation Parameters

91

Option Meaning

:default Default value

:kind_of Value must be a kind_of?(Klass)

:required Raise exception if this is missing

:regex Match the value with regular expression

:equal_to Value must match.

:name_attribute Set to the name of the resource.

:callbacks Hash of Procs, should return true.

:respond_to Ensure the value has the given method.

http://bit.ly/cheflwrp


This is on the LWRP page of the wiki.


Example resource

92

actions :create_db

attribute :host, :kind_of => Stringattribute :username, :kind_of => Stringattribute :password, :kind_of => Stringattribute :database, :kind_of => Stringattribute :exists, :default => false

cookbooks/mysql/resources/database.rb


Simple example of a resource. Sorry that they’re called attributes, not the same as node attributes.


Lightweight Providers

Resources need providersDSL defines action methodsChef Recipe DSL is extended

You can use Chef resources in action methods!



Provider action code

94

action :create_db do unless @mysqldb.exists Chef::Log.info "Creating database #{new_resource.database}" db.query("create database #{new_resource.database}") new_resource.updated_by_last_action(true) endend

action :create_db do unless @mysqldb.exists Chef::Log.info "Creating database #{new_resource.database}" execute "create #{new_resource.database}" do command "mysqladmin -uroot -h localhost create #{new_resource.database}" end new_resource.updated_by_last_action(true) endend

Good!

Better!



Design Pattern 47: Use moar Ruby!


Reusing resources is cool, but sometimes its better style to use Ruby. It depends. The advantage of reusing resources is that they’re already idempotent. Except Execute.


Make it idempotent

96

def load_current_resource @mysqldb = Chef::Resource::MysqlDatabase.new(new_resource.name) @mysqldb.database(new_resource.database) exists = db.list_dbs.include?(new_resource.database) @mysqldb.exists(exists)end


Its important to make providers idempotent. The load current resource is called by chef to see what state the resource is. You have to write the code that determines the state.


Use it in a recipe!

97

mysql_database "my_app" do host "localhost" username "root" password node['mysql']['server_root_passwd'] database "my_app_production" action :create_dbend



Metadata


A thing about metadata


Metadatacookbooks/gnu_parallel/metadata.rb

maintainer "Opscode, Inc."maintainer_email "[email protected]"license "Apache 2.0"description "Installs/Configures gnu_parallel"long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))version "0.99.0"

depends "build-essential"


Declare dependencies on other cookbooks with metadata.


include_recipe



libraries



LWRPs



templates



Anything from other cookbooks.



Metadata not required for Chef Solo


because you have to ship all the cookbooks to the node because there’s no chef server to distribute the required cookbooks


Testing cookbooks


Gotta test that the design was good, right?


No, not BDD/TDD

107http://www.flickr.com/photos/davies/4782586685


Though the chef source code itself has a heap of rspec/cucumber tests

http://www.flickr.com/photos/davies/4782586685

http://www.flickr.com/photos/davies/4782586685


Ruby

108http://www.flickr.com/photos/thisisbossi/3526698689/


Though the code is just ruby, you could write specs and features etc, we don’t have any examples or know of (m)any people doing this because its easy to test.


knife cookbook upload

109

(chef-server)


No I mean test it for reals. Upload to the Chef server.


tar -czf cookbooks.tar.gz

110

(chef-solo)


Or create a tarball for solo.


vagrantup.com


Anyway, you have your cookbooks somewhere the node(s) can get them. Use what you like to test the recipes. A lot of people like Vagrant.


knife ec2 server create


Some people like ec2. Or rackspace, whatever.

Copyright © 2010 Opscode, Inc - All Rights Reserved 113http://www.flickr.com/photos/valeriebb/290711738

NOT


Use what you like for test machines. That part isn’t important.

http://www.flickr.com/photos/valeriebb/290711738

http://www.flickr.com/photos/valeriebb/290711738


sudo chef-client

114

sudo chef-solo


Whatever, you need to run Chef to test.

Copyright © 2010 Opscode, Inc - All Rights Reserved 115http://www.flickr.com/photos/billburris/2245430380/


Show your work

http://www.flickr.com/photos/billburris/2245430380/

http://www.flickr.com/photos/billburris/2245430380/


git push


Put it on your github repository!


knife cookbook site share


go that extra step and publish on the cookbooks site.

flat namespace like rubygems.org though but we’re working to change that


Posted it to the cookbooks site.


Opscode’s cookbook examples

awsgnu_parallel mysqlnagiosradiant

http://ckbk.it/NAME



Thanks!

121

www.opscode.com/chefIRC and Mailing lists‣ irc.freenode.net #chef‣ lists.opscode.com

Twitter:‣ @opscode, #opschef‣ @jtimberman


Mwrc2011 cookbook design patterns

Documents

Transcript of Mwrc2011 cookbook design patterns