Using Vagrant, Puppet, Testing & Hadoop

Hadoop in Box

From Playground to Produc5on – Using Vagrant, Puppet, Tes5ng and

Hadoop.

Who am I?

•  Dennis Matotek Technical Lead, PlaForms Experian Hitwise Co-‐Author: Pro Linux System Administra5on: Turnbull, Lieverdink, Matotek, Apress 2009

Technical Reviewer: Pulling Strings with Puppet: Turnbull, Apress 2008

What are we solving?

•  We have a group of developers...

They want to build something cool!

We don’t want to end up with this..

So let’s get together early in the design

How can we help?

•  Don’t put implementa5on plans at the end of a project.

•  Everyone gets involved in wri5ng infrastructure code

•  Infrastructure code should be included in the development build pipeline and have to pass tests.

•  Push infrastructure code from playground to produc5on. Design, test and deploy your infrastructure code like your applica5on code.

How can we do it?

•  As administrators we can help build the development environment for projects.

•  Infrastructure on the desktop – A lot of the concep5on phase coding work can be done on the desktop. • What packages are needed for the project? • What configura5on should they be in? •  How can you share your ideas?

Choices •  Virtualiza5on technologies to choose from

–  Virtual Box –  LXC –  KVM/XEN

•  Configura5on management tools –  Puppet –  Chef –  SaltStack

•  Tes5ng tools –  Cucumber-‐puppet –  Rspec

•  CI tools –  Jenkins

Let’s look at Vagrant

What’s it about? •  A project on Github wriaen in Ruby to manage Oracle’s VirtualBox virtual

machines ( originators: Mitchell Hashimoto and John Bender, 2010). •  You can build and distribute projects amongst teams or colleagues.

•  Download ‘boxes’ and build project environments that are the same •  Boxes are reusable testbeds. When you are ready, push your

development code environment to others. •  Take those environments and run them against Jenkins or other CI

tools. •  Sandbox, develop, test and push your infrastructure code into

produc5on. •  How easy is it?

$ vagrant box add base hap://files.vagrantup.com/lucid32.box $ vagrant init $ vagrant up

Vagrant boxes

•  What’s a box? •  Boxes come from standard VirtualBox instances. With specific configura5ons that Vagrant requires.

•  What ever VirtualBox supports, so does Vagrant. •  Boxes are basically a tar of an exported VirtualBox.

•  Configured harddisks, CPU, RAM, Networks. •  You can create them yourself or use ones that others have created and distributed.

•  How to build a box is documented here: hap://vagrantup.com/docs/base_boxes.html

Launching a box Install VirtualBox, install Ruby, install vagrant. Create your own box or find one that is distributed already

$ mkdir project ; cd project $ vagrant box add <box_name> <url or file_path>

This adds and makes it available to the vagrant init command

$ vagrant box list hadoop_in_a_box $ vagrant init <box_name>

You will now have the default Vagran2ile created in your directory

$ ls VagranFile $ vagrant up $ vagrant ssh

VagranFile

Vagrant::Config.run do |config| # All Vagrant configura5on is done here. The most common configura5on # op5ons are documented and commented below. For a complete reference, # please see the online documenta5on at vagrantup.com. # Every Vagrant virtual environment requires a box to build off of. config.vm.box = "hadoop_in_a_box" config.ssh.private_key_path = "./.ssh/vagrant.key" # shared_folders -‐ this folder must exist in your project directory config.vm.share_folder("shared_folder", "/shared", "./shared_folder") end

Vagrant::Config.run do |config| # general setngs: # config.vm.boot_mode = :gui config.vm.customize [ "modifyvm", :id, "-‐-‐memory", "512" ] # ssh setngs: # Set the following to point to your ssh key config.ssh.private_key_path = "./.ssh/vagrant.key" # Change these to suit, some5mes it takes awhile to the virtual box to respond config.ssh.max_tries = 25 config.ssh.5meout = 3 # shared_folders -‐ this folder must exist in your project directory config.vm.share_folder("shared_folder", "/shared", "./shared_folder“) # Below is an example of a mul5ple VM config.vm.define :node1 do |base_config| base_config.vm.box = "my_base" base_config.vm.forward_port 22, 2102 base_config.vm.network :hostonly, "192.168.222.10" end # config.vm.define :node2 do |base_config| # base_config.vm.box = "my_base" # base_config.vm.forward_port 22, 2103 # base_config.vm.network :hostonly, "192.168.222.11" # end end

Provisioning Your Box

•  Ruby plugins for Vagrant –  Build your own specific plugins that make provisioning easy for you

•  Shell provisioning –  Bash shell scripts or commands

base_config.vm.provision :shell do |shell| shell.inline = "hostname $1“ shell.args = “node1“ end

•  Chef Solo/Chef Server

config.vm.provision :chef_solo do |chef| chef.add_recipe("apache") chef.add_recipe("php") end

•  Puppet/Puppet Server

config.vm.provision :puppet, do |puppet| puppet.manifests_path = “manifests" puppet.manifest_file = "default.pp“ end

Provision with Puppet

•  You can use a Puppet Master or locally apply Puppet modules and manifests to provision your Vagrant nodes. – Locally applied Puppet modules and manifests:

base_config.vm.provision :puppet, :module_path => ["puppet_modules","puppet_modules_private"], :op5ons => "-‐-‐verbose" do |basepuppet| basepuppet.manifests_path = "puppet_manifests“ basepuppet.manifest_file = "default.pp“ basepuppet.pp_path = "/tmp/vagrant-‐puppet“ end

Vagrant and Puppet

Cont’d

– Using Puppet Master to provision: – Point your configura5on at your local Puppet Master

Vagrant::Config.run do |config| .... <snip> .... base_config.vm.provision :puppet_server do |puppet| puppet.puppet_server = "puppet.yourdomain.com" end end

Vagrant and Puppet

•  The basic manifest is made up of the following components: /etc/puppet -‐ -‐ manifests/site.pp -‐ manifests/nodes.pp -‐ modules/<module_name>/manifests -‐ modules/<module_name>/files -‐ modules/<module_name>/templates -‐ modules/<module_name>/lib

Puppet Manifest design

Think about using ‘environments’

•  Puppet allows you to use environments. Environments are separate namespaces where you can run and test your code on the same puppet master. – Namespaces like produc5on, staging, tes5ng, etc

•  The puppet.conf file needs the following: modulepath = /etc/puppet/environments/$environments

Environments cont’d

•  Allows you to checkout code under the /etc/puppet/environments/<checkout> and then pass the following to the client $ puppet agent -‐-‐test -‐-‐noop -‐-‐environment <checkout>

•  Test changes against systems before pushing code to produc5on

Things to think about in Module Design

•  Puppet modules are a collec5on of resources to install, configure and manage a specific applica5on or perform some kind of func5on. –  Eg, install and configure the hapd service for your applica5ons.

•  Keep modules separate. Don’t have hapd resources being managed from your postgresql module.

•  Keep data separate from code. –  Have a separate class that contains your data ( class modname::data

{ } ) –  Use an external node classifier (ENC). That is a CMDB like service that

Puppet can extract and build configura5ons from. •  Keep an ear on the Puppet User list as many design

ques5ons are asked and answered there.

•  Nodes are tedious to manage. nodes.pp node base { include yum } node node1 inherits base { include hapd }

•  Just have this: node default { include roles }

•  Group nodes based on Facts or other data.

Manage nodes?

Roles, everything has a role

•  If it doesn’t have a role, it has a default role. •  Roles decide what the node has. –  Easier to manage than node and doesn’t rely on ‘inheritance’. •  Commonly, inheritance is not like programming inheritance.

–  Roles with Hiera. class roles { $my_role = hiera(‘my_role’) if $my_role == ‘webservice’ { include roles::webservices } }

Tes5ng Modules

Tes5ng, phhhuu!

•  Why test? – As your module complexity grows you need to make sure that it will work.

– Puppet is CONSTANTLY changing •  Ensure your code is keeping up with new puppet versions

– Your infrastructure code is code – why not test it? – Test driven code is beaer code, helps to think about what the outcome should be.

Introducing the tools •  RSpec – –  hap://rspec-‐puppet.com/

•  Cucumber-‐Puppet – –  haps://github.com/nistude/cucumber-‐puppet

•  Both tools do the same thing and are based on common tes5ng frameworks.

•  Both tools support Business Driven Development •  How do I use it? –  RSpec tests the modules –  Cucumber tests the manifests as a whole

RSpec

class hadoop::namenode::config { require hadoop::config include hadoop::install::namenode include hadoop::namenode::cluster_config_files # realise the user and group and the configfiles Group <| tag == 'hadoop_node' |> -‐> <snip> file { '/usr/lib/hadoop-‐0.20/logs/SecurityAuth.audit': ensure => present, <snip> require => Package['hadoop-‐0.20-‐namenode'] } -‐> Exec <| tags == 'common_execs' |> -‐> hadoop::namenode::create_namenode_dirs {$hadoop::config::hadoop_default_dirs: } -‐> class {"hadoop::namenode::namenode_format": } } #end class

RSpec require 'spec_helper' describe 'hadoop::namenode::config' do let(:facts) { {:hostname => 'node2', :hadoop_node => 'namenode', :role =>

'hadoop_namenode' } } let(:5tle) { 'config' } it { should include_class('hadoop::install::namenode') } it { should contain_file('/usr/lib/hadoop-‐0.20/logs/SecurityAuth.audit') } it { should contain_file('/etc/hadoop-‐0.20/conf.default/core-‐site.xml') } it { should contain_service('hadoop-‐0.20-‐namenode').with_ensure('present') } end

Cucumber-‐Puppet

•  Does the catalog compile for your nodes? – Tests run on the master (or alterna5ve) – When nodes check in, Puppet creates a yaml file in /var/lib/puppet/yaml/node

– cucumber-‐puppet uses the output, the node cache file, from the last puppet run

–  In Puppet v3 this changes some what as you can use the puppet node find interface to retrieve the same informa5on.

Cucumber Basics Feature: General policy for all catalogs In order to ensure applicability of a host's catalog As a manifest developer I want all catalogs to obey some general rules Scenario Outline: Compile and verify catalog Given a node specified by "features/yaml/<hostname>.mylocal.yaml" When I compile its catalog Then compila5on should succeed And all resource dependencies should resolve Examples: | hostname | | puppet | | node2 | | node3 | | node4 |

Cucumber-‐Puppet Then /^service "([^\"]*)" should be "([^\"]*)"$/ do |name, state| steps %Q{ Then there should be a resource "Service[#{name}]" } if state == "disabled" steps %Q{ Then the service should have "enable" set to "false" } elsif state == "running" steps %Q{ Then the state should be "#{state}" } end end

Automa5ng Tes5ng

Jenkins

•  Helps maintain build pipelines •  Push your infrastructure into the so�ware project pipelines.

•  Con5nuous integra5on used main by so�ware projects, not o�en by infrastructure

•  Get greater certainty of your infrastructure deployments.

Useful plugins

Demonstra5on

Success

Using Vagrant, Puppet, Testing & Hadoop

Documents

Transcript of Using Vagrant, Puppet, Testing & Hadoop