Puppetizing Complex Puppetizing Complex ApplicationsApplications

with sipXecs as an examplewith sipXecs as an example

Kris Buytaert

Kris BuytaertKris Buytaert● I used to be a Dev, Then Became an OpI used to be a Dev, Then Became an Op● Senior Linux and Open Source Consultant Senior Linux and Open Source Consultant

@inuits.be@inuits.be● „„Infrastructure Architect“Infrastructure Architect“● Building Clouds since before the Cloud Building Clouds since before the Cloud ● Surviving the 10Surviving the 10thth floor test floor test● Co-Author of some books Co-Author of some books ● Guest Editor at some sitesGuest Editor at some sites

TodayToday

● About SIPXAbout SIPX

● About PuppetAbout Puppet

● Deploying SipXDeploying SipX

● ......

Introduction 2 PuppetIntroduction 2 Puppet

Not quite a Muppet...Not quite a Muppet...

● Puppet is...Puppet is...

● OSSOSS

● A DSL languageA DSL language

● Written in RubyWritten in Ruby

● Client/server orientedClient/server oriented

● Contains abstraction layersContains abstraction layers

● Repeatable processesRepeatable processes

Master of PuppetsMaster of Puppets

● Puppet masterPuppet master• CA authorityCA authority

• ModulesModules

• Node descriptionsNode descriptions

• Compare, compile, applyCompare, compile, apply

● Master is not a requirement !Master is not a requirement !

Puppet ClientsPuppet Clients

● Puppet client nodesPuppet client nodes• DaemonDaemon

• Cron jobsCron jobs

• External orchestration:External orchestration:

• for i in $hosts; do ssh $i “puppetd --test”; donefor i in $hosts; do ssh $i “puppetd --test”; done

• mCollective, Func, …mCollective, Func, …

FactsFacts

● FactsFacts# facter# facter

memoryfree => 387.21 MBmemoryfree => 387.21 MBmemorysize => 492.75 MBmemorysize => 492.75 MBswapfree => 481.00 MBswapfree => 481.00 MBswapsize => 481.00 MBswapsize => 481.00 MB

domain => dev.inuits.bedomain => dev.inuits.befqdn => node3.dev.inuits.befqdn => node3.dev.inuits.behostname => node3hostname => node3interfaces => eth0interfaces => eth0ipaddress => 172.16.142.141ipaddress => 172.16.142.141macaddress => 00:0c:29:42:0b:8amacaddress => 00:0c:29:42:0b:8anetmask => 255.255.255.0netmask => 255.255.255.0

ModulesModules

● Dedicated per serviceDedicated per service

● ReusableReusable

● Called from the manifestsCalled from the manifests

● Live in /etc/puppet/modules/Live in /etc/puppet/modules/

Module StructureModule Structure● FilesFiles

● TemplatesTemplates

• Dynamic contentDynamic content

• VariablesVariables

<IfModule mpm_worker_module><IfModule mpm_worker_module>StartServers StartServers <%= <%= StartServersStartServers %>%>MaxClients MaxClients <%= <%= MaxClientsMaxClients %>%>MinSpareThreads MinSpareThreads <%= <%= MinSpareMinSpare %>%>MaxSpareThreads MaxSpareThreads <%= <%= MaxSpareMaxSpare %>%>ThreadsPerChild ThreadsPerChild <%= <%= ThreadsChildThreadsChild %>%>MaxRequestsPerChild MaxRequestsPerChild <%= <%= RequestsChildRequestsChild %>%>

</IfModule></IfModule>

● ManifestsManifests

ModulesModules

● FilesFiles

● TemplatesTemplates

● ManifestsManifests• DSLDSL

• ClassesClasses

• ElementsElements

Node definitionsNode definitions● Nodes.ppNodes.pp

cclass defaults {lass defaults {$search = "inuits.be"$search = "inuits.be"$nameservers = ['208.67.220.220', '208.67.222.222']$nameservers = ['208.67.220.220', '208.67.222.222']

include dns::resolvinclude dns::resolvinclude ssh::keysinclude ssh::keysinclude ssh::serverinclude ssh::server

}}

node "ns1.dev.inuits.be" {node "ns1.dev.inuits.be" {include defaultsinclude defaultsinclude dns::powerdns::serverinclude dns::powerdns::serverinclude dns::powerdns::resolverinclude dns::powerdns::resolver

}}

node “web1.dev.inuits.be” {node “web1.dev.inuits.be” {include defaultsinclude defaultsinclude apache2include apache2include mysqlinclude mysql

}}

RalshRalsh● Simplifies writing manifestsSimplifies writing manifests

● Will generate parts of the manifest for youWill generate parts of the manifest for you

● Based on your running configBased on your running config

● Limited functionalityLimited functionality

master1.dev.inuits.be:~#master1.dev.inuits.be:~# ralsh user root ralsh user rootuser { 'root':user { 'root': uid => '0', uid => '0', gid => '0', gid => '0', comment => 'root', comment => 'root', ensure => 'present', ensure => 'present', password => 'f34wi94$PmlI0CxQLb9HD', password => 'f34wi94$PmlI0CxQLb9HD', shell => '/bin/bash', shell => '/bin/bash', home => '/root' home => '/root'}}

master1.dev.inuits.be:~#master1.dev.inuits.be:~# ralsh service apache2 ralsh service apache2service { 'apache2':service { 'apache2': ensure => 'running', ensure => 'running', enable => 'true' enable => 'true'}}

Puppetizing your InfraPuppetizing your Infra● Define common parts Define common parts

● Define unique parts Define unique parts

● Write your manifestsWrite your manifests

● Use modules Use modules

• Puppet ForgePuppet Forge

• GitHubGitHub

• Your own modulesYour own modules

SipXecsSipXecsSipXecsSipXecs

What is sipXecs ?What is sipXecs ?● sipX ECS (Enterprise Communications Server) sipX ECS (Enterprise Communications Server)

● Open Source voice over IP telephony serverOpen Source voice over IP telephony server

● Implementation of the Session Initiation Protocol (SIP) Implementation of the Session Initiation Protocol (SIP)

● IP based communications system (IP PBX)IP based communications system (IP PBX)

● Not unlike AsteriskNot unlike Asterisk

● Development started in 1999Development started in 1999

● GNU Lesser General Public License (LGPL) GNU Lesser General Public License (LGPL)

● Commercial offering from eZuce Inc.Commercial offering from eZuce Inc.

● Designed around FreeSWITCH Designed around FreeSWITCH

● Modular and highly scalable systemModular and highly scalable system

We don't know VOIPWe don't know VOIP● External VOIP consultancyExternal VOIP consultancy

• Hardware selection Hardware selection

• Codecs etc Codecs etc

• Scale outScale out

● Irc.freenode.org #sipxIrc.freenode.org #sipx

● s/don/didn/ts/don/didn/t

● Don't buy the book Don't buy the book

Installing sipxecs Installing sipxecs ● Prebuilt ISO Prebuilt ISO

● Kickstart Kickstart

● Install scripts placed in .bashrcInstall scripts placed in .bashrc

● Ncurses basedNcurses based

● Lots of python scriptsLots of python scripts

● Heavy GUI usageHeavy GUI usage

Why not Just ?Why not Just ?● Backup and Restore ? Backup and Restore ?

• CDR Integration etcCDR Integration etc

● Image ? Image ?

● Productization Productization

• Think 20-100 setupsThink 20-100 setups

• For different customersFor different customers

• Different networks, different domains Different networks, different domains

So, that Python Script ?So, that Python Script ?● Configures your networkConfigures your network

● Configures your dhcpdConfigures your dhcpd

● Configures your dnsConfigures your dns

● Configures your ntpdConfigures your ntpd

● Configures your tftpConfigures your tftp

● Generates SSL stuff for youGenerates SSL stuff for you

There's puppet modules for that !There's puppet modules for that !

SipXconfigSipXconfig● Is enabled by writing Is enabled by writing

““enabled” to /var/sipxdata/process-state/ConfigServerenabled” to /var/sipxdata/process-state/ConfigServer

● The configuration and management server (sipXconfig) The configuration and management server (sipXconfig) provides Web administration and user portals, Web services provides Web administration and user portals, Web services APIs, as well as all the abstraction logic to make using APIs, as well as all the abstraction logic to make using sipXecs as simple as it is. It provides centralized sipXecs as simple as it is. It provides centralized management of all the aspects of sipXecs, including management of all the aspects of sipXecs, including installation, configuration, backup & restore, upgrade, installation, configuration, backup & restore, upgrade, troubleshooting and cluster management.troubleshooting and cluster management.

● ““Pushes” configs to other nodesPushes” configs to other nodes● Should be rewritten in Puppet or a like.Should be rewritten in Puppet or a like.

Configuring sipXecsConfiguring sipXecs● A couple of filesA couple of files

● Some of them even obsoletedSome of them even obsoleted

● Putting the SSL stuff in the right locationPutting the SSL stuff in the right location

Everything is a funky SSL Everything is a funky SSL problemproblem● Sipx generates keys at install timeSipx generates keys at install time

• Ca + keypairs per nodeCa + keypairs per node

● 22ndnd node needs those keys node needs those keys

● Copy to puppetmaster and transfer back to other nodes ?Copy to puppetmaster and transfer back to other nodes ?

● Or generate on puppetmaster and redistribute ?Or generate on puppetmaster and redistribute ?

=> Generated on Puppetmaster => Generated on Puppetmaster

Adding a second nodeAdding a second node● <> clustering <> clustering

● <> high availability ( please don't start crying)<> high availability ( please don't start crying)

● Create an entry in the management interfaceCreate an entry in the management interface

● Then repeat manual installation using ncursesThen repeat manual installation using ncurses

● Or just do a wget to register it with the primaryOr just do a wget to register it with the primary

class voip::sipx {class voip::sipx { sipx::netconfig {sipx::netconfig { "sipx":"sipx": ipaddress => $ip_address,ipaddress => $ip_address, netmask => $netmask;netmask => $netmask; }} if $nodename == 'sipx-a' {if $nodename == 'sipx-a' { sipx::configserver{ "sipx": }sipx::configserver{ "sipx": } sipx::staticcertdbca{ "$hostname": }sipx::staticcertdbca{ "$hostname": } sipx::staticcertdbnodes{ "SIPX-A.${platformdomainextension}":sipx::staticcertdbnodes{ "SIPX-A.${platformdomainextension}": clientname => "SIPX-A"; }clientname => "SIPX-A"; } sipx::staticcertdbnodes{ "SIPX-B.${platformdomainextension}":sipx::staticcertdbnodes{ "SIPX-B.${platformdomainextension}": clientname => "SIPX-B"; }clientname => "SIPX-B"; } include sipx::runmasterinclude sipx::runmaster }} else {else { include sipx::runslaveinclude sipx::runslave sipx::register{ "$nodename":sipx::register{ "$nodename": clientname =>"${nodename}.${platformdomainextension}",clientname =>"${nodename}.${platformdomainextension}", password =>"yourpw",}password =>"yourpw",} }} sipx::supervisor { "$hostname":sipx::supervisor { "$hostname": sipx_supervisor => "sipx-a.$platformdomainextension";sipx_supervisor => "sipx-a.$platformdomainextension"; }} sipx::staticssl{ "$hostname": }sipx::staticssl{ "$hostname": }}}

More complexity More complexity Or regular puppet ordering Or regular puppet ordering

● Sipx requires PgSQLSipx requires PgSQL

● You want PgSQL on an isolated LVYou want PgSQL on an isolated LV

● PgSQL configuration has to be done after it initialized a DBPgSQL configuration has to be done after it initialized a DB

● SipX insist on starting PgSQL for youSipX insist on starting PgSQL for you

class voip::storage {class voip::storage { file {file { "/var/lib/pgsql":"/var/lib/pgsql": ensure => directory;ensure => directory; lvm::volume { "pgsql":lvm::volume { "pgsql": vg => "systemvg",vg => "systemvg", pv => "/dev/cciss/c0d0p2",pv => "/dev/cciss/c0d0p2", fstype => "ext3",fstype => "ext3", size => "20G",size => "20G", ensure => present,ensure => present, }} mount { "/var/lib/pgsql":mount { "/var/lib/pgsql": atboot => true,atboot => true, device => "/dev/systemvg/pgsql",device => "/dev/systemvg/pgsql", ensure => mounted,ensure => mounted, fstype => "ext3",fstype => "ext3", options => "defaults",options => "defaults", require => [Logical_volume['pgsql'],File['/var/lib/pgsql']],require => [Logical_volume['pgsql'],File['/var/lib/pgsql']], }}}}class voip::pgsql {class voip::pgsql { include postgresinclude postgres postgres::initdb { "sipx": }postgres::initdb { "sipx": } postgres::config{ "sipx":postgres::config{ "sipx": listen => "*",listen => "*", postgres::hba { "sipx":postgres::hba { "sipx": allowedrules => [allowedrules => [ "host SIPXCDR all ${clientip}/32 trust","host SIPXCDR all ${clientip}/32 trust", ],], }}}}

include voip::storageinclude voip::storage

include voip::pgsqlinclude voip::pgsql

include voip::sipxinclude voip::sipx

Class["voip::storage"] -> Class["voip::pgsql"] -> Class["voip::sipx"]Class["voip::storage"] -> Class["voip::pgsql"] -> Class["voip::sipx"]

Manual config of the Manual config of the services via the gui is still services via the gui is still

required :(required :(

I want toI want to● Automatically create my admin pwAutomatically create my admin pw

● Automatically add that second nodeAutomatically add that second node

● Automatically disable/ enable functions in the sipX serverAutomatically disable/ enable functions in the sipX server

• e.g conferencing, openfire e.g conferencing, openfire

● Add users/phonesAdd users/phones

● There's an API !There's an API !

● Which only implements limited functionality , and no Which only implements limited functionality , and no configurationconfiguration

Screen scraping ?Screen scraping ?(03:28:30 PM) lazyboy: y, you just need a form processing library, one that can read a form (03:28:30 PM) lazyboy: y, you just need a form processing library, one that can read a form values and allow you to post back your changesvalues and allow you to post back your changes

(03:30:04 PM) lazyboy: the problem w/this method as you know is that it is constantly (03:30:04 PM) lazyboy: the problem w/this method as you know is that it is constantly breakingbreaking

(03:30:41 PM) sdog: yep .. whan you change the gui .. it will break .... (03:30:41 PM) sdog: yep .. whan you change the gui .. it will break ....

(03:30:45 PM) lazyboy: maybe we need a serverside abstraction layer, that does the (03:30:45 PM) lazyboy: maybe we need a serverside abstraction layer, that does the screenscraping and exports out a clean REST APIscreenscraping and exports out a clean REST API

(03:31:13 PM) lazyboy: overtime, APIs go straight thru (03:31:13 PM) lazyboy: overtime, APIs go straight thru

(03:36:18 PM) lazyboy: so it's possible some of what you want to do is available w/not a lot (03:36:18 PM) lazyboy: so it's possible some of what you want to do is available w/not a lot of screen scraping.of screen scraping.

Abusing Test Frameworks to Abusing Test Frameworks to configure services on a configure services on a

webguiwebgui

CucumberCucumber● Looks extremely easyLooks extremely easy

• ““Hey our manager could write these test”Hey our manager could write these test”

● Isn'tIsn't

• Heavily under documentedHeavily under documented

• Best docs are in the RSpec bookBest docs are in the RSpec book

• Online examples are mostly brokenOnline examples are mostly broken

● Requires to write a lot of code Requires to write a lot of code

Apache JmeterApache Jmeter● Test tool Test tool

● Load generation toolLoad generation tool

● Lets you record session by Lets you record session by using a proxyusing a proxy

● Only recent versions support Only recent versions support SSLSSL

SeleniumSelenium● Firefox pluginFirefox plugin

● Replays your actionsReplays your actions

• No need to write codeNo need to write code

● Can export to perl, php, Can export to perl, php, ruby .. ruby ..

• Which requires the a Which requires the a Selenium Remote Control Selenium Remote Control Server Server

• Which launches Firefox Which launches Firefox

● SSL Fun ahead SSL Fun ahead

AlternativesAlternatives● SahiSahi

• Similar to seleniumSimilar to selenium

• Requires proxy Requires proxy

● www::mechanizewww::mechanize

● Mechanize rubygem Mechanize rubygem

● WebtestWebtest

● Your idea ?Your idea ?

I want an APII want an API

ConclusionsConclusions● No good solution yet :(No good solution yet :(

● Talk to your upstream supplierTalk to your upstream supplier

• Vendor / projectVendor / project

● Be patientBe patient

● Show the good exampleShow the good example

● All bugs produced during this experience are on All bugs produced during this experience are on

https://github.com/KrisBuytaert https://github.com/KrisBuytaert

ContactContactKris Buytaert Kris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be

Further ReadingFurther Reading@krisbuytaert @krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.be/http://www.inuits.be/http://www.virtualizatihttp://www.virtualization.com/on.com/http://www.oreillygmt.com/http://www.oreillygmt.com/ EsquimauxEsquimaux

Kheops Business Kheops Business CenterCenterAvenque Georges Avenque Georges Lemaître 54Lemaître 546041 Gosselies6041 Gosselies889.780.406889.780.406+32 495 698 668 +32 495 698 668

InuitsInuits't Hemeltje't HemeltjeGemeentepark 2Gemeentepark 22930 Brasschaat2930 Brasschaat891.514.231891.514.231

+32 473 441 636 +32 473 441 636