Real world Django deployment using Chef
-
Upload
coderanger -
Category
Technology
-
view
16.926 -
download
3
description
Transcript of Real world Django deployment using Chef
![Page 2: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/2.jpg)
Who am I?
• Django hacker
• Ruby user
• Developer
• Sys admin
![Page 3: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/3.jpg)
Who are you?
• System administrators?
• Designers?
• Developers?
• “Business” People?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/
![Page 4: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/4.jpg)
What are we talking about?
http://www.flickr.com/photos/peterkaminski/2174679908/
![Page 5: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/5.jpg)
Agenda
• How’s and Why’s
• Getting Started with Chef
• Python-specific Tools
• Case Study: Packaginator
http://www.flickr.com/photos/koalazymonkey/3590953001/
![Page 6: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/6.jpg)
Infrastructure as Code
![Page 7: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/7.jpg)
A technical domain revolving around building and managing infrastructure programmatically
![Page 8: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/8.jpg)
Enable the reconstruction of the business from
nothing but a source code repository, an application
data backup, and bare metal resources.
![Page 9: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/9.jpg)
Configuration Management
![Page 10: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/10.jpg)
System Integration
http://www.flickr.com/photos/opalsson/3773629074/
![Page 11: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/11.jpg)
n-Tier Infrastructure
Database Master
Load Balancer
App Server App Server
• Provision
• Configure
• Integrate
{{
![Page 12: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/12.jpg)
![Page 13: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/13.jpg)
The Chef Framework
With thanks (and apologies) to Stephen Nelson-Smith
![Page 14: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/14.jpg)
The Chef Framework
• Reasonability
• Flexibility
• Library & Primitives
![Page 15: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/15.jpg)
The Chef Tool(s)
With thanks (and apologies) to Stephen Nelson-Smith
![Page 16: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/16.jpg)
The Chef Tool(s)
• ohai
• chef-client
• chef-server
• knife
![Page 17: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/17.jpg)
The Chef API
With thanks (and apologies) to Stephen Nelson-Smith
![Page 18: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/18.jpg)
The Chef API
• Client/Server
• RESTful API w/ JSON
• Search Service
• Derivative Services
![Page 19: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/19.jpg)
The Chef Community
With thanks (and apologies) to Stephen Nelson-Smith
![Page 20: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/20.jpg)
The Chef Community
• Apache License, Version 2.0
• 400+ Individual contributors
• 90+ Corporate contributors
• Dell, Rackspace,VMware, RightScale, Heroku, and more
• 240+ cookbooks
• http://community.opscode.com/
![Page 21: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/21.jpg)
Chef Enables Infrastructure as Code
• Resources
• Recipes
• Roles
• Source Code
package "haproxy" do action :installend
template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"end
service "haproxy" do supports :restart => true action [:enable, :start]end
![Page 22: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/22.jpg)
Chef Resources
• Have a type.
• Have a name.
• Have parameters.
• Take action to put the resource in the declared state.
• Can send notifications to other resources.
package "haproxy" do action :installend
template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"end
service "haproxy" do supports :restart => true action [:enable, :start]end
![Page 23: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/23.jpg)
Resources take action through Providers
![Page 24: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/24.jpg)
package “haproxy” { yum install haproxyapt-get install haproxypacman sync haproxypkg_add -r haproxy
Chef Providers
![Page 25: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/25.jpg)
Common Resources
package "apache2"
template "/etc/apache2/httpd.conf"
cookbook_file "/etc/apache2/key.pem"
user "www-data"
execute "/etc/init.d/apache2 restart"
![Page 26: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/26.jpg)
Idempotence
• Convergence
• Guard clauses
execute "createdb myapp" do not_if "psql -c \list | grep myapp"end
execute "createdb myapp" do only_if do db.query("SELECT ...").first == 0 endend
![Page 27: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/27.jpg)
Recipes are collections of Resources
![Page 28: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/28.jpg)
Chef Recipes
• Recipes are evaluated for resources in the order they appear.
• Each resource object is added to the Resource Collection.
package "haproxy" do action :installend
template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"end
service "haproxy" do supports :restart => true action [:enable, :start]end
![Page 29: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/29.jpg)
Chef Recipes
• Recipes can include other recipes.
• Included recipes are processed in order.
include_recipe "apache2"include_recipe "apache2::mod_rewrite"include_recipe "apache2::mod_deflate"include_recipe "apache2::mod_headers"include_recipe "apache2::mod_php5"
![Page 30: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/30.jpg)
template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]"end
%w{ php5 php5-dev php5-cgi }.each do |pkg|
package pkg do action :install end
end
Chef Recipes
• Extend recipes with Ruby.
• Dynamic configuration through search.
pool_members = search("node", "role:mediawiki")
template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members notifies :restart, "service[haproxy]"end
![Page 31: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/31.jpg)
Chef Roles
• Roles describe nodes.
• Roles have a run list.
• Roles can have attributes.
name "mediawiki"description "mediawiki app server"run_list( "recipe[mysql::client]", "recipe[application]", "recipe[mediawiki::status]")
name "mediawiki_load_balancer"description "mediawiki load balancer"run_list( "recipe[haproxy::app_lb]")override_attributes( "haproxy" => { "app_server_role" => "mediawiki" })
![Page 32: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/32.jpg)
Track it like source code...
% git logcommit d640a8c6b370134d7043991894107d806595cc35Author: jtimberman <[email protected]>
Import nagios version 1.0.0
commit c40c818498710e78cf73c7f71e722e971fa574e7Author: jtimberman <[email protected]>
installation and usage instruction docs
commit 99d0efb024314de17888f6b359c14414fda7bb91Author: jtimberman <[email protected]>
Import haproxy version 1.0.1
commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4Author: jtimberman <[email protected]>
add mediawiki cookbook
commit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700Author: jtimberman <[email protected]>
multiple environments in data bag for mediawiki
![Page 33: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/33.jpg)
Other Chef Terms
• Cookbooks are collections of recipes
• Environments are pegged cookbook versions
• Servers are Nodes
• Data bags are JSON blobs
![Page 34: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/34.jpg)
Python Cookbook
• python::package, python::source — Install Python
• python::pip, python::virtualenv — Make it dance
• python::default — All of the above!
• python_pip, python_virtualenv
![Page 35: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/35.jpg)
Installing a package
python_virtualenv "/srv/myapp/env" do action :createend
python_pip "django" do action :upgrade version "1.3" virtualenv "/srv/myapp/env"end
![Page 36: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/36.jpg)
Gunicorn Cookbook
• gunicorn::default
• gunicorn_config
![Page 37: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/37.jpg)
Supervisor Cookbook
• supervisor::default
• Debian-style for now
• supervisor_service
![Page 38: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/38.jpg)
Packaginator
![Page 39: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/39.jpg)
Role: base
• Install users
• Configure sudo
• apt-get update
• Install gcc
name "base"description "Base role applied to all nodes."run_list( "recipe[users::sysadmins]", "recipe[sudo]", "recipe[apt]", "recipe[build-essential]")override_attributes( :authorization => { :sudo => { :users => ["ubuntu"], :passwordless => true } })
![Page 40: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/40.jpg)
Recipe: packaginator
• The good stuff
• General parameters
• Sub-resources
application "packaginator" do path "/srv/packaginator" owner "nobody" group "nogroup" repository "https://github.com/coderanger/packaginator.git" revision "master" migrate true packages ["libpq-dev", "git-core", "mercurial"]
django do packages ["redis"] requirements "requirements/mkii.txt" settings_template "settings.py.erb" debug true database do database "packaginator" engine "postgresql_psycopg2" username "packaginator" password "awesome_password" end database_master_role "packaginator_database_master" collectstatic "build_static --noinput" end
gunicorn do only_if { node['roles'].include? 'packaginator_application_server' } app_module :django port 8080 end
celery do only_if { node['roles'].include? 'packaginator_application_server' } config "celery_settings.py" django true celerybeat true celerycam true broker do transport "redis" end end
nginx_load_balancer do only_if { node['roles'].include? 'packaginator_load_balancer' } application_port 8080 static_files "/site_media" => "site_media" end
end
![Page 41: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/41.jpg)
Application Resource
• Who are we?
• Where are we going?
application "packaginator" do path "/srv/packaginator" owner "nobody" group "nogroup"
![Page 42: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/42.jpg)
Application Resource
directory "/srv/packaginator"
directory "/srv/packaginator/shared"
deploy_revision "packaginator" do deploy_to "/srv/packaginator" owner "nobody" group "nogroup"
![Page 43: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/43.jpg)
Application Resource
• Source code
• Migration status
• Base packages
repository "https://github.com/..."revision "master"migrate truepackages ["libpq-dev", "git-core","mercurial"]
![Page 44: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/44.jpg)
Django Resource
• Python packages
• pip requirements
• Settings file
• Debug mode
• Static files
django do packages ["redis"] requirements "requirements/mkii.txt" settings_template "settings.py.erb" debug true collectstatic "build_static --noinput"
![Page 45: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/45.jpg)
Django Resource
python_pip "redis"
execute "pip -E ... -r ..."
template "shared/local_settings.py"
execute "manage.py build_static"
![Page 46: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/46.jpg)
Folder Layout
• /srv/myapp
• /srv/myapp/shared
• /srv/myapp/shared/releases
• /srv/myapp/shared/releases/4faedb2ee...
• /srv/myapp/shared/releases/9457be295...
• ...
• /srv/myapp/current
![Page 47: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/47.jpg)
Folder Layout
• /srv/myapp
• ...9457be295/local_settings.py -> ../shared/local_settings.py
• /srv/myapp/current -> .../releases/9457be295...
![Page 48: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/48.jpg)
Django Resource
• Database
• Where?
database do database "packaginator" engine "postgresql_psycopg2" username "packaginator" password "awesome_password"enddatabase_master_role "packaginator_database_master"
![Page 49: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/49.jpg)
Django Resource
• Check local first
• Cloud IP!
if node["roles"].include? ... nodeelse search(:node, "roles:...").firstend
![Page 50: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/50.jpg)
Gunicorn Resource
• Only on frontends
• Use Django mode
• Internal port
gunicorn do only_if do node["roles"].include? ... end app_module :django port 8080end
![Page 51: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/51.jpg)
Gunicorn Resource
• Write out config
• Symlink it in
• Load service
gunicorn_config ...
supervisor_service ...
![Page 52: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/52.jpg)
Celery Resource
• Where?
• Django-mode
• Enable all three
• Broker and results
celery do only_if ... config "celery_settings.py" django true celerybeat true celerycam true broker do transport "redis" endend
![Page 53: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/53.jpg)
Nginx Resource
• Internal port again
• Static map
• URL => path
nginx_load_balancer do only_if ... application_port 8080 static_files({
"/site_media" => "site_media"})
end
![Page 54: Real world Django deployment using Chef](https://reader033.fdocuments.net/reader033/viewer/2022042813/54b744764a7959746e8b464a/html5/thumbnails/54.jpg)
https://github.com/coderanger/djangocon2011