Belvedere

104
Belvedere Belvedere Environment consistency Environment consistency from Dev to Prod from Dev to Prod Infracoders, April 2013 Infracoders, April 2013

Transcript of Belvedere

Page 1: Belvedere

BelvedereBelvedere

Environment consistencyEnvironment consistencyfrom Dev to Prodfrom Dev to Prod

Infracoders, April 2013Infracoders, April 2013

Page 2: Belvedere

Colin PanissetColin Panisset

@nonspecialist@nonspecialist

Tech Lead – InfrastructureTech Lead – Infrastructure

REA GroupREA Group

Page 3: Belvedere
Page 4: Belvedere
Page 5: Belvedere

belvederebelvedere nounnoun: : a building, or a building, or architectural feature of a building, architectural feature of a building,

designed and situated to look out upon a designed and situated to look out upon a pleasing scene.pleasing scene.

Latin Latin bellusbellus fine + fine + vidērevidēre to see to see

Page 6: Belvedere

In other words:In other words:

A platform that lets you see nice A platform that lets you see nice thingsthings

Page 7: Belvedere

In the beginning …

Page 8: Belvedere

Devs Devs wrotewrote code code

Ops Ops deployeddeployed it it

Page 9: Belvedere

Release cycles were longRelease cycles were long

Page 10: Belvedere

Code moved through Code moved through environments like ...environments like ...

Page 11: Belvedere
Page 12: Belvedere

When code got to staging:When code got to staging:

Page 13: Belvedere
Page 14: Belvedere

#!/bin/sdlc

while !staging.ok:

Fix problems

Redeploy

Page 15: Belvedere

#!/bin/sdlc#!/bin/sdlc

whilewhile !staging.ok: !staging.ok:

devs.fix_problemsdevs.fix_problems

ops.deployops.deploy

Page 16: Belvedere

Then, when the code got to Then, when the code got to prod:prod:

Page 17: Belvedere
Page 18: Belvedere

This was not considered idealThis was not considered ideal

Page 19: Belvedere

But the code hadn't But the code hadn't changed ...changed ...

Page 20: Belvedere

EnvironmentsEnvironments had had

Page 21: Belvedere

The more we looked at it,The more we looked at it,

the more problems we foundthe more problems we found

Page 22: Belvedere

OS version differencesOS version differences

Page 23: Belvedere

OS version differencesOS version differences

Deployment methodsDeployment methods

Page 24: Belvedere

OS version differencesOS version differences

Deployment methodsDeployment methods

Package versionsPackage versions

Page 25: Belvedere

OS version differencesOS version differences

Deployment methodsDeployment methods

Package versionsPackage versionsApp configsApp configs

Page 26: Belvedere

OS version differencesOS version differences

Deployment methodsDeployment methods

Package versionsPackage versionsApp configsApp configs

AuthenticationAuthentication

Page 27: Belvedere

OS version differencesOS version differences

Deployment methodsDeployment methods

Package versionsPackage versionsApp configsApp configs

AuthenticationAuthentication

Hardcoded IPsHardcoded IPs

Page 28: Belvedere

What to do?What to do?

Page 29: Belvedere

Designing BelvedereDesigning Belvedere

Page 30: Belvedere

3-pronged approach3-pronged approach

Page 31: Belvedere

3-pronged approach3-pronged approach

1. Move environment-specific 1. Move environment-specific config into the environmentconfig into the environment

Page 32: Belvedere

3-pronged approach3-pronged approach

1. Move environment-specific 1. Move environment-specific config into the environmentconfig into the environment

2. Convention over 2. Convention over configurationconfiguration

Page 33: Belvedere

3-pronged approach3-pronged approach

1. Move environment-specific 1. Move environment-specific config into the environmentconfig into the environment

2. Convention over 2. Convention over configurationconfiguration

3. Use the same OS image 3. Use the same OS image everywhere (including dev)everywhere (including dev)

Page 34: Belvedere

Find problems as early in the Find problems as early in the development pipeline as development pipeline as

possiblepossible

Page 35: Belvedere

Give devs familiarity with a Give devs familiarity with a production-like system to aid production-like system to aid

transition to a more open transition to a more open access modelaccess model

Page 36: Belvedere

1. Move environment-specific 1. Move environment-specific config into the environmentconfig into the environment

Page 37: Belvedere

CredentialsCredentials

New Relic keysNew Relic keys

Database passwordsDatabase passwords

ConfigurationConfiguration

App-emitted endpoint URLsApp-emitted endpoint URLs

Page 38: Belvedere

Move Move environmentenvironment-specific -specific configuration into … configuration into …

environmentenvironment variables variables

Page 39: Belvedere

Populate environment variables Populate environment variables before app startsbefore app starts

Page 40: Belvedere

Update source files for Update source files for environment variables each environment variables each

time the app restartstime the app restarts

Page 41: Belvedere

Provide a “config service” as the Provide a “config service” as the source of truth for values in the source of truth for values in the

environmentenvironment

Page 42: Belvedere

#!/bin/shinit script

#!/bin/rubyupdate script

#!/bin/shcredentials

application

configservice

(source)

Page 43: Belvedere

Config service is hierarchicalConfig service is hierarchical

based on client hostnamebased on client hostname

Page 44: Belvedere

Common values sit higher up Common values sit higher up the treethe tree

Page 45: Belvedere

globalglobal

zone1.foo.comzone1.foo.com zone2.foo.comzone2.foo.com

hostA.zone1.foo.comhostA.zone1.foo.com

hostB.zone1.foo.comhostB.zone1.foo.com

hostA.zone2.foo.comhostA.zone2.foo.com

Page 46: Belvedere

globalglobal

zone1.foo.comzone1.foo.com zone2.foo.comzone2.foo.com

hostA.zone1.foo.comhostA.zone1.foo.com

hostB.zone1.foo.comhostB.zone1.foo.com

hostA.zone2.foo.comhostA.zone2.foo.com

New Relic keyNew Relic key New Relic keyNew Relic key

Page 47: Belvedere

Override values for host or Override values for host or domain-specific casesdomain-specific cases

Page 48: Belvedere

2. Convention 2. Convention over over

configurationconfiguration

Page 49: Belvedere

Use short DNS CNAMEs which Use short DNS CNAMEs which resolve differently in different resolve differently in different

environmentsenvironments

Page 50: Belvedere

ExamplesExamplesprod.foo.com dev.foo.com

smtp

db-rw

auth.near

auth.far

Page 51: Belvedere

ExamplesExamplesprod.foo.com dev.foo.com

smtp mailsvr.foo.com

db-rw master-db.foo.com

auth.near ldap.prod.foo.com

auth.far ldap.dr.foo.com

Page 52: Belvedere

ExamplesExamplesprod.foo.com dev.foo.com

smtp mailsvr.foo.com null.dev.foo.com

db-rw master-db.foo.com dev-db.foo.com

auth.near ldap.prod.foo.com ldap.dev.foo.com

auth.far ldap.dr.foo.com ldap.test.foo.com

Page 53: Belvedere

You need to have DNS workingYou need to have DNS working

Page 54: Belvedere

You need to have environments You need to have environments with different DNS domains or with different DNS domains or

subdomainssubdomains

Page 55: Belvedere

Environment-independent Environment-independent config exampleconfig example

ntp.confntp.conf

Page 56: Belvedere

/etc/ntp.conf:/etc/ntp.conf:

......

serverserver ntp1.nearntp1.near iburstiburst

serverserver ntp2.nearntp2.near iburstiburst

serverserver ntp1.farntp1.far iburstiburst

restrictrestrict nagios.nearnagios.near maskmask 255.255.255.255 255.255.255.255 nomodify notrapnomodify notrap

restrictrestrict nagios.farnagios.far maskmask 255.255.255.255 255.255.255.255 nomodify notrapnomodify notrap

......

Page 57: Belvedere

Note use of Note use of .near.near and and .far.far to to indicate relative preference; indicate relative preference; you can't weight A recordsyou can't weight A records

Page 58: Belvedere

DNS resolvers support search DNS resolvers support search paths; use that feature!paths; use that feature!

Page 59: Belvedere

3. Use the same OS image 3. Use the same OS image everywhere (including dev)everywhere (including dev)

Page 60: Belvedere

Basic principlesBasic principles

a. Build the “platform image” oncea. Build the “platform image” once

Page 61: Belvedere

Basic principlesBasic principles

a. Build the “platform image” oncea. Build the “platform image” once

b. Transform the built image b. Transform the built image format, not the contentformat, not the content

Page 62: Belvedere

Basic principlesBasic principles

a. Build the “platform image” oncea. Build the “platform image” once

b. Transform the built image b. Transform the built image format, not the contentformat, not the content

c. Provide a minimal-function c. Provide a minimal-function image, let application RPM image, let application RPM dependencies fill in the restdependencies fill in the rest

Page 63: Belvedere

Creating the Creating the platform imageplatform image

Page 64: Belvedere

CentOS 6 x86_64CentOS 6 x86_64

Page 65: Belvedere

KojiKoji

Raw disk imageRaw disk image

Page 66: Belvedere

In wordsIn words

Commits to the Commits to the belvederebelvedere repo in repo in githubgithub trigger a build in Jenkins trigger a build in Jenkins

which uses which uses koji spin-appliancekoji spin-appliance to to create a disk image defined by a create a disk image defined by a

kickstart file which calls kickstart file which calls puppet applypuppet apply to impose configuration and which to impose configuration and which

results in a raw, bootable disk imageresults in a raw, bootable disk image

Page 67: Belvedere

KojiKoji

Raw disk imageRaw disk image

Page 68: Belvedere

Images have the commit SHA Images have the commit SHA burned into the filesystem for burned into the filesystem for

identificationidentification

Page 69: Belvedere

Build time: 26 minutesBuild time: 26 minutes

Promotion via tags: 1 minutePromotion via tags: 1 minute

Page 70: Belvedere

Testing the built imageTesting the built image

Page 71: Belvedere

Raw disk imageRaw disk image

VMVM

ovftoolovftoolknife vsphereknife vsphere

Page 72: Belvedere

puppet/modules/postfix/manifests/config.pp:puppet/modules/postfix/manifests/config.pp:

classclass postfixpostfix::::configconfig { {

includeinclude nrpe nrpe

filefile { { '/etc/nagios/nrpe.d/postfix.cfg''/etc/nagios/nrpe.d/postfix.cfg'::

ensureensure => present, => present,

ownerowner => root, => root,

groupgroup => root, => root,

modemode => => '0644''0644',,

sourcesource => => 'puppet:///modules/postfix/nrpe.cfg''puppet:///modules/postfix/nrpe.cfg',,

requirerequire => => ClassClass[['nrpe''nrpe']]

}}

}}

Page 73: Belvedere

On the platform image:On the platform image:

/etc/nagios/nrpe.d/postfix.cfg/etc/nagios/nrpe.d/postfix.cfg

command[check_postfix]command[check_postfix]==/usr/lib64/nagios/pl/usr/lib64/nagios/plugins/check_smtp -4 -H localhostugins/check_smtp -4 -H localhost

Page 74: Belvedere

The test script in the build pipeline:The test script in the build pipeline:

echoecho -n-n ""Checking Postfix (SMTP): Checking Postfix (SMTP): ""

RESRES==$( check_nrpe -H $TARGET -c check_postfix $( check_nrpe -H $TARGET -c check_postfix 22>>&&1 1 ))

if [if [ $?$? -ne -ne 00 ]; then]; then

failurefailure; echo; echo

echo "echo "$RES$RES""

failfail==truetrue

elseelse

successsuccess; echo; echo

echo "echo "$RES$RES""

fifi

Page 75: Belvedere

Use the same Nagios probes in Use the same Nagios probes in prod to test running instancesprod to test running instances

Page 76: Belvedere

Distributing theDistributing theplatform imageplatform image

Page 77: Belvedere

Raw disk imageRaw disk image

ovftoolovftool

TemplateTemplateAMIAMI

aws-cliaws-cli

VMVM

Page 78: Belvedere

Multiple VMware environments Multiple VMware environments in different datacentresin different datacentres

Page 79: Belvedere

Raw disk image →Raw disk image →

qemu-img convertqemu-img convert → →

ovftoolovftool → →

knife vsphereknife vsphere

Page 80: Belvedere

Multiple AWS Regions and Multiple AWS Regions and accounts (dev/staging/prod)accounts (dev/staging/prod)

Page 81: Belvedere

Raw disk image →Raw disk image →aws cloudformation create-stack →aws cloudformation create-stack →

rsync →rsync →

dd →dd →

aws ec2 create-snapshotaws ec2 create-snapshot → →aws ec2 register-imageaws ec2 register-image

(awscli command line)(awscli command line)

Page 82: Belvedere

For distant regionsFor distant regions

maintain a persistent EC2 maintain a persistent EC2 instanceinstance

and use rsync's delta to and use rsync's delta to minimise transmitted bytesminimise transmitted bytes

Page 83: Belvedere

Raw disk image: 1GB Raw disk image: 1GB (uncompressed)(uncompressed)

Page 84: Belvedere

Built-in auto-resizing on reboot Built-in auto-resizing on reboot when the underlying device when the underlying device

growsgrows

Page 85: Belvedere

Works for physical nodes too:Works for physical nodes too:

same Puppet manifestsame Puppet manifest

different kickstart filedifferent kickstart file

Page 86: Belvedere

We can bring persistent We can bring persistent (physical) boxes up to date (physical) boxes up to date

simply:simply:

box# box# puppet apply ...puppet apply ...

Page 87: Belvedere

Raw disk image can be used as Raw disk image can be used as a local VM, tooa local VM, too

(KVM, Xen, vagrant, VMWare fusion, (KVM, Xen, vagrant, VMWare fusion, Parallels, etc)Parallels, etc)

Page 88: Belvedere

Image promotion is the only Image promotion is the only manual stepmanual step

Page 89: Belvedere

Features!Features!

Page 90: Belvedere

AWS CloudFormation cfn-initAWS CloudFormation cfn-init

Page 91: Belvedere

AWS CloudFormation cfn-initAWS CloudFormation cfn-init

VMWare ToolsVMWare Tools

Page 92: Belvedere

AWS CloudFormation cfn-initAWS CloudFormation cfn-init

VMWare ToolsVMWare ToolsEnvironment-independent configEnvironment-independent config

Page 93: Belvedere

AWS CloudFormation cfn-initAWS CloudFormation cfn-init

VMWare ToolsVMWare ToolsEnvironment-independent configEnvironment-independent config

Standardised base packagesStandardised base packages

Page 94: Belvedere

AWS CloudFormation cfn-initAWS CloudFormation cfn-init

VMWare ToolsVMWare ToolsEnvironment-independent configEnvironment-independent config

Standardised base packagesStandardised base packages

Platform-level Nagios checksPlatform-level Nagios checks

Page 95: Belvedere

AWS CloudFormation cfn-initAWS CloudFormation cfn-init

VMWare ToolsVMWare ToolsEnvironment-independent configEnvironment-independent config

Standardised base packagesStandardised base packages

Platform-level Nagios checksPlatform-level Nagios checks

OS-level tuning OS-level tuning (eg IO scheduler)(eg IO scheduler)

Page 96: Belvedere

Example installed baseExample installed base

kernel auditingkernel auditing

Splunk forwarderSplunk forwarder

LDAP clientLDAP client

NFS client with automounter (homedirs)NFS client with automounter (homedirs)

postfixpostfix

sudo configuration (with LDAP support)sudo configuration (with LDAP support)

NTPNTP

munin/graphitemunin/graphite

iSCSI supportiSCSI support

Page 97: Belvedere

Network config via DHCP:Network config via DHCP:

Standardised,Standardised,

ubiquitous,ubiquitous,

environment-relevantenvironment-relevant

Page 98: Belvedere

In prod environments:In prod environments:

60% of all VMs are platform60% of all VMs are platform

In dev AWS environment:In dev AWS environment:

37% of instances are platform37% of instances are platform

Page 99: Belvedere

In the end …

Page 100: Belvedere

Things to ImproveThings to Improve

Page 101: Belvedere

Support for multiple partitionsSupport for multiple partitions

Better testingBetter testing

on-image IDS (ossec, snort)on-image IDS (ossec, snort)

Shareable code (?)Shareable code (?)

Page 102: Belvedere

Questions?Questions?

Page 103: Belvedere

Colin PanissetColin Panisset

@nonspecialist@nonspecialist

Page 104: Belvedere

Photo creditsPhoto creditsMr. BelvedereMr. Belvedere

http://en.wikipedia.org/wiki/File:Mr_Belvedere.jpghttp://en.wikipedia.org/wiki/File:Mr_Belvedere.jpg

Iowa LandscapeIowa Landscapehttp://www.flickr.com/photos/yoorock/7842391144/ (by-nc-nd)http://www.flickr.com/photos/yoorock/7842391144/ (by-nc-nd)

Neonate queen snake choking on a crayfishNeonate queen snake choking on a crayfishhttp://www.flickr.com/photos/peteandnoewoods/4367200217/ (by-sa)http://www.flickr.com/photos/peteandnoewoods/4367200217/ (by-sa)

Ferry toll sign at CowesFerry toll sign at Coweshttp://www.flickr.com/photos/auntiep/5281268994/http://www.flickr.com/photos/auntiep/5281268994/

Keep LeftKeep Lefthttp://www.flickr.com/photos/mrlederhosen/4283136097/http://www.flickr.com/photos/mrlederhosen/4283136097/

Brunel mixed gauge trackBrunel mixed gauge trackhttp://www.flickr.com/photos/nox_noctis_silentium/7929226488/http://www.flickr.com/photos/nox_noctis_silentium/7929226488/

Train wreck at Montparnasse 1895Train wreck at Montparnasse 1895 http://commons.wikimedia.org/wiki/File:Train_wreck_at_Montparnasse_1895.jpg (pd)http://commons.wikimedia.org/wiki/File:Train_wreck_at_Montparnasse_1895.jpg (pd)