Belvedere

BelvedereBelvedere

Environment consistencyEnvironment consistencyfrom Dev to Prodfrom Dev to Prod

Infracoders, April 2013Infracoders, April 2013

Colin PanissetColin Panisset

@nonspecialist@nonspecialist

Tech Lead – InfrastructureTech Lead – Infrastructure

REA GroupREA Group

belvederebelvedere nounnoun: : a building, or a building, or architectural feature of a building, architectural feature of a building,

designed and situated to look out upon a designed and situated to look out upon a pleasing scene.pleasing scene.

Latin Latin bellusbellus fine + fine + vidērevidēre to see to see

In other words:In other words:

A platform that lets you see nice A platform that lets you see nice thingsthings

In the beginning …

Devs Devs wrotewrote code code

Ops Ops deployeddeployed it it

Release cycles were longRelease cycles were long

Code moved through Code moved through environments like ...environments like ...

When code got to staging:When code got to staging:

#!/bin/sdlc

while !staging.ok:

Fix problems

Redeploy

#!/bin/sdlc#!/bin/sdlc

whilewhile !staging.ok: !staging.ok:

devs.fix_problemsdevs.fix_problems

ops.deployops.deploy

Then, when the code got to Then, when the code got to prod:prod:

This was not considered idealThis was not considered ideal

But the code hadn't But the code hadn't changed ...changed ...

EnvironmentsEnvironments had had

The more we looked at it,The more we looked at it,

the more problems we foundthe more problems we found

OS version differencesOS version differences

Deployment methodsDeployment methods

Package versionsPackage versions

Package versionsPackage versionsApp configsApp configs

AuthenticationAuthentication

Hardcoded IPsHardcoded IPs

What to do?What to do?

Designing BelvedereDesigning Belvedere

3-pronged approach3-pronged approach

1. Move environment-specific 1. Move environment-specific config into the environmentconfig into the environment

2. Convention over 2. Convention over configurationconfiguration

3. Use the same OS image 3. Use the same OS image everywhere (including dev)everywhere (including dev)

Find problems as early in the Find problems as early in the development pipeline as development pipeline as

possiblepossible

Give devs familiarity with a Give devs familiarity with a production-like system to aid production-like system to aid

transition to a more open transition to a more open access modelaccess model

CredentialsCredentials

New Relic keysNew Relic keys

Database passwordsDatabase passwords

ConfigurationConfiguration

App-emitted endpoint URLsApp-emitted endpoint URLs

Move Move environmentenvironment-specific -specific configuration into … configuration into …

environmentenvironment variables variables

Populate environment variables Populate environment variables before app startsbefore app starts

Update source files for Update source files for environment variables each environment variables each

time the app restartstime the app restarts

Provide a “config service” as the Provide a “config service” as the source of truth for values in the source of truth for values in the

environmentenvironment

#!/bin/shinit script

#!/bin/rubyupdate script

#!/bin/shcredentials

application

configservice

(source)

Config service is hierarchicalConfig service is hierarchical

based on client hostnamebased on client hostname

Common values sit higher up Common values sit higher up the treethe tree

globalglobal

zone1.foo.comzone1.foo.com zone2.foo.comzone2.foo.com

hostA.zone1.foo.comhostA.zone1.foo.com

hostB.zone1.foo.comhostB.zone1.foo.com

globalglobal

zone1.foo.comzone1.foo.com zone2.foo.comzone2.foo.com

hostB.zone1.foo.comhostB.zone1.foo.com

New Relic keyNew Relic key New Relic keyNew Relic key

Override values for host or Override values for host or domain-specific casesdomain-specific cases

2. Convention 2. Convention over over

configurationconfiguration

Use short DNS CNAMEs which Use short DNS CNAMEs which resolve differently in different resolve differently in different

environmentsenvironments

ExamplesExamplesprod.foo.com dev.foo.com

auth.near

auth.far

smtp mailsvr.foo.com

db-rw master-db.foo.com

auth.near ldap.prod.foo.com

auth.far ldap.dr.foo.com

smtp mailsvr.foo.com null.dev.foo.com

db-rw master-db.foo.com dev-db.foo.com

auth.near ldap.prod.foo.com ldap.dev.foo.com

auth.far ldap.dr.foo.com ldap.test.foo.com

You need to have DNS workingYou need to have DNS working

You need to have environments You need to have environments with different DNS domains or with different DNS domains or

subdomainssubdomains

Environment-independent Environment-independent config exampleconfig example

ntp.confntp.conf

/etc/ntp.conf:/etc/ntp.conf:

......

serverserver ntp1.nearntp1.near iburstiburst

serverserver ntp2.nearntp2.near iburstiburst

serverserver ntp1.farntp1.far iburstiburst

restrictrestrict nagios.nearnagios.near maskmask 255.255.255.255 255.255.255.255 nomodify notrapnomodify notrap

restrictrestrict nagios.farnagios.far maskmask 255.255.255.255 255.255.255.255 nomodify notrapnomodify notrap

......

Note use of Note use of .near.near and and .far.far to to indicate relative preference; indicate relative preference; you can't weight A recordsyou can't weight A records

DNS resolvers support search DNS resolvers support search paths; use that feature!paths; use that feature!

3. Use the same OS image 3. Use the same OS image everywhere (including dev)everywhere (including dev)

Basic principlesBasic principles

a. Build the “platform image” oncea. Build the “platform image” once

b. Transform the built image b. Transform the built image format, not the contentformat, not the content

c. Provide a minimal-function c. Provide a minimal-function image, let application RPM image, let application RPM dependencies fill in the restdependencies fill in the rest

Creating the Creating the platform imageplatform image

CentOS 6 x86_64CentOS 6 x86_64

KojiKoji

Raw disk imageRaw disk image

In wordsIn words

Commits to the Commits to the belvederebelvedere repo in repo in githubgithub trigger a build in Jenkins trigger a build in Jenkins

which uses which uses koji spin-appliancekoji spin-appliance to to create a disk image defined by a create a disk image defined by a

kickstart file which calls kickstart file which calls puppet applypuppet apply to impose configuration and which to impose configuration and which

results in a raw, bootable disk imageresults in a raw, bootable disk image

KojiKoji

Images have the commit SHA Images have the commit SHA burned into the filesystem for burned into the filesystem for

identificationidentification

Build time: 26 minutesBuild time: 26 minutes

Promotion via tags: 1 minutePromotion via tags: 1 minute

Testing the built imageTesting the built image

ovftoolovftoolknife vsphereknife vsphere

puppet/modules/postfix/manifests/config.pp:puppet/modules/postfix/manifests/config.pp:

classclass postfixpostfix::::configconfig { {

includeinclude nrpe nrpe

filefile { { '/etc/nagios/nrpe.d/postfix.cfg''/etc/nagios/nrpe.d/postfix.cfg'::

ensureensure => present, => present,

ownerowner => root, => root,

groupgroup => root, => root,

modemode => => '0644''0644',,

sourcesource => => 'puppet:///modules/postfix/nrpe.cfg''puppet:///modules/postfix/nrpe.cfg',,

requirerequire => => ClassClass[['nrpe''nrpe']]

On the platform image:On the platform image:

/etc/nagios/nrpe.d/postfix.cfg/etc/nagios/nrpe.d/postfix.cfg

command[check_postfix]command[check_postfix]==/usr/lib64/nagios/pl/usr/lib64/nagios/plugins/check_smtp -4 -H localhostugins/check_smtp -4 -H localhost

The test script in the build pipeline:The test script in the build pipeline:

echoecho -n-n ""Checking Postfix (SMTP): Checking Postfix (SMTP): ""

RESRES==$( check_nrpe -H $TARGET -c check_postfix $( check_nrpe -H $TARGET -c check_postfix 22>>&&1 1 ))

if [if [ $?$? -ne -ne 00 ]; then]; then

failurefailure; echo; echo

echo "echo "$RES$RES""

failfail==truetrue

elseelse

successsuccess; echo; echo

echo "echo "$RES$RES""

Use the same Nagios probes in Use the same Nagios probes in prod to test running instancesprod to test running instances

Distributing theDistributing theplatform imageplatform image

ovftoolovftool

TemplateTemplateAMIAMI

aws-cliaws-cli

Multiple VMware environments Multiple VMware environments in different datacentresin different datacentres

Raw disk image →Raw disk image →

qemu-img convertqemu-img convert → →

ovftoolovftool → →

knife vsphereknife vsphere

Multiple AWS Regions and Multiple AWS Regions and accounts (dev/staging/prod)accounts (dev/staging/prod)

Raw disk image →Raw disk image →aws cloudformation create-stack →aws cloudformation create-stack →

rsync →rsync →

dd →dd →

aws ec2 create-snapshotaws ec2 create-snapshot → →aws ec2 register-imageaws ec2 register-image

(awscli command line)(awscli command line)

For distant regionsFor distant regions

maintain a persistent EC2 maintain a persistent EC2 instanceinstance

and use rsync's delta to and use rsync's delta to minimise transmitted bytesminimise transmitted bytes

Raw disk image: 1GB Raw disk image: 1GB (uncompressed)(uncompressed)

Built-in auto-resizing on reboot Built-in auto-resizing on reboot when the underlying device when the underlying device

growsgrows

Works for physical nodes too:Works for physical nodes too:

same Puppet manifestsame Puppet manifest

different kickstart filedifferent kickstart file

We can bring persistent We can bring persistent (physical) boxes up to date (physical) boxes up to date

simply:simply:

box# box# puppet apply ...puppet apply ...

Raw disk image can be used as Raw disk image can be used as a local VM, tooa local VM, too

(KVM, Xen, vagrant, VMWare fusion, (KVM, Xen, vagrant, VMWare fusion, Parallels, etc)Parallels, etc)

Image promotion is the only Image promotion is the only manual stepmanual step

Features!Features!

AWS CloudFormation cfn-initAWS CloudFormation cfn-init

VMWare ToolsVMWare Tools

VMWare ToolsVMWare ToolsEnvironment-independent configEnvironment-independent config

Standardised base packagesStandardised base packages

Platform-level Nagios checksPlatform-level Nagios checks

OS-level tuning OS-level tuning (eg IO scheduler)(eg IO scheduler)

Example installed baseExample installed base

kernel auditingkernel auditing

Splunk forwarderSplunk forwarder

LDAP clientLDAP client

NFS client with automounter (homedirs)NFS client with automounter (homedirs)

postfixpostfix

sudo configuration (with LDAP support)sudo configuration (with LDAP support)

NTPNTP

munin/graphitemunin/graphite

iSCSI supportiSCSI support

Network config via DHCP:Network config via DHCP:

Standardised,Standardised,

ubiquitous,ubiquitous,

environment-relevantenvironment-relevant

In prod environments:In prod environments:

60% of all VMs are platform60% of all VMs are platform

In dev AWS environment:In dev AWS environment:

37% of instances are platform37% of instances are platform

In the end …

Things to ImproveThings to Improve

Support for multiple partitionsSupport for multiple partitions

Better testingBetter testing

on-image IDS (ossec, snort)on-image IDS (ossec, snort)

Shareable code (?)Shareable code (?)

Questions?Questions?

Colin PanissetColin Panisset

@nonspecialist@nonspecialist

Photo creditsPhoto creditsMr. BelvedereMr. Belvedere

http://en.wikipedia.org/wiki/File:Mr_Belvedere.jpghttp://en.wikipedia.org/wiki/File:Mr_Belvedere.jpg

Iowa LandscapeIowa Landscapehttp://www.flickr.com/photos/yoorock/7842391144/ (by-nc-nd)http://www.flickr.com/photos/yoorock/7842391144/ (by-nc-nd)

Neonate queen snake choking on a crayfishNeonate queen snake choking on a crayfishhttp://www.flickr.com/photos/peteandnoewoods/4367200217/ (by-sa)http://www.flickr.com/photos/peteandnoewoods/4367200217/ (by-sa)

Ferry toll sign at CowesFerry toll sign at Coweshttp://www.flickr.com/photos/auntiep/5281268994/http://www.flickr.com/photos/auntiep/5281268994/

Keep LeftKeep Lefthttp://www.flickr.com/photos/mrlederhosen/4283136097/http://www.flickr.com/photos/mrlederhosen/4283136097/

Brunel mixed gauge trackBrunel mixed gauge trackhttp://www.flickr.com/photos/nox_noctis_silentium/7929226488/http://www.flickr.com/photos/nox_noctis_silentium/7929226488/

Train wreck at Montparnasse 1895Train wreck at Montparnasse 1895 http://commons.wikimedia.org/wiki/File:Train_wreck_at_Montparnasse_1895.jpg (pd)http://commons.wikimedia.org/wiki/File:Train_wreck_at_Montparnasse_1895.jpg (pd)

Belvedere

Technology

Transcript of Belvedere

Activities Belvedere

BELVEDERE TECH GUIDE

Folder Residencial Damha Belvedere

Corfu - Belvedere

Belvedere Vodka

Belvedere Gruppen Guide

Belvedere scavolini

PUZ Pasunii Belvedere

CRABTREE MANORWAY NORTH | BELVEDERE | LONDON | DA17 … · BELVEDERE STATION PHASE 1 SOLD BELVEDERE WHARF UNITS 2A & 2B Units 2A and 2B form the next phase of Belvedere Wharf. Construction

Fantini, Belvedere

Belvedere Palace Vienna

Belvedere Memorandum

Belvedere new

Constanza - Belvedere Gold

Belvedere Castle

The Belvedere

Ajnara belvedere - 9560090024

Kaap Belvedere

RESIDENZA BELVEDERE

BELVEDERE THAMESMEAD - National Infrastructure Planning · 2019. 5. 22. · THAMESMEAD & BELVEDERE HEAT NETWORK FEASIBILITY STUDY: WORK PACKAGE 2 Project name Thamesmead & Belvedere