monitoring configuration management - from dev to prod
-
Upload
christoph-oelmueller -
Category
Engineering
-
view
184 -
download
7
description
Transcript of monitoring configuration management - from dev to prod
monitoring configuration,
release & configuration management
Christoph Oelmüller
*
*
*say hello
*monitoring design considerations
*monitoring configuration management
*the problem
*two approaches to solve the problem
*1:n approach
*n:m approach
*a quick n:m approach howto
*summary: 1:n versus n:m
*
*physicist
*2008 - 2013: linux consultant
*2014 : operations engineer at EPost-
Development GmbH
*linux infrastructure, identity management,
configuration management, monitoring...
*
*
*requirement analysis
*technical requirements
* revision controlled configurations
* rollback
* automization
*organizational requirements
* feasable workflow
* keep it simple stupid (KISS)
*
*infrastructural requirements
*design your staging
* complete?
* identical?
*
*time & material & risk
*cost of a false alarm
*cost of a grumpy admin
devmonitor
tstmonitor
prdmonitor
*
separate your stages!
*
what to configure?
*
*hosts
*add host to monitoring
master
*services
*add service to
monitoring master
*deploy check/plugin
on node
dev-mastertst-
masterprd-
master
dev-nodetst-
nodeprd-node
*
*
*all stages are identical
*all stages are complete
*one monitoring master/stage
*call it 1:n approach
*1 master
*n nodes
CODE YOUR CONFIG
CLONE YOUR CONFIG
*
*stages aren‘t identical
*stages are incomplete
*stages are only implemented via different
monitoring masters
*call it n:m approach
*n master
*m nodes
*
dev-master
dev-node 1
dev-node 2
tst-master
tst-node 1
tst-node 2
prd-master
prd-node 1
prd-node 2
nodes
dev-master
tst-master
prd-master
*
*no staging related false alarms
*checks can‘t go amok in production
*configuration is tested properly in prd
*we‘ve talked about this...
*puppet & icinga, exported resources
monitoring configuration is completely described bypuppet
*
*stages are only reflected in monitoring masters
*non-prod doesn‘t alarm
*agent configuration is done via
puppet/package/however
*master configuration is done natively
*KISS
CLONE YOUR CONFIG!
*a quick howto
*
1. transfer prd stage to dev stage (files...)
2. edit configuration files/generate files (check_MK!)
3. sync files to VCS repository
* add/remove/modify
4. merge/copy dev-repo to prd-repo
* svn copy/svn merge/git branching
5. sync prd-repo to prd-filesystem
* again: add/remove/modify
workflow is done on dev-master
*
*
*local operations (on dev-master)
*checkout VCS
*checkin VCS
* remote operations (on prd-master)
*checkout VCS
*VCS operations
*transfer repositories (svn merge/git merge/...)
*operations are wrapped
*directly called for local operations
*puppet exec resources for remote operations
*
*checkout VCS
*from which repository?
* use a fact „stage“!
* a monitoring master can only checkout/sync its own stage
repository
*transfer repositories
*from where to where? the workflow knows...
*puppet runs on remote hosts are triggered by
mCollective
*addressing is done via „stage“ fact
*
*remote execution framework
*a simple ssh loop won‘t do it...
*message subscribe architecture
*parallel job execution
*a job => mCollective plugin
*ping, service, puppet, your own plugin
*watch out
*mco agent = subscriber
*mco client = message sender
*
*each directory is a repository
*??? => monitoring configuration is spread over the
filesystem
*ask cmk --paths
*the assignment dir to repo is configured in
„locations.cfg“
*configuration directories are working copies
*
file{‘/etc/mon/locations.cfg‘:
ensure => present,
source => puppet:///...,
notify => Exec[‘mon_checkout‘]
}
exec{‘mon_checkout‘:
command => ‘/usr/local/bin/mon checkout‘,
notify => Service[‘nagios‘],
}
service{‘nagios‘:
ensure => running,
}
*
*dirty little secrets
*only complete directories
*puppet run feedback?
*file permissions
*distributed/simultaneous work
*1:n versus n:m – the battle...
1:n n:m
separation complete separation partial separation
transfer
dev2prd
via release management
tools
on request
puppetization
level required
high low
risk low ...a bit more
complexity
(KISS)
high low
configuration via puppet natively
*
*risk versus implementation effort
risk
implementation effort
1:n
n:m