Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak
-
Upload
netways -
Category
Technology
-
view
200 -
download
0
description
Transcript of Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak
![Page 1: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/1.jpg)
1
Marcin PiebiakSolutions ArchitectLinux Polska Sp. z o.o.
Workflow story – theory versus practice in Large Enterprises
![Page 2: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/2.jpg)
2
What is it?
package { 'ssh': ensure => 'installed', name => 'openssh-server',}
service { 'ssh': ensure => 'running', name => 'sshd', require => Package['ssh'],}
● ResourcesA lot of ready to use resources:
• user • group • host • cron • exec • file • package • service • ...
• Resources are building blocks.• They can be combined to make larger components.• Together they can model the expected state of your system.
![Page 3: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/3.jpg)
3
package { 'ssh': ensure => 'installed', name => 'openssh-server',}
service { 'ssh': ensure => 'running', name => 'sshd',}
● Resources● Declarative language
Rather than listing a series of steps to carry out we can describe the desired final state only.
What is it?
![Page 4: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/4.jpg)
4
package { 'ssh': ensure => 'installed', name => 'openssh-server',}
root@debian ~]# apt-get install openssh-server
root@redhat ~]# yum install openssh-server
● Resources● Declarative language● Abstraction
Resources in Puppet are abstracted from underlying providers.
What is it?
![Page 5: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/5.jpg)
5
root@redhat ~]# facterarchitecture => x86_64domain => linuxpolska.plfacterversion => 1.5.2fqdn => redhat.linuxpolska.plhardwaremodel => x86_64hostname => redhatinterfaces => eth0ipaddress => 172.16.10.1kernel => Linux...
● Resources● Declarative language● Abstraction● Facts
Puppet uses facter to gather information about the host system.
Custom Facts
Facter.add('role') do setcode do 'cat /etc/role' endend
What is it?
![Page 6: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/6.jpg)
6
$pkg_name = hiera('pkg_name')
package { 'apache': ensure => 'installed', name => $pkg_name,}
● Resources● Declarative language● Abstraction● Facts● Data separation
Puppet uses Hiera as its single source of truth data abstraction layer.
Hiera uses Facter facts to determine a hierarchy.
---:backends: - yaml:yaml: :datadir:/etc/hiera:hierarchy: - %{fqdn} - %{osfamily} - %{environment} - common
What is it?
![Page 7: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/7.jpg)
7
class ssh ( $pkg_name = 'openssh-server', $srv_name = 'sshd',) { package { 'ssh': ensure => 'installed', name => $pkg_name, }
service { 'ssh': ensure => 'running', enable => 'true', name => $srv_name, require => Package['ssh'], }}
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code
Classes define a collection of resources that are managed together as a single unit.
What is it?
![Page 8: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/8.jpg)
8
ssh├── manifests│ ├── init.pp│ └── server.pp├── files│ └── ssh_config└── templates └── sshd_config.erb
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code
Modules are directories that contain your configuration. They are designed to encapsulate all
of the components related to a given configuration in a single folder hierarchy.
Classes are abstracted by modules:
What is it?
![Page 9: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/9.jpg)
9
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code
If a more complex deployment is needed, reusing existing classes saves effort and reduces error.
What is it?
![Page 10: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/10.jpg)
10
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code
You own abstraction layers...Profiles:class profiles::application { include tomcat include mysql include myapp}class profiles::base { include ssh include ntp include users}
Roles:class role::webapp { include profiles::base include profiles::customapp include profiles::test_tools}
What is it?
![Page 11: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/11.jpg)
11
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code● Supports many OS
Supported OS
What is it?
![Page 12: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/12.jpg)
12
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code● Supports many OS
Big environment – no problem
Changes?
base/ntp.yaml --- ntp::local_servers: - 192.168.0.45+ - 192.168.0.1
What is it?
![Page 13: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/13.jpg)
13
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code● Supports many OS● Provisioning● Orchestration● Puppet Forge● Live Management● Environments● Reporting● Dry-run mode
And more...
VM/Cloud Provisioning
Orchestration
Live Management
MCollectiveReporting
Custom:- types & providers- facts- functions
Environments
Dry-run mode
What is it?
![Page 14: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/14.jpg)
14
● Resources● Declarative language● Abstraction● Facts● Data separation● Reusable code● Supports many OS● Provisioning● Orchestration● Puppet Forge● Live Management● Environments● Reporting● Dry-run mode
What is it?
Thanks to these all superior features, Puppet is:- fast in deployment- easy to use- universal for a lot of operating systems- with unlimited possibilities- easy to expand- flexible
![Page 15: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/15.jpg)
15
VS.
In large enterprises like banks, telco, insurance, etc. those features are not the most relevant. Implementing Puppet in enterprises we must consider another
priority map, another mindset.
We must answer not trivial questions dealing with the core IT Departments way of work.
![Page 16: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/16.jpg)
16
VS.
So... everyone hasaccess to everything?!
![Page 17: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/17.jpg)
17
VS.
So... can I destroythe whole infrastructure
from one place?
![Page 18: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/18.jpg)
18
SolutionPuppet master installation
on hardened system with limited direct access.
tcp:8140
tcp:443
tcp:22
ssh keys
RBAC
For maintenance.
![Page 19: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/19.jpg)
19
Solution
Git as a communication layer between developers and puppet master.
Release manager
developer
developer
developer
Pull request
Fetch
![Page 20: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/20.jpg)
20
VS.
So... everyone now mustuse puppet?!
![Page 21: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/21.jpg)
21
Solution
On beginning each department can have own environments.
Developers
Systems administrators Databases
administrators
Securitydepartment
![Page 22: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/22.jpg)
22
SolutionEach department can have
many environments and its own idea how to organizework with puppet. Security
department
prodtestdev
Integrator
Releasemanager
Testsintegrator
![Page 23: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/23.jpg)
23
SolutionEach department can have
many environments and its own idea how to organizework with puppet. Security
department
prodtestdev
Integrator
Releasemanager
Testsintegrator
v0.2v0.1
commits
Release
![Page 24: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/24.jpg)
24
Solution
After time some departments will start working together in one environment.
Developers
Systems administrators Databases
administrators
Securitydepartment
integrator
![Page 25: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/25.jpg)
25
SolutionDevelopers
Systems administrators Databases
administrators
Securitydepartment
integrator
integrator
![Page 26: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/26.jpg)
26
Solution
At the end all departments will use one environment.
Developers
Systems administrators Databases
administrators
Securitydepartment
integrator
integrator
![Page 27: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/27.jpg)
27
VS.
How can I find outwho made what change
and who approvedthis modification?
![Page 28: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/28.jpg)
28
Solution
prodtestdev
v0.2v0.1
commits
Release
commit 220938c5a2e51ecd4166eb7d75d14974cbcff897Author: Marcin Piebiak <[email protected]>Date: Fri Jul 5 11:27:43 2013 +0200
Description....
Person who approvedmodifications.
Using git we have:● date● author● description
We can use git as a place for history of the infrastructure.● git status● git log● git diff● git blame
Git history is cryptographically secured.
![Page 29: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/29.jpg)
29
VS.
If I have a lot ofenvironments how can
I use them?
![Page 30: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/30.jpg)
30
Solution
tcp:8140
prodtestdev
We can specify a set of environments for each host to use.
Database testing system, uses three testing environments from different departments.
![Page 31: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/31.jpg)
31
VS.
Great!But... using commandline I can connect to
different environments!
![Page 32: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/32.jpg)
32
Solution
tcp:8140
puppet agent -t --environment
We use imp module, to control puppet agents behavior and their access to environments.
![Page 33: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/33.jpg)
33
Solutionpuppet.conf
[main]modulepath = /etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modulesmanifest = /etc/puppetlabs/manifests/site.pp
[env_sec_prod]modulepath = /var/lib/git/env_sec_prod/modules:/etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules
[env_sec_test]modulepath = /var/lib/git/env_sec_test/modules:/etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules
site.pp
include imp
hiera_include('include')
/etc/puppetlabs/puppet/modules – place for modules from forge and well tested modules and module imp
/etc/puppetlabs/manifests/site.pp - common manifest for all environments
/var/lib/git/$environment/modules – git repository for environment
imp.yaml
---imp::environments: env_sec_prod: order: 'deny,allow' deny: 'all' allow: - 'host1.linuxpolska.lab' - 'www.*.linuxpolska.lab' commiters: - 'john.smith' priority: 1
![Page 34: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/34.jpg)
34
VS.
How can I auditchanges made in
the infrastructure?
![Page 35: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/35.jpg)
35
Solution
Log collector and analyzer.
puppet reports
Puppet reports
When nodes fetch their configurations from the puppet master, they send back inventory data and a report of their run.
![Page 36: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/36.jpg)
36
VS.
How can puppet helpme with audit? How can
I recreate life cycle of each host?
![Page 37: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/37.jpg)
37
Solution
Log collector and analyzer.
● puppet reports● puppet agents catalog● hosts facts● git diffs after commit● hiera configuration for each host● filebuckets● ...
![Page 38: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/38.jpg)
38
Solution
app db system security
file {'/etc/important': ensure => 'file', group => 'apache', mode => '0660',}
file {'/etc/important': ensure => 'file', user => 'root', group => 'root', mode => '0600',}
If we have many environment there is always risk of overwriting someone's changes.
Log collector and analyzer.
![Page 39: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/39.jpg)
39
VS.
How will the modification in puppet
manifestsaffect the wholeinfrastructure?
![Page 40: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/40.jpg)
40
Solution
prodtestdev
v0.2v0.1
commitsLog collector
and analyzer.
Report from puppet normal run.
Report from puppet dry-run.
Using log collector we can analyze the infrastructure modifications before they get to production environment.
![Page 41: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/41.jpg)
41
VS.
All changes are madeautomatically? First I would
like to see what is going to be changed.
![Page 42: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/42.jpg)
42
VS.
How can we rollbackchanges?
![Page 43: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/43.jpg)
43
VS.
After we installpuppet will we know
everything about the infrastructure?
![Page 44: Workflow story: Theory versus Practice in large enterprises by Marcin Piebiak](https://reader034.fdocuments.net/reader034/viewer/2022042715/55878f9fd8b42a365d8b46cf/html5/thumbnails/44.jpg)
44
THE END
Marcin PiebiakSolutions ArchitectLinux Polska Sp. z o.o.