Delegated Config with Multiple

Hiera DatabasesRobert Terhaar rob@atlanticdynamic.com Atlantic Dynamic NYC

PuppetConf 2014 - September 24

Company & Personal Bio

• Build custom cloud, automation, deployment, and management systems for:

• Finance // Bio-Tech // Start-ups // Advertising

• Sysadmin since 1998

• Puppet user since 2007

• Based in NYC 🗽

Hiera Databaseswith Multiple

Delegated Config

What is Hiera?

Hiera is a framework for hierarchically

organizing data, and abstracting it from

your puppet manifests.

With Hiera, you can externalize your data, and easily understand

how configuration data is assigned to your servers.

What data belongs in Hiera?

Keep your secrets in

Hiera

https://github.com/TomPoulton/hiera-eyaml

Secrets in Hiera

Store your business-

logic in Hiera

Store all OS-specific config in params.pp

Hiera is not params.pp

Business Logic (Hiera) vs.

OS-Specific Config (params.pp)• Servers in production: use database IP 10.0.0.1

• In us-east1: use NTP server 167.88.119.29

• On RHEL7: SELINUX=enforcing

• Package names for Apache on Debian/RHEL: apache2/httpd

• 1 CPU = default to 1 workerbut on 4 CPUs = default to 5 workers

A Basic Hiera Example

Hiera Basics

$servers = hiera('ntp::servers')

As a Puppet Function

Hiera BasicsFunction in a parameterized class

class ntp( $servers = hiera('ntp::servers'),) { < ntp config goes here… >}

Hiera BasicsImplicit Lookup w/ Data Bindingsclass ntp( $servers,) { < ntp config goes here… >}

Hiera Basics

$servers = hiera(‘ntp::servers’, )

As a Puppet Function, w/ Default

‘pool.ntp.org’

$ cat /etc/puppet/hiera.yaml ---:backends: - yaml!:logger: console!:hierarchy: - "fqdn/%{fqdn}" - "role/%{role}" - "lifecycle/%{lifecycle}" - "location/%{location}" - common!:yaml: :datadir: /etc/puppet/hieradb

$ tree /etc/puppet/hieradb!

"## lifecycle$   "## dev.yaml$   "## production.yaml$   &## staging.yaml"## location$   &## us-east1.yaml&## os "## rhel6.yaml &## rhel7.yaml

Hiera supports multiple storage

backends

• eyaml • http (REST) • mysql • postgres

• redis • mongodb • json • yaml • and more…

$ cat /etc/puppet/hiera.yaml ---:backends:!!!:logger: console!:hierarchy: - "fqdn/%{fqdn}" - "role/%{role}" - "lifecycle/%{lifecycle}" - "location/%{location}" - common!:yaml: :datadir: /etc/puppet/hieradb!:postgres: :datadir: /etc/puppet/hieradb :host: <hostname> :user: <username> :pass: <password> :database: <database>

- yaml- postgres

$ tree /etc/puppet/hieradb!

"## lifecycle$   "## dev.yaml$   "##$   "## production.yaml$   &## staging.yaml"## location$   &## us-east1.yaml&## os "## rhel6.yaml &## rhel7.yaml

production.sql

But, let’s not focus on the tools… yet

Designing the solution

“Good design is as little design as possible”

https://www.vitsoe.com/us/about/good-design

Design and Architecture

http://commons.wikimedia.org/wiki/File:Fallingwater_%28Kaufmann_Residence_by_Frank_Lloyd_Wright%29_-_26_June_2012.jpg

“Architecture is the stuff that's hard to change later.

And there should be as little of that stuff as possible.”

- Martin Fowler

http://martinfowler.com/ieeeSoftware/whoNeedsArchitect.pdf

Semantics !

Architecture: Concrete Bricks !

Design: LEGO® Blocks

The Setup

Everyone ElseDevOps Team

Node

resource

resource

resource

resource

Node

resource

resource

resource

resource

environments

modules

hierapuppetdb

manifeststemplates

! - fqdn/%{::fqdn}! - lifecycle/%{::lifecycle}! - location/%{::location}! - os/%{::osfamily}! - common

Puppet Master

Everyone ElseDevOps Team

hiera

! - fqdn/%{::fqdn}! - lifecycle/%{::lifecycle}! - location/%{::location}! - os/windows.yml! - common

Everyone Else!! ntp::servers = [! “server1.corp”,! “server2.corp”,! ]

DevOps Team

hiera

! - fqdn/%{::fqdn}! - lifecycle/%{::lifecycle}! - location/%{::location}! - os/windows.yml! - common

Delegation

Step 1

Develop the Problem

Statement.

We need a way to delegate access to a few Hiera keys.

We need a way to delegate access to a few Hiera keys.

Colleagues who are not in the “DevOps Team” need to manage

a few pre-defined parameters. (but only on a subset of servers)

Everyone Else!! ntp::servers = [! “server1.corp”,! “server2.corp”,! ]

Special People Club

hiera

! - fqdn/%{::fqdn}! - lifecycle/%{::lifecycle}! - location/%{::location}! - os/windows.yml! - common

Colleagues who are not in the “DevOps Team” need to manage a

few pre-defined parameters. (but only on a subset of servers)

Step 2

Gathering Requirements

Requirement: The Solution Must Fly

Don’t over-engineer

Don’t over-engineer your

solution

Don’t over-engineer your requirements

document

Requirement “Types”

http://en.wikipedia.org/wiki/Requirements_analysis#Types_of_Requirements

• What are we building (1-2 sentence overview)

• What are the basic goals? (write them down!)

• How will we know when it’s done?

• What assumptions are we are making?

• What are some risks?

The Requirements Document

• Get feedback from…

• your boss

• the client

• your colleagues

• other stakeholders

The Requirements Document

Iterate the doc

Own the doc

Be Realistic & Prioritize

Feedback, Surprises

The Results

What are we building? !

We are building a data import system for Hiera which allows secure delegated access to end users. The system filters data, and can import data from various external systems.

• Import filtered data from various sources to a database. !

• That database is secondary Hiera backend datastore. !

• Adding additional import sources should be simple. !

• Easy to understand where keys are imported from.

Goals

How will we know when it’s done? The first version will be complete once we: • Build a prototype • Document the solution • Test importing data from a few sources • Create a deployment plan • Deploy to production

Step 3

Brainstorm, and Prototype

The solution without design

Everyone Else!!

UPDATE windows!SET value=‘[“server1.corp”, “server2.corp”]!WHERE key=‘ntp::servers’;

DevOps Team

hiera

We need a way to delegate access to a few Hiera keys.

PostgreSQL &! hiera-postgresql-

backend

! - fqdn/%{::fqdn}! - lifecycle/%{::lifecycle}! - location/%{::location}! - os/windows.sql! - common

windows.sql! ———! ntp::servers: SELECT value FROM windows WHERE key=‘ntp::servers’;

Single SQL Hiera Backend

The designed solution

Importer !App

Puppet!Master

Node

Node

resource

resource

resource

resource

resource

resource

resource

resource

DevOps Team

Delegated Hiera DB

Primary Hiera DB

filter

Everyone Else

Import Plugin 1

Import Plugin 2External

Data Source 2

White List!(What keys & namespaces are allowed)

Authoritative Hiera Data

External Data

Source 1

Simple data import script (run via cron)

The slightly better solution (logical diagram)

Custom Hiera Backend

The slightly better solution (with implementation detail)

Importer !App

Puppet!Master

Node

Node

resource

resource

resource

resource

resource

resource

resource

resource

DevOps Team

Delegated Hiera DB

Primary Hiera DB

filter

Everyone Else

Import Plugin 1

Import Plugin 2External

Data Source 2

External Data

Source 1

Python import script, with pluggable import backends

PostgreSQL DB

CMDB API, and LDAP

.yaml files stored in git

Results

Intentional design, and stakeholder

feedback will lead to a better solution.

DevOps Team

Everyone Else

Useful Resources

Good Design: https://www.vitsoe.com/us/about/good-design Learn More about Hiera: http://garylarizza.com/blog/2013/12/08/when-to-hiera/ Postgres Hiera Backend: https://github.com/adrianlzt/hiera-postgres-backend Hiera Encryption (eyaml): https://github.com/TomPoulton/hiera-eyaml Requirements: http://en.wikipedia.org/wiki/Requirements_analysis

Robert Terhaar rob@atlanticdynamic.com Atlantic Dynamic - NYC

PuppetConf 2014 - September 24

Questions?

Thank You!